* [Qemu-devel] [PATCH 01/17] block: access copy_on_read with atomic ops
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 11:15 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 02/17] block: access quiesce_counter " Paolo Bonzini
` (20 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block.c | 6 ++++--
block/io.c | 8 ++++----
blockdev.c | 2 +-
include/block/block_int.h | 11 ++++++-----
4 files changed, 15 insertions(+), 12 deletions(-)
diff --git a/block.c b/block.c
index 1fbbb8d..a48fdd8 100644
--- a/block.c
+++ b/block.c
@@ -1189,7 +1189,9 @@ static int bdrv_open_common(BlockDriverState *bs, BlockBackend *file,
goto fail_opts;
}
- assert(bs->copy_on_read == 0); /* bdrv_new() and bdrv_close() make it so */
+ /* bdrv_new() and bdrv_close() make it so */
+ assert(atomic_read(&bs->copy_on_read) == 0);
+
if (bs->open_flags & BDRV_O_COPY_ON_READ) {
if (!bs->read_only) {
bdrv_enable_copy_on_read(bs);
@@ -2937,7 +2939,7 @@ static void bdrv_close(BlockDriverState *bs)
g_free(bs->opaque);
bs->opaque = NULL;
- bs->copy_on_read = 0;
+ atomic_set(&bs->copy_on_read, 0);
bs->backing_file[0] = '\0';
bs->backing_format[0] = '\0';
bs->total_sectors = 0;
diff --git a/block/io.c b/block/io.c
index a54e5c8..08de488 100644
--- a/block/io.c
+++ b/block/io.c
@@ -130,13 +130,13 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
*/
void bdrv_enable_copy_on_read(BlockDriverState *bs)
{
- bs->copy_on_read++;
+ atomic_inc(&bs->copy_on_read);
}
void bdrv_disable_copy_on_read(BlockDriverState *bs)
{
- assert(bs->copy_on_read > 0);
- bs->copy_on_read--;
+ assert(atomic_read(&bs->copy_on_read) > 0);
+ atomic_dec(&bs->copy_on_read);
}
/* Check if any requests are in-flight (including throttled requests) */
@@ -1144,7 +1144,7 @@ int coroutine_fn bdrv_co_preadv(BdrvChild *child,
bdrv_inc_in_flight(bs);
/* Don't do copy-on-read if we read data before write operation */
- if (bs->copy_on_read && !(flags & BDRV_REQ_NO_SERIALISING)) {
+ if (atomic_read(&bs->copy_on_read) && !(flags & BDRV_REQ_NO_SERIALISING)) {
flags |= BDRV_REQ_COPY_ON_READ;
}
diff --git a/blockdev.c b/blockdev.c
index 9098233..e9b5717 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1794,7 +1794,7 @@ static void external_snapshot_commit(BlkActionState *common)
/* We don't need (or want) to use the transactional
* bdrv_reopen_multiple() across all the entries at once, because we
* don't want to abort all of them if one of them fails the reopen */
- if (!state->old_bs->copy_on_read) {
+ if (!atomic_read(&state->old_bs->copy_on_read)) {
bdrv_reopen(state->old_bs, state->old_bs->open_flags & ~BDRV_O_RDWR,
NULL);
}
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 59400bd..d0b88ff 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -581,11 +581,6 @@ struct BlockDriverState {
/* Protected by AioContext lock */
- /* If true, copy read backing sectors into image. Can be >1 if more
- * than one client has requested copy-on-read.
- */
- int copy_on_read;
-
/* If we are reading a disk image, give its size in sectors.
* Generally read-only; it is written to by load_vmstate and save_vmstate,
* but the block layer is quiescent during those.
@@ -619,6 +614,12 @@ struct BlockDriverState {
QLIST_HEAD(, BdrvDirtyBitmap) dirty_bitmaps;
+ /* If true, copy read backing sectors into image. Can be >1 if more
+ * than one client has requested copy-on-read. Accessed with atomic
+ * ops.
+ */
+ int copy_on_read;
+
/* do we need to tell the quest if we have a volatile write cache? */
int enable_write_cache;
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH 01/17] block: access copy_on_read with atomic ops
2017-04-20 12:00 ` [Qemu-devel] [PATCH 01/17] block: access copy_on_read with atomic ops Paolo Bonzini
@ 2017-05-04 11:15 ` Stefan Hajnoczi
2017-05-04 11:51 ` Paolo Bonzini
0 siblings, 1 reply; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-04 11:15 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 380 bytes --]
On Thu, Apr 20, 2017 at 02:00:42PM +0200, Paolo Bonzini wrote:
> void bdrv_disable_copy_on_read(BlockDriverState *bs)
> {
> - assert(bs->copy_on_read > 0);
> - bs->copy_on_read--;
> + assert(atomic_read(&bs->copy_on_read) > 0);
> + atomic_dec(&bs->copy_on_read);
> }
To make this truly thread-safe:
old = atomic_dec_fetch(&bs->copy_on_read);
assert(old > 0);
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH 01/17] block: access copy_on_read with atomic ops
2017-05-04 11:15 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
@ 2017-05-04 11:51 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 11:51 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: qemu-devel, qemu-block
On 04/05/2017 13:15, Stefan Hajnoczi wrote:
> On Thu, Apr 20, 2017 at 02:00:42PM +0200, Paolo Bonzini wrote:
>> void bdrv_disable_copy_on_read(BlockDriverState *bs)
>> {
>> - assert(bs->copy_on_read > 0);
>> - bs->copy_on_read--;
>> + assert(atomic_read(&bs->copy_on_read) > 0);
>> + atomic_dec(&bs->copy_on_read);
>> }
>
> To make this truly thread-safe:
>
> old = atomic_dec_fetch(&bs->copy_on_read);
> assert(old > 0);
Good point. It feels wrong to assert after the fact, but then so does
making the assertion not quite thread safe.
Paolo
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 02/17] block: access quiesce_counter with atomic ops
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
2017-04-20 12:00 ` [Qemu-devel] [PATCH 01/17] block: access copy_on_read with atomic ops Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 12:33 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 03/17] block: access io_limits_disabled " Paolo Bonzini
` (19 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/io.c | 4 ++--
include/block/block_int.h | 1 +
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/block/io.c b/block/io.c
index 08de488..3a27fd5 100644
--- a/block/io.c
+++ b/block/io.c
@@ -241,7 +241,7 @@ void bdrv_drained_begin(BlockDriverState *bs)
return;
}
- if (!bs->quiesce_counter++) {
+ if (atomic_fetch_inc(&bs->quiesce_counter) == 0) {
aio_disable_external(bdrv_get_aio_context(bs));
bdrv_parent_drained_begin(bs);
}
@@ -252,7 +252,7 @@ void bdrv_drained_begin(BlockDriverState *bs)
void bdrv_drained_end(BlockDriverState *bs)
{
assert(bs->quiesce_counter > 0);
- if (--bs->quiesce_counter > 0) {
+ if (atomic_fetch_dec(&bs->quiesce_counter) > 1) {
return;
}
diff --git a/include/block/block_int.h b/include/block/block_int.h
index d0b88ff..a43fe78 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -623,6 +623,7 @@ struct BlockDriverState {
/* do we need to tell the quest if we have a volatile write cache? */
int enable_write_cache;
+ /* Accessed with atomic ops. */
int quiesce_counter;
};
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* [Qemu-devel] [PATCH 03/17] block: access io_limits_disabled with atomic ops
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
2017-04-20 12:00 ` [Qemu-devel] [PATCH 01/17] block: access copy_on_read with atomic ops Paolo Bonzini
2017-04-20 12:00 ` [Qemu-devel] [PATCH 02/17] block: access quiesce_counter " Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 12:38 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 04/17] block: access serialising_in_flight " Paolo Bonzini
` (18 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/block-backend.c | 4 ++--
block/throttle-groups.c | 2 +-
include/sysemu/block-backend.h | 3 ++-
3 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/block/block-backend.c b/block/block-backend.c
index 7405024..915ccc5 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -1927,7 +1927,7 @@ static void blk_root_drained_begin(BdrvChild *child)
/* Note that blk->root may not be accessible here yet if we are just
* attaching to a BlockDriverState that is drained. Use child instead. */
- if (blk->public.io_limits_disabled++ == 0) {
+ if (atomic_fetch_inc(&blk->public.io_limits_disabled) == 0) {
throttle_group_restart_blk(blk);
}
}
@@ -1938,7 +1938,7 @@ static void blk_root_drained_end(BdrvChild *child)
assert(blk->quiesce_counter);
assert(blk->public.io_limits_disabled);
- --blk->public.io_limits_disabled;
+ atomic_dec(&blk->public.io_limits_disabled);
if (--blk->quiesce_counter == 0) {
if (blk->dev_ops && blk->dev_ops->drained_end) {
diff --git a/block/throttle-groups.c b/block/throttle-groups.c
index b73e7a8..69bfbd4 100644
--- a/block/throttle-groups.c
+++ b/block/throttle-groups.c
@@ -240,7 +240,7 @@ static bool throttle_group_schedule_timer(BlockBackend *blk, bool is_write)
ThrottleGroup *tg = container_of(ts, ThrottleGroup, ts);
bool must_wait;
- if (blkp->io_limits_disabled) {
+ if (atomic_read(&blkp->io_limits_disabled)) {
return false;
}
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index 7462228..87a43b0 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -80,7 +80,8 @@ typedef struct BlockBackendPublic {
CoQueue throttled_reqs[2];
/* Nonzero if the I/O limits are currently being ignored; generally
- * it is zero. */
+ * it is zero. Accessed with atomic operations.
+ */
unsigned int io_limits_disabled;
/* The following fields are protected by the ThrottleGroup lock.
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* [Qemu-devel] [PATCH 04/17] block: access serialising_in_flight with atomic ops
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (2 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 03/17] block: access io_limits_disabled " Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 12:39 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 05/17] block: access wakeup " Paolo Bonzini
` (17 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/io.c | 6 +++---
include/block/block_int.h | 10 ++++++----
2 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/block/io.c b/block/io.c
index 3a27fd5..869322a 100644
--- a/block/io.c
+++ b/block/io.c
@@ -375,7 +375,7 @@ void bdrv_drain_all(void)
static void tracked_request_end(BdrvTrackedRequest *req)
{
if (req->serialising) {
- req->bs->serialising_in_flight--;
+ atomic_dec(&req->bs->serialising_in_flight);
}
QLIST_REMOVE(req, list);
@@ -414,7 +414,7 @@ static void mark_request_serialising(BdrvTrackedRequest *req, uint64_t align)
- overlap_offset;
if (!req->serialising) {
- req->bs->serialising_in_flight++;
+ atomic_inc(&req->bs->serialising_in_flight);
req->serialising = true;
}
@@ -519,7 +519,7 @@ static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self)
bool retry;
bool waited = false;
- if (!bs->serialising_in_flight) {
+ if (!atomic_read(&bs->serialising_in_flight)) {
return false;
}
diff --git a/include/block/block_int.h b/include/block/block_int.h
index a43fe78..7317db6 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -590,10 +590,6 @@ struct BlockDriverState {
/* Callback before write request is processed */
NotifierWithReturnList before_write_notifiers;
- /* number of in-flight requests; overall and serialising */
- unsigned int in_flight;
- unsigned int serialising_in_flight;
-
bool wakeup;
/* Offset after the highest byte written to */
@@ -620,6 +616,12 @@ struct BlockDriverState {
*/
int copy_on_read;
+ /* number of in-flight requests; overall and serialising.
+ * Accessed with atomic ops.
+ */
+ unsigned int in_flight;
+ unsigned int serialising_in_flight;
+
/* do we need to tell the quest if we have a volatile write cache? */
int enable_write_cache;
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* [Qemu-devel] [PATCH 05/17] block: access wakeup with atomic ops
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (3 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 04/17] block: access serialising_in_flight " Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 6:39 ` Fam Zheng
2017-05-04 12:47 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 06/17] block: access io_plugged " Paolo Bonzini
` (16 subsequent siblings)
21 siblings, 2 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/io.c | 3 ++-
block/nfs.c | 4 +++-
block/sheepdog.c | 3 ++-
include/block/block.h | 5 +++--
include/block/block_int.h | 4 ++--
5 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/block/io.c b/block/io.c
index 869322a..3b2ede9 100644
--- a/block/io.c
+++ b/block/io.c
@@ -501,7 +501,8 @@ static void dummy_bh_cb(void *opaque)
void bdrv_wakeup(BlockDriverState *bs)
{
- if (bs->wakeup) {
+ /* The barrier (or an atomic op) is in the caller. */
+ if (atomic_read(&bs->wakeup)) {
aio_bh_schedule_oneshot(qemu_get_aio_context(), dummy_bh_cb, NULL);
}
}
diff --git a/block/nfs.c b/block/nfs.c
index 0816678..fd2508a 100644
--- a/block/nfs.c
+++ b/block/nfs.c
@@ -736,7 +736,9 @@ nfs_get_allocated_file_size_cb(int ret, struct nfs_context *nfs, void *data,
if (task->ret < 0) {
error_report("NFS Error: %s", nfs_get_error(nfs));
}
- task->complete = 1;
+
+ /* Set task->complete before reading bs->wakeup. */
+ atomic_mb_set(&task->complete, 1);
bdrv_wakeup(task->bs);
}
diff --git a/block/sheepdog.c b/block/sheepdog.c
index fb9203e..7e90a24 100644
--- a/block/sheepdog.c
+++ b/block/sheepdog.c
@@ -704,7 +704,8 @@ out:
srco->co = NULL;
srco->ret = ret;
- srco->finished = true;
+ /* Set srco->finished before reading bs->wakeup. */
+ atomic_mb_set(&srco->finished, true);
if (srco->bs) {
bdrv_wakeup(srco->bs);
}
diff --git a/include/block/block.h b/include/block/block.h
index 5ddc0cf..486b6ed 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -398,7 +398,8 @@ void bdrv_drain_all(void);
* block_job_defer_to_main_loop for how to do it). \
*/ \
assert(!bs_->wakeup); \
- bs_->wakeup = true; \
+ /* Set bs->wakeup before evaluating cond. */ \
+ atomic_mb_set(&bs_->wakeup, true); \
while (busy_) { \
if ((cond)) { \
waited_ = busy_ = true; \
@@ -410,7 +411,7 @@ void bdrv_drain_all(void);
waited_ |= busy_; \
} \
} \
- bs_->wakeup = false; \
+ atomic_set(&bs_->wakeup, false); \
} \
waited_; })
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 7317db6..ca34c99 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -590,8 +590,6 @@ struct BlockDriverState {
/* Callback before write request is processed */
NotifierWithReturnList before_write_notifiers;
- bool wakeup;
-
/* Offset after the highest byte written to */
uint64_t wr_highest_offset;
@@ -622,6 +620,8 @@ struct BlockDriverState {
unsigned int in_flight;
unsigned int serialising_in_flight;
+ bool wakeup;
+
/* do we need to tell the quest if we have a volatile write cache? */
int enable_write_cache;
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 05/17] block: access wakeup with atomic ops
2017-04-20 12:00 ` [Qemu-devel] [PATCH 05/17] block: access wakeup " Paolo Bonzini
@ 2017-05-04 6:39 ` Fam Zheng
2017-05-04 7:12 ` Paolo Bonzini
2017-05-04 12:47 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
1 sibling, 1 reply; 65+ messages in thread
From: Fam Zheng @ 2017-05-04 6:39 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, 04/20 14:00, Paolo Bonzini wrote:
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> block/io.c | 3 ++-
> block/nfs.c | 4 +++-
> block/sheepdog.c | 3 ++-
> include/block/block.h | 5 +++--
> include/block/block_int.h | 4 ++--
> 5 files changed, 12 insertions(+), 7 deletions(-)
>
> diff --git a/block/io.c b/block/io.c
> index 869322a..3b2ede9 100644
> --- a/block/io.c
> +++ b/block/io.c
> @@ -501,7 +501,8 @@ static void dummy_bh_cb(void *opaque)
>
> void bdrv_wakeup(BlockDriverState *bs)
> {
> - if (bs->wakeup) {
> + /* The barrier (or an atomic op) is in the caller. */
Why not add a barrier here so that callers don't need to worry about that?
> + if (atomic_read(&bs->wakeup)) {
> aio_bh_schedule_oneshot(qemu_get_aio_context(), dummy_bh_cb, NULL);
> }
> }
Fam
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 05/17] block: access wakeup with atomic ops
2017-05-04 6:39 ` Fam Zheng
@ 2017-05-04 7:12 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 7:12 UTC (permalink / raw)
To: Fam Zheng; +Cc: qemu-devel, qemu-block
On 04/05/2017 08:39, Fam Zheng wrote:
> On Thu, 04/20 14:00, Paolo Bonzini wrote:
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> ---
>> block/io.c | 3 ++-
>> block/nfs.c | 4 +++-
>> block/sheepdog.c | 3 ++-
>> include/block/block.h | 5 +++--
>> include/block/block_int.h | 4 ++--
>> 5 files changed, 12 insertions(+), 7 deletions(-)
>>
>> diff --git a/block/io.c b/block/io.c
>> index 869322a..3b2ede9 100644
>> --- a/block/io.c
>> +++ b/block/io.c
>> @@ -501,7 +501,8 @@ static void dummy_bh_cb(void *opaque)
>>
>> void bdrv_wakeup(BlockDriverState *bs)
>> {
>> - if (bs->wakeup) {
>> + /* The barrier (or an atomic op) is in the caller. */
>
> Why not add a barrier here so that callers don't need to worry about that?
Barriers are relatively expensive, and the common case is
dataplane7:block/io.c-void bdrv_dec_in_flight(BlockDriverState *bs)
dataplane7:block/io.c-{
dataplane7:block/io.c- atomic_dec(&bs->in_flight);
dataplane7:block/io.c: bdrv_wakeup(bs);
dataplane7:block/io.c-}
Paolo
>> + if (atomic_read(&bs->wakeup)) {
>> aio_bh_schedule_oneshot(qemu_get_aio_context(), dummy_bh_cb, NULL);
>> }
>> }
>
> Fam
>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH 05/17] block: access wakeup with atomic ops
2017-04-20 12:00 ` [Qemu-devel] [PATCH 05/17] block: access wakeup " Paolo Bonzini
2017-05-04 6:39 ` Fam Zheng
@ 2017-05-04 12:47 ` Stefan Hajnoczi
1 sibling, 0 replies; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-04 12:47 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 330 bytes --]
On Thu, Apr 20, 2017 at 02:00:46PM +0200, Paolo Bonzini wrote:
> @@ -622,6 +620,8 @@ struct BlockDriverState {
> unsigned int in_flight;
> unsigned int serialising_in_flight;
>
> + bool wakeup;
You added a comment that this field is accessed atomically in the other
patches. Should it be added here too?
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 06/17] block: access io_plugged with atomic ops
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (4 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 05/17] block: access wakeup " Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 12:48 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 07/17] throttle-groups: do not use qemu_co_enter_next Paolo Bonzini
` (15 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/io.c | 4 ++--
include/block/block_int.h | 8 +++++---
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/block/io.c b/block/io.c
index 3b2ede9..3ab9476 100644
--- a/block/io.c
+++ b/block/io.c
@@ -2653,7 +2653,7 @@ void bdrv_io_plug(BlockDriverState *bs)
bdrv_io_plug(child->bs);
}
- if (bs->io_plugged++ == 0) {
+ if (atomic_fetch_inc(&bs->io_plugged) == 0) {
BlockDriver *drv = bs->drv;
if (drv && drv->bdrv_io_plug) {
drv->bdrv_io_plug(bs);
@@ -2666,7 +2666,7 @@ void bdrv_io_unplug(BlockDriverState *bs)
BdrvChild *child;
assert(bs->io_plugged);
- if (--bs->io_plugged == 0) {
+ if (atomic_fetch_dec(&bs->io_plugged) == 1) {
BlockDriver *drv = bs->drv;
if (drv && drv->bdrv_io_unplug) {
drv->bdrv_io_unplug(bs);
diff --git a/include/block/block_int.h b/include/block/block_int.h
index ca34c99..283d079 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -597,9 +597,6 @@ struct BlockDriverState {
uint64_t write_threshold_offset;
NotifierWithReturn write_threshold_notifier;
- /* counter for nested bdrv_io_plug */
- unsigned io_plugged;
-
QLIST_HEAD(, BdrvTrackedRequest) tracked_requests;
CoQueue flush_queue; /* Serializing flush queue */
bool active_flush_req; /* Flush request in flight? */
@@ -622,6 +619,11 @@ struct BlockDriverState {
bool wakeup;
+ /* counter for nested bdrv_io_plug.
+ * Accessed with atomic ops.
+ */
+ unsigned io_plugged;
+
/* do we need to tell the quest if we have a volatile write cache? */
int enable_write_cache;
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* [Qemu-devel] [PATCH 07/17] throttle-groups: do not use qemu_co_enter_next
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (5 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 06/17] block: access io_plugged " Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 13:27 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 08/17] throttle-groups: protect throttled requests with a CoMutex Paolo Bonzini
` (14 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Prepare for removing this function; always restart throttled requests
from coroutine context. This will matter when restarting throttled
requests will have to acquire a CoMutex.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/throttle-groups.c | 65 +++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 58 insertions(+), 7 deletions(-)
diff --git a/block/throttle-groups.c b/block/throttle-groups.c
index 69bfbd4..d66bf62 100644
--- a/block/throttle-groups.c
+++ b/block/throttle-groups.c
@@ -260,6 +260,18 @@ static bool throttle_group_schedule_timer(BlockBackend *blk, bool is_write)
return must_wait;
}
+/* Start the next pending I/O request for a BlockBackend.
+ *
+ * @blk: the current BlockBackend
+ * @is_write: the type of operation (read/write)
+ */
+static bool throttle_group_co_restart_queue(BlockBackend *blk, bool is_write)
+{
+ BlockBackendPublic *blkp = blk_get_public(blk);
+
+ return qemu_co_queue_next(&blkp->throttled_reqs[is_write]);
+}
+
/* Look for the next pending I/O request and schedule it.
*
* This assumes that tg->lock is held.
@@ -287,7 +299,7 @@ static void schedule_next_request(BlockBackend *blk, bool is_write)
if (!must_wait) {
/* Give preference to requests from the current blk */
if (qemu_in_coroutine() &&
- qemu_co_queue_next(&blkp->throttled_reqs[is_write])) {
+ throttle_group_co_restart_queue(blk, is_write)) {
token = blk;
} else {
ThrottleTimers *tt = &blk_get_public(token)->throttle_timers;
@@ -340,18 +352,57 @@ void coroutine_fn throttle_group_co_io_limits_intercept(BlockBackend *blk,
qemu_mutex_unlock(&tg->lock);
}
-void throttle_group_restart_blk(BlockBackend *blk)
+typedef struct {
+ BlockBackend *blk;
+ bool is_write;
+ int ret;
+} RestartData;
+
+static void throttle_group_restart_queue_entry(void *opaque)
{
- BlockBackendPublic *blkp = blk_get_public(blk);
+ RestartData *data = opaque;
+
+ data->ret = throttle_group_co_restart_queue(data->blk, data->is_write);
+}
+
+static int throttle_group_restart_queue(BlockBackend *blk, bool is_write)
+{
+ Coroutine *co;
+ RestartData rd = {
+ .blk = blk,
+ .is_write = is_write
+ };
+
+ aio_context_acquire(blk_get_aio_context(blk));
+ co = qemu_coroutine_create(throttle_group_restart_queue_entry, &rd);
+ /* The request doesn't start until after throttle_group_restart_queue_entry
+ * returns, so the coroutine cannot yield.
+ */
+ qemu_coroutine_enter(co);
+ aio_context_release(blk_get_aio_context(blk));
+ return rd.ret;
+}
+
+static void throttle_group_restart_blk_entry(void *opaque)
+{
+ BlockBackend *blk = opaque;
int i;
for (i = 0; i < 2; i++) {
- while (qemu_co_enter_next(&blkp->throttled_reqs[i])) {
+ while (throttle_group_co_restart_queue(blk, i)) {
;
}
}
}
+void throttle_group_restart_blk(BlockBackend *blk)
+{
+ Coroutine *co;
+
+ co = qemu_coroutine_create(throttle_group_restart_blk_entry, blk);
+ qemu_coroutine_enter(co);
+}
+
/* Update the throttle configuration for a particular group. Similar
* to throttle_config(), but guarantees atomicity within the
* throttling group.
@@ -376,8 +427,8 @@ void throttle_group_config(BlockBackend *blk, ThrottleConfig *cfg)
throttle_config(ts, tt, cfg);
qemu_mutex_unlock(&tg->lock);
- qemu_co_enter_next(&blkp->throttled_reqs[0]);
- qemu_co_enter_next(&blkp->throttled_reqs[1]);
+ throttle_group_restart_queue(blk, 0);
+ throttle_group_restart_queue(blk, 1);
}
/* Get the throttle configuration from a particular group. Similar to
@@ -417,7 +468,7 @@ static void timer_cb(BlockBackend *blk, bool is_write)
/* Run the request that was waiting for this timer */
aio_context_acquire(blk_get_aio_context(blk));
- empty_queue = !qemu_co_enter_next(&blkp->throttled_reqs[is_write]);
+ empty_queue = !throttle_group_restart_queue(blk, is_write);
aio_context_release(blk_get_aio_context(blk));
/* If the request queue was empty then we have to take care of
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH 07/17] throttle-groups: do not use qemu_co_enter_next
2017-04-20 12:00 ` [Qemu-devel] [PATCH 07/17] throttle-groups: do not use qemu_co_enter_next Paolo Bonzini
@ 2017-05-04 13:27 ` Stefan Hajnoczi
0 siblings, 0 replies; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-04 13:27 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 5558 bytes --]
On Thu, Apr 20, 2017 at 02:00:48PM +0200, Paolo Bonzini wrote:
> Prepare for removing this function; always restart throttled requests
> from coroutine context. This will matter when restarting throttled
> requests will have to acquire a CoMutex.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> block/throttle-groups.c | 65 +++++++++++++++++++++++++++++++++++++++++++------
> 1 file changed, 58 insertions(+), 7 deletions(-)
I don't understand this patch :(.
> diff --git a/block/throttle-groups.c b/block/throttle-groups.c
> index 69bfbd4..d66bf62 100644
> --- a/block/throttle-groups.c
> +++ b/block/throttle-groups.c
> @@ -260,6 +260,18 @@ static bool throttle_group_schedule_timer(BlockBackend *blk, bool is_write)
> return must_wait;
> }
>
> +/* Start the next pending I/O request for a BlockBackend.
> + *
> + * @blk: the current BlockBackend
> + * @is_write: the type of operation (read/write)
The return value is undocumented.
> + */
> +static bool throttle_group_co_restart_queue(BlockBackend *blk, bool is_write)
This function calls qemu_co_queue_next(), which has coroutine_fn, so
this function must also be marked coroutine_fn.
> +{
> + BlockBackendPublic *blkp = blk_get_public(blk);
> +
> + return qemu_co_queue_next(&blkp->throttled_reqs[is_write]);
> +}
> +
> /* Look for the next pending I/O request and schedule it.
> *
> * This assumes that tg->lock is held.
> @@ -287,7 +299,7 @@ static void schedule_next_request(BlockBackend *blk, bool is_write)
> if (!must_wait) {
> /* Give preference to requests from the current blk */
> if (qemu_in_coroutine() &&
> - qemu_co_queue_next(&blkp->throttled_reqs[is_write])) {
> + throttle_group_co_restart_queue(blk, is_write)) {
> token = blk;
> } else {
> ThrottleTimers *tt = &blk_get_public(token)->throttle_timers;
> @@ -340,18 +352,57 @@ void coroutine_fn throttle_group_co_io_limits_intercept(BlockBackend *blk,
> qemu_mutex_unlock(&tg->lock);
> }
>
> -void throttle_group_restart_blk(BlockBackend *blk)
> +typedef struct {
> + BlockBackend *blk;
> + bool is_write;
> + int ret;
s/ret/bool/
> +} RestartData;
> +
> +static void throttle_group_restart_queue_entry(void *opaque)
This is a coroutine entry function, it must be marked coroutine_fn.
> {
> - BlockBackendPublic *blkp = blk_get_public(blk);
> + RestartData *data = opaque;
> +
> + data->ret = throttle_group_co_restart_queue(data->blk, data->is_write);
> +}
> +
> +static int throttle_group_restart_queue(BlockBackend *blk, bool is_write)
s/int/bool/
> +{
> + Coroutine *co;
> + RestartData rd = {
> + .blk = blk,
> + .is_write = is_write
> + };
> +
> + aio_context_acquire(blk_get_aio_context(blk));
> + co = qemu_coroutine_create(throttle_group_restart_queue_entry, &rd);
> + /* The request doesn't start until after throttle_group_restart_queue_entry
> + * returns, so the coroutine cannot yield.
> + */
I don't understand this sentence.
This function assumes that throttle_group_restart_queue_entry(),
throttle_group_co_restart_queue(), and qemu_co_queue_next() do not
yield. Making these assumptions is fragile :(.
But how does that relate to "the request doesn't start until after
throttle_group_restart_queue_entry returns"?
> + qemu_coroutine_enter(co);
> + aio_context_release(blk_get_aio_context(blk));
> + return rd.ret;
> +}
> +
> +static void throttle_group_restart_blk_entry(void *opaque)
This is a coroutine entry function so it must be marked coroutine_fn.
> +{
> + BlockBackend *blk = opaque;
> int i;
>
> for (i = 0; i < 2; i++) {
> - while (qemu_co_enter_next(&blkp->throttled_reqs[i])) {
> + while (throttle_group_co_restart_queue(blk, i)) {
> ;
> }
> }
Why is the throttle_group_restart_blk_entry() coroutine necessary?
throttle_group_co_restart_queue() already spawns a coroutine which is
used for queuing. I think this function could be a non-coroutine
function.
> }
>
> +void throttle_group_restart_blk(BlockBackend *blk)
> +{
> + Coroutine *co;
> +
> + co = qemu_coroutine_create(throttle_group_restart_blk_entry, blk);
> + qemu_coroutine_enter(co);
> +}
> +
> /* Update the throttle configuration for a particular group. Similar
> * to throttle_config(), but guarantees atomicity within the
> * throttling group.
> @@ -376,8 +427,8 @@ void throttle_group_config(BlockBackend *blk, ThrottleConfig *cfg)
> throttle_config(ts, tt, cfg);
> qemu_mutex_unlock(&tg->lock);
>
> - qemu_co_enter_next(&blkp->throttled_reqs[0]);
> - qemu_co_enter_next(&blkp->throttled_reqs[1]);
> + throttle_group_restart_queue(blk, 0);
> + throttle_group_restart_queue(blk, 1);
> }
>
> /* Get the throttle configuration from a particular group. Similar to
> @@ -417,7 +468,7 @@ static void timer_cb(BlockBackend *blk, bool is_write)
>
> /* Run the request that was waiting for this timer */
> aio_context_acquire(blk_get_aio_context(blk));
> - empty_queue = !qemu_co_enter_next(&blkp->throttled_reqs[is_write]);
> + empty_queue = !throttle_group_restart_queue(blk, is_write);
> aio_context_release(blk_get_aio_context(blk));
>
> /* If the request queue was empty then we have to take care of
> --
> 2.9.3
>
>
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 08/17] throttle-groups: protect throttled requests with a CoMutex
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (6 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 07/17] throttle-groups: do not use qemu_co_enter_next Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 6:57 ` Fam Zheng
2017-04-20 12:00 ` [Qemu-devel] [PATCH 09/17] util: add stats64 module Paolo Bonzini
` (13 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/block-backend.c | 1 +
block/throttle-groups.c | 11 +++++++++--
include/sysemu/block-backend.h | 7 ++-----
3 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/block/block-backend.c b/block/block-backend.c
index 915ccc5..a37d74d 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -163,6 +163,7 @@ BlockBackend *blk_new(uint64_t perm, uint64_t shared_perm)
blk->shared_perm = shared_perm;
blk_set_enable_write_cache(blk, true);
+ qemu_co_mutex_init(&blk->public.reqs_lock);
qemu_co_queue_init(&blk->public.throttled_reqs[0]);
qemu_co_queue_init(&blk->public.throttled_reqs[1]);
diff --git a/block/throttle-groups.c b/block/throttle-groups.c
index d66bf62..695d28d 100644
--- a/block/throttle-groups.c
+++ b/block/throttle-groups.c
@@ -268,8 +268,13 @@ static bool throttle_group_schedule_timer(BlockBackend *blk, bool is_write)
static bool throttle_group_co_restart_queue(BlockBackend *blk, bool is_write)
{
BlockBackendPublic *blkp = blk_get_public(blk);
+ bool ret;
- return qemu_co_queue_next(&blkp->throttled_reqs[is_write]);
+ qemu_co_mutex_lock(&blkp->reqs_lock);
+ ret = qemu_co_queue_next(&blkp->throttled_reqs[is_write]);
+ qemu_co_mutex_unlock(&blkp->reqs_lock);
+
+ return ret;
}
/* Look for the next pending I/O request and schedule it.
@@ -338,7 +343,9 @@ void coroutine_fn throttle_group_co_io_limits_intercept(BlockBackend *blk,
if (must_wait || blkp->pending_reqs[is_write]) {
blkp->pending_reqs[is_write]++;
qemu_mutex_unlock(&tg->lock);
- qemu_co_queue_wait(&blkp->throttled_reqs[is_write], NULL);
+ qemu_co_mutex_lock(&blkp->reqs_lock);
+ qemu_co_queue_wait(&blkp->throttled_reqs[is_write], &blkp->reqs_lock);
+ qemu_co_mutex_unlock(&blkp->reqs_lock);
qemu_mutex_lock(&tg->lock);
blkp->pending_reqs[is_write]--;
}
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index 87a43b0..e9529fb 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -72,11 +72,8 @@ typedef struct BlockDevOps {
* fields that must be public. This is in particular for QLIST_ENTRY() and
* friends so that BlockBackends can be kept in lists outside block-backend.c */
typedef struct BlockBackendPublic {
- /* I/O throttling has its own locking, but also some fields are
- * protected by the AioContext lock.
- */
-
- /* Protected by AioContext lock. */
+ /* reqs_lock protects the CoQueues for throttled requests. */
+ CoMutex reqs_lock;
CoQueue throttled_reqs[2];
/* Nonzero if the I/O limits are currently being ignored; generally
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 08/17] throttle-groups: protect throttled requests with a CoMutex
2017-04-20 12:00 ` [Qemu-devel] [PATCH 08/17] throttle-groups: protect throttled requests with a CoMutex Paolo Bonzini
@ 2017-05-04 6:57 ` Fam Zheng
2017-05-04 13:56 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
0 siblings, 1 reply; 65+ messages in thread
From: Fam Zheng @ 2017-05-04 6:57 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, 04/20 14:00, Paolo Bonzini wrote:
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> block/block-backend.c | 1 +
> block/throttle-groups.c | 11 +++++++++--
> include/sysemu/block-backend.h | 7 ++-----
> 3 files changed, 12 insertions(+), 7 deletions(-)
>
> diff --git a/block/block-backend.c b/block/block-backend.c
> index 915ccc5..a37d74d 100644
> --- a/block/block-backend.c
> +++ b/block/block-backend.c
> @@ -163,6 +163,7 @@ BlockBackend *blk_new(uint64_t perm, uint64_t shared_perm)
> blk->shared_perm = shared_perm;
> blk_set_enable_write_cache(blk, true);
>
> + qemu_co_mutex_init(&blk->public.reqs_lock);
> qemu_co_queue_init(&blk->public.throttled_reqs[0]);
> qemu_co_queue_init(&blk->public.throttled_reqs[1]);
>
> diff --git a/block/throttle-groups.c b/block/throttle-groups.c
> index d66bf62..695d28d 100644
> --- a/block/throttle-groups.c
> +++ b/block/throttle-groups.c
> @@ -268,8 +268,13 @@ static bool throttle_group_schedule_timer(BlockBackend *blk, bool is_write)
> static bool throttle_group_co_restart_queue(BlockBackend *blk, bool is_write)
> {
> BlockBackendPublic *blkp = blk_get_public(blk);
> + bool ret;
>
> - return qemu_co_queue_next(&blkp->throttled_reqs[is_write]);
> + qemu_co_mutex_lock(&blkp->reqs_lock);
> + ret = qemu_co_queue_next(&blkp->throttled_reqs[is_write]);
> + qemu_co_mutex_unlock(&blkp->reqs_lock);
> +
> + return ret;
> }
>
> /* Look for the next pending I/O request and schedule it.
> @@ -338,7 +343,9 @@ void coroutine_fn throttle_group_co_io_limits_intercept(BlockBackend *blk,
> if (must_wait || blkp->pending_reqs[is_write]) {
> blkp->pending_reqs[is_write]++;
> qemu_mutex_unlock(&tg->lock);
> - qemu_co_queue_wait(&blkp->throttled_reqs[is_write], NULL);
> + qemu_co_mutex_lock(&blkp->reqs_lock);
> + qemu_co_queue_wait(&blkp->throttled_reqs[is_write], &blkp->reqs_lock);
> + qemu_co_mutex_unlock(&blkp->reqs_lock);
> qemu_mutex_lock(&tg->lock);
> blkp->pending_reqs[is_write]--;
> }
> diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
> index 87a43b0..e9529fb 100644
> --- a/include/sysemu/block-backend.h
> +++ b/include/sysemu/block-backend.h
> @@ -72,11 +72,8 @@ typedef struct BlockDevOps {
> * fields that must be public. This is in particular for QLIST_ENTRY() and
> * friends so that BlockBackends can be kept in lists outside block-backend.c */
> typedef struct BlockBackendPublic {
> - /* I/O throttling has its own locking, but also some fields are
> - * protected by the AioContext lock.
> - */
> -
> - /* Protected by AioContext lock. */
> + /* reqs_lock protects the CoQueues for throttled requests. */
> + CoMutex reqs_lock;
Bikeshedding: will it be more readable if it's called throttled_reqs_lock or
throttled_lock?
Fam
> CoQueue throttled_reqs[2];
>
> /* Nonzero if the I/O limits are currently being ignored; generally
> --
> 2.9.3
>
>
>
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH 08/17] throttle-groups: protect throttled requests with a CoMutex
2017-05-04 6:57 ` Fam Zheng
@ 2017-05-04 13:56 ` Stefan Hajnoczi
0 siblings, 0 replies; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-04 13:56 UTC (permalink / raw)
To: Fam Zheng; +Cc: Paolo Bonzini, qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 994 bytes --]
On Thu, May 04, 2017 at 02:57:48PM +0800, Fam Zheng wrote:
> On Thu, 04/20 14:00, Paolo Bonzini wrote:
> > diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
> > index 87a43b0..e9529fb 100644
> > --- a/include/sysemu/block-backend.h
> > +++ b/include/sysemu/block-backend.h
> > @@ -72,11 +72,8 @@ typedef struct BlockDevOps {
> > * fields that must be public. This is in particular for QLIST_ENTRY() and
> > * friends so that BlockBackends can be kept in lists outside block-backend.c */
> > typedef struct BlockBackendPublic {
> > - /* I/O throttling has its own locking, but also some fields are
> > - * protected by the AioContext lock.
> > - */
> > -
> > - /* Protected by AioContext lock. */
> > + /* reqs_lock protects the CoQueues for throttled requests. */
> > + CoMutex reqs_lock;
>
> Bikeshedding: will it be more readable if it's called throttled_reqs_lock or
> throttled_lock?
Yes, please.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 09/17] util: add stats64 module
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (7 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 08/17] throttle-groups: protect throttled requests with a CoMutex Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 7:19 ` Fam Zheng
` (2 more replies)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 10/17] block: use Stat64 for wr_highest_offset Paolo Bonzini
` (12 subsequent siblings)
21 siblings, 3 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
This module provides fast paths for 64-bit atomic operations on machines
that only have 32-bit atomic access.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
include/qemu/stats64.h | 210 +++++++++++++++++++++++++++++++++++++++++++++++++
util/Makefile.objs | 1 +
util/stats64.c | 135 ++++++++++++++++++++++++++++++++
3 files changed, 346 insertions(+)
create mode 100644 include/qemu/stats64.h
create mode 100644 util/stats64.c
diff --git a/include/qemu/stats64.h b/include/qemu/stats64.h
new file mode 100644
index 0000000..70963f4
--- /dev/null
+++ b/include/qemu/stats64.h
@@ -0,0 +1,210 @@
+/*
+ * Atomic operations on 64-bit quantities.
+ *
+ * Copyright (C) 2017 Red Hat, Inc.
+ *
+ * Author: Paolo Bonzini <pbonzini@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_STATS64_H
+#define QEMU_STATS64_H 1
+
+#include "qemu/atomic.h"
+
+/* FIXME: i386 doesn't need the spinlock. Are there any others? */
+#if __SIZEOF_LONG__ < 8
+#define STAT64_NEED_SPINLOCK 1
+#endif
+
+/* This provides atomic operations on 64-bit type, using a reader-writer
+ * spinlock on architectures that do not have 64-bit accesses. However
+ * it tries hard not to take the lock.
+ */
+
+typedef struct Stat64 {
+#ifdef STAT64_NEED_SPINLOCK
+ uint32_t low, high;
+ uint32_t lock;
+#else
+ uint64_t value;
+#endif
+} Stat64;
+
+#ifndef STAT64_NEED_SPINLOCK
+static inline void stat64_init(Stat64 *s, uint64_t value)
+{
+ /* This is not guaranteed to be atomic! */
+ *s = (Stat64) { value };
+}
+
+static inline uint64_t stat64_get(const Stat64 *s)
+{
+ return atomic_read(&s->value);
+}
+
+static inline void stat64_add(Stat64 *s, uint64_t value)
+{
+ atomic_add(&s->value, value);
+}
+
+static inline void stat64_min(Stat64 *s, uint32_t value)
+{
+ for (;;) {
+ uint64_t orig = atomic_read(&s->value);
+ if (orig <= value) {
+ break;
+ }
+ orig = atomic_cmpxchg(&s->value, orig, value);
+ if (orig <= value) {
+ break;
+ }
+ }
+}
+
+static inline void stat64_max(Stat64 *s, uint32_t value)
+{
+ for (;;) {
+ uint64_t orig = atomic_read(&s->value);
+ if (orig >= value) {
+ break;
+ }
+ orig = atomic_cmpxchg(&s->value, orig, value);
+ if (orig >= value) {
+ break;
+ }
+ }
+}
+#else
+uint64_t stat64_get(const Stat64 *s);
+bool stat64_min_slow(Stat64 *s, uint64_t value);
+bool stat64_max_slow(Stat64 *s, uint64_t value);
+bool stat64_add32_carry(Stat64 *s, uint32_t low, uint32_t high);
+
+static inline void stat64_init(Stat64 *s, uint64_t value)
+{
+ /* This is not guaranteed to be atomic! */
+ *s = (Stat64) { .low = value, .high = value >> 32, .lock = 0 };
+}
+
+static inline void stat64_add(Stat64 *s, uint64_t value)
+{
+ uint32_t low, high;
+ high = value >> 32;
+ low = (uint32_t) value;
+ if (!low) {
+ if (high) {
+ atomic_add(&s->high, high);
+ }
+ return;
+ }
+
+ for (;;) {
+ uint32_t orig = s->low;
+ uint32_t result = orig + low;
+ uint32_t old;
+
+ if (result < low || high) {
+ /* If the high part is affected, take the lock. */
+ if (stat64_add32_carry(s, low, high)) {
+ return;
+ }
+ continue;
+ }
+
+ /* No carry, try with a 32-bit cmpxchg. The result is independent of
+ * the high 32 bits, so it can race just fine with stat64_add32_carry
+ * and even stat64_get!
+ */
+ old = atomic_cmpxchg(&s->low, orig, result);
+ if (orig == old) {
+ return;
+ }
+ }
+}
+
+static inline void stat64_min(Stat64 *s, uint64_t value)
+{
+ uint32_t low, high;
+ uint32_t orig_low, orig_high;
+
+ high = value >> 32;
+ low = (uint32_t) value;
+ do {
+ orig_high = atomic_read(&s->high);
+ if (orig_high < high) {
+ return;
+ }
+
+ if (orig_high == high) {
+ /* High 32 bits are equal. Read low after high, otherwise we
+ * can get a false positive (e.g. 0x1235,0x0000 changes to
+ * 0x1234,0x8000 and we read it as 0x1234,0x0000). Pairs with
+ * the write barrier in stat64_min_slow.
+ */
+ smp_rmb();
+ orig_low = atomic_read(&s->low);
+ if (orig_low <= low) {
+ return;
+ }
+
+ /* See if we were lucky and a writer raced against us. The
+ * barrier is theoretically unnecessary, but if we remove it
+ * we may miss being lucky.
+ */
+ smp_rmb();
+ orig_high = atomic_read(&s->high);
+ if (orig_high < high) {
+ return;
+ }
+ }
+
+ /* If the value changes in any way, we have to take the lock. */
+ } while (!stat64_min_slow(s, value));
+}
+
+static inline void stat64_max(Stat64 *s, uint64_t value)
+{
+ uint32_t low, high;
+ uint32_t orig_low, orig_high;
+
+ high = value >> 32;
+ low = (uint32_t) value;
+ do {
+ orig_high = atomic_read(&s->high);
+ if (orig_high > high) {
+ return;
+ }
+
+ if (orig_high == high) {
+ /* High 32 bits are equal. Read low after high, otherwise we
+ * can get a false positive (e.g. 0x1234,0x8000 changes to
+ * 0x1235,0x0000 and we read it as 0x1235,0x8000). Pairs with
+ * the write barrier in stat64_max_slow.
+ */
+ smp_rmb();
+ orig_low = atomic_read(&s->low);
+ if (orig_low >= low) {
+ return;
+ }
+
+ /* See if we were lucky and a writer raced against us. The
+ * barrier is theoretically unnecessary, but if we remove it
+ * we may miss being lucky.
+ */
+ smp_rmb();
+ orig_high = atomic_read(&s->high);
+ if (orig_high > high) {
+ return;
+ }
+ }
+
+ /* If the value changes in any way, we have to take the lock. */
+ } while (!stat64_max_slow(s, value));
+}
+
+#endif
+
+#endif
diff --git a/util/Makefile.objs b/util/Makefile.objs
index c6205eb..8a333d3 100644
--- a/util/Makefile.objs
+++ b/util/Makefile.objs
@@ -42,4 +42,5 @@ util-obj-y += log.o
util-obj-y += qdist.o
util-obj-y += qht.o
util-obj-y += range.o
+util-obj-y += stats64.o
util-obj-y += systemd.o
diff --git a/util/stats64.c b/util/stats64.c
new file mode 100644
index 0000000..b9238d7
--- /dev/null
+++ b/util/stats64.c
@@ -0,0 +1,135 @@
+/*
+ * Atomic operations on 64-bit quantities.
+ *
+ * Copyright (C) 2017 Red Hat, Inc.
+ *
+ * Author: Paolo Bonzini <pbonzini@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/atomic.h"
+#include "qemu/stats64.h"
+
+#ifdef STAT64_NEED_SPINLOCK
+static inline void stat64_rdlock(Stat64 *s)
+{
+ /* Keep out incoming writers to avoid them starving us. */
+ atomic_add(&s->lock, 2);
+
+ /* If there is a concurrent writer, wait for it. */
+ while (atomic_read(&s->lock) & 1) {
+ g_usleep(5);
+ }
+}
+
+static inline void stat64_rdunlock(Stat64 *s)
+{
+ atomic_sub(&s->lock, 2);
+}
+
+static inline bool stat64_wrtrylock(Stat64 *s)
+{
+ return atomic_cmpxchg(&s->lock, 0, 1) == 0;
+}
+
+static inline void stat64_wrunlock(Stat64 *s)
+{
+ atomic_dec(&s->lock);
+}
+
+uint64_t stat64_get(const Stat64 *s)
+{
+ uint32_t high, low;
+
+ stat64_rdlock((Stat64 *)s);
+
+ /* 64-bit writes always take the lock, so we can read in
+ * any order.
+ */
+ high = atomic_read(&s->high);
+ low = atomic_read(&s->low);
+ stat64_rdunlock((Stat64 *)s);
+
+ return ((uint64_t)high << 32) | low;
+}
+
+bool stat64_add32_carry(Stat64 *s, uint32_t low, uint32_t high)
+{
+ uint32_t old;
+
+ if (!stat64_wrtrylock(s)) {
+ return false;
+ }
+
+ /* 64-bit reads always take the lock, so they don't care about the
+ * order of our update. By updating s->low first, we can check
+ * whether we have to carry into s->high.
+ */
+ old = atomic_fetch_add(&s->low, value);
+ high += (old + value < old);
+ atomic_add(&s->high, high);
+ stat64_wrunlock(s);
+ return true;
+}
+
+bool stat64_min_slow(Stat64 *s, uint64_t value)
+{
+ uint32_t high, low;
+ uint64_t orig;
+
+ if (!stat64_wrtrylock(s)) {
+ return false;
+ }
+
+ high = atomic_read(&s->high);
+ low = atomic_read(&s->low);
+
+ orig = ((uint64_t)high << 32) | low;
+ if (orig < value) {
+ /* The value may become higher temporarily, but stat64_get does not
+ * notice (it takes the lock) and the only effect on stat64_min is
+ * that the slow path may be triggered unnecessarily.
+ *
+ * But, we have to set low before high, just like stat64_min reads
+ * high before low.
+ */
+ atomic_set(&s->low, (uint32_t)value);
+ smp_wmb();
+ atomic_set(&s->high, value >> 32);
+ }
+ stat64_wrunlock(s);
+ return true;
+}
+
+bool stat64_max_slow(Stat64 *s, uint64_t value)
+{
+ uint32_t high, low;
+ uint64_t orig;
+
+ if (!stat64_wrtrylock(s)) {
+ return false;
+ }
+
+ high = atomic_read(&s->high);
+ low = atomic_read(&s->low);
+
+ orig = ((uint64_t)high << 32) | low;
+ if (orig > value) {
+ /* The value may become lower temporarily, but stat64_get does not
+ * notice (it takes the lock) and the only effect on stat64_max is
+ * that the slow path may be triggered unnecessarily.
+ *
+ * But, we have to set low before high, just like stat64_max reads
+ * high before low.
+ */
+ atomic_set(&s->low, (uint32_t)value);
+ smp_wmb();
+ atomic_set(&s->high, value >> 32);
+ }
+ stat64_wrunlock(s);
+ return true;
+}
+#endif
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 09/17] util: add stats64 module
2017-04-20 12:00 ` [Qemu-devel] [PATCH 09/17] util: add stats64 module Paolo Bonzini
@ 2017-05-04 7:19 ` Fam Zheng
2017-05-04 7:24 ` Paolo Bonzini
2017-05-04 7:36 ` Fam Zheng
2017-05-04 8:55 ` [Qemu-devel] [Qemu-block] " Roman Kagan
2 siblings, 1 reply; 65+ messages in thread
From: Fam Zheng @ 2017-05-04 7:19 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, 04/20 14:00, Paolo Bonzini wrote:
> This module provides fast paths for 64-bit atomic operations on machines
> that only have 32-bit atomic access.
Interesting patch!
Out of curiosity: what are the machines here, besides i386? It strikes me as
they are old and slow anyway, then why this optimization?
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> include/qemu/stats64.h | 210 +++++++++++++++++++++++++++++++++++++++++++++++++
> util/Makefile.objs | 1 +
> util/stats64.c | 135 ++++++++++++++++++++++++++++++++
> 3 files changed, 346 insertions(+)
> create mode 100644 include/qemu/stats64.h
> create mode 100644 util/stats64.c
>
> diff --git a/include/qemu/stats64.h b/include/qemu/stats64.h
> new file mode 100644
> index 0000000..70963f4
> --- /dev/null
> +++ b/include/qemu/stats64.h
> @@ -0,0 +1,210 @@
> +/*
> + * Atomic operations on 64-bit quantities.
> + *
> + * Copyright (C) 2017 Red Hat, Inc.
> + *
> + * Author: Paolo Bonzini <pbonzini@redhat.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#ifndef QEMU_STATS64_H
> +#define QEMU_STATS64_H 1
> +
> +#include "qemu/atomic.h"
Include qemu/osdep.h first, to honor scripts/clean-includes?
> +
> +/* FIXME: i386 doesn't need the spinlock. Are there any others? */
> +#if __SIZEOF_LONG__ < 8
> +#define STAT64_NEED_SPINLOCK 1
> +#endif
> +
> +/* This provides atomic operations on 64-bit type, using a reader-writer
> + * spinlock on architectures that do not have 64-bit accesses. However
> + * it tries hard not to take the lock.
> + */
> +
> +typedef struct Stat64 {
> +#ifdef STAT64_NEED_SPINLOCK
> + uint32_t low, high;
> + uint32_t lock;
> +#else
> + uint64_t value;
> +#endif
> +} Stat64;
> +
> +#ifndef STAT64_NEED_SPINLOCK
> +static inline void stat64_init(Stat64 *s, uint64_t value)
> +{
> + /* This is not guaranteed to be atomic! */
> + *s = (Stat64) { value };
> +}
> +
> +static inline uint64_t stat64_get(const Stat64 *s)
> +{
> + return atomic_read(&s->value);
> +}
> +
> +static inline void stat64_add(Stat64 *s, uint64_t value)
> +{
> + atomic_add(&s->value, value);
> +}
> +
> +static inline void stat64_min(Stat64 *s, uint32_t value)
> +{
> + for (;;) {
> + uint64_t orig = atomic_read(&s->value);
> + if (orig <= value) {
> + break;
> + }
> + orig = atomic_cmpxchg(&s->value, orig, value);
> + if (orig <= value) {
> + break;
> + }
> + }
> +}
> +
> +static inline void stat64_max(Stat64 *s, uint32_t value)
> +{
> + for (;;) {
> + uint64_t orig = atomic_read(&s->value);
> + if (orig >= value) {
> + break;
> + }
> + orig = atomic_cmpxchg(&s->value, orig, value);
> + if (orig >= value) {
> + break;
> + }
> + }
> +}
> +#else
> +uint64_t stat64_get(const Stat64 *s);
> +bool stat64_min_slow(Stat64 *s, uint64_t value);
> +bool stat64_max_slow(Stat64 *s, uint64_t value);
> +bool stat64_add32_carry(Stat64 *s, uint32_t low, uint32_t high);
> +
> +static inline void stat64_init(Stat64 *s, uint64_t value)
> +{
> + /* This is not guaranteed to be atomic! */
> + *s = (Stat64) { .low = value, .high = value >> 32, .lock = 0 };
> +}
> +
> +static inline void stat64_add(Stat64 *s, uint64_t value)
> +{
> + uint32_t low, high;
> + high = value >> 32;
> + low = (uint32_t) value;
> + if (!low) {
> + if (high) {
> + atomic_add(&s->high, high);
> + }
> + return;
> + }
> +
> + for (;;) {
> + uint32_t orig = s->low;
> + uint32_t result = orig + low;
> + uint32_t old;
> +
> + if (result < low || high) {
> + /* If the high part is affected, take the lock. */
> + if (stat64_add32_carry(s, low, high)) {
> + return;
> + }
> + continue;
> + }
> +
> + /* No carry, try with a 32-bit cmpxchg. The result is independent of
> + * the high 32 bits, so it can race just fine with stat64_add32_carry
> + * and even stat64_get!
> + */
> + old = atomic_cmpxchg(&s->low, orig, result);
> + if (orig == old) {
> + return;
> + }
> + }
> +}
> +
> +static inline void stat64_min(Stat64 *s, uint64_t value)
> +{
> + uint32_t low, high;
> + uint32_t orig_low, orig_high;
> +
> + high = value >> 32;
> + low = (uint32_t) value;
> + do {
> + orig_high = atomic_read(&s->high);
> + if (orig_high < high) {
> + return;
> + }
> +
> + if (orig_high == high) {
> + /* High 32 bits are equal. Read low after high, otherwise we
> + * can get a false positive (e.g. 0x1235,0x0000 changes to
> + * 0x1234,0x8000 and we read it as 0x1234,0x0000). Pairs with
> + * the write barrier in stat64_min_slow.
> + */
> + smp_rmb();
> + orig_low = atomic_read(&s->low);
> + if (orig_low <= low) {
> + return;
> + }
> +
> + /* See if we were lucky and a writer raced against us. The
> + * barrier is theoretically unnecessary, but if we remove it
> + * we may miss being lucky.
> + */
> + smp_rmb();
> + orig_high = atomic_read(&s->high);
> + if (orig_high < high) {
> + return;
> + }
> + }
> +
> + /* If the value changes in any way, we have to take the lock. */
> + } while (!stat64_min_slow(s, value));
> +}
> +
> +static inline void stat64_max(Stat64 *s, uint64_t value)
> +{
> + uint32_t low, high;
> + uint32_t orig_low, orig_high;
> +
> + high = value >> 32;
> + low = (uint32_t) value;
> + do {
> + orig_high = atomic_read(&s->high);
> + if (orig_high > high) {
> + return;
> + }
> +
> + if (orig_high == high) {
> + /* High 32 bits are equal. Read low after high, otherwise we
> + * can get a false positive (e.g. 0x1234,0x8000 changes to
> + * 0x1235,0x0000 and we read it as 0x1235,0x8000). Pairs with
> + * the write barrier in stat64_max_slow.
> + */
> + smp_rmb();
> + orig_low = atomic_read(&s->low);
> + if (orig_low >= low) {
> + return;
> + }
> +
> + /* See if we were lucky and a writer raced against us. The
> + * barrier is theoretically unnecessary, but if we remove it
> + * we may miss being lucky.
> + */
> + smp_rmb();
> + orig_high = atomic_read(&s->high);
> + if (orig_high > high) {
> + return;
> + }
> + }
> +
> + /* If the value changes in any way, we have to take the lock. */
> + } while (!stat64_max_slow(s, value));
> +}
> +
> +#endif
> +
> +#endif
> diff --git a/util/Makefile.objs b/util/Makefile.objs
> index c6205eb..8a333d3 100644
> --- a/util/Makefile.objs
> +++ b/util/Makefile.objs
> @@ -42,4 +42,5 @@ util-obj-y += log.o
> util-obj-y += qdist.o
> util-obj-y += qht.o
> util-obj-y += range.o
> +util-obj-y += stats64.o
> util-obj-y += systemd.o
> diff --git a/util/stats64.c b/util/stats64.c
> new file mode 100644
> index 0000000..b9238d7
> --- /dev/null
> +++ b/util/stats64.c
> @@ -0,0 +1,135 @@
> +/*
> + * Atomic operations on 64-bit quantities.
> + *
> + * Copyright (C) 2017 Red Hat, Inc.
> + *
> + * Author: Paolo Bonzini <pbonzini@redhat.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu/atomic.h"
> +#include "qemu/stats64.h"
> +
> +#ifdef STAT64_NEED_SPINLOCK
> +static inline void stat64_rdlock(Stat64 *s)
> +{
> + /* Keep out incoming writers to avoid them starving us. */
> + atomic_add(&s->lock, 2);
> +
> + /* If there is a concurrent writer, wait for it. */
> + while (atomic_read(&s->lock) & 1) {
> + g_usleep(5);
> + }
> +}
> +
> +static inline void stat64_rdunlock(Stat64 *s)
> +{
> + atomic_sub(&s->lock, 2);
> +}
> +
> +static inline bool stat64_wrtrylock(Stat64 *s)
> +{
> + return atomic_cmpxchg(&s->lock, 0, 1) == 0;
> +}
> +
> +static inline void stat64_wrunlock(Stat64 *s)
> +{
> + atomic_dec(&s->lock);
> +}
> +
> +uint64_t stat64_get(const Stat64 *s)
> +{
> + uint32_t high, low;
> +
> + stat64_rdlock((Stat64 *)s);
> +
> + /* 64-bit writes always take the lock, so we can read in
> + * any order.
> + */
> + high = atomic_read(&s->high);
> + low = atomic_read(&s->low);
> + stat64_rdunlock((Stat64 *)s);
> +
> + return ((uint64_t)high << 32) | low;
> +}
> +
> +bool stat64_add32_carry(Stat64 *s, uint32_t low, uint32_t high)
Maybe add "try" in the name too, for this, and the two below?
> +{
> + uint32_t old;
> +
> + if (!stat64_wrtrylock(s)) {
> + return false;
> + }
> +
> + /* 64-bit reads always take the lock, so they don't care about the
> + * order of our update. By updating s->low first, we can check
> + * whether we have to carry into s->high.
> + */
> + old = atomic_fetch_add(&s->low, value);
> + high += (old + value < old);
> + atomic_add(&s->high, high);
> + stat64_wrunlock(s);
> + return true;
> +}
> +
> +bool stat64_min_slow(Stat64 *s, uint64_t value)
> +{
> + uint32_t high, low;
> + uint64_t orig;
> +
> + if (!stat64_wrtrylock(s)) {
> + return false;
> + }
> +
> + high = atomic_read(&s->high);
> + low = atomic_read(&s->low);
> +
> + orig = ((uint64_t)high << 32) | low;
> + if (orig < value) {
> + /* The value may become higher temporarily, but stat64_get does not
> + * notice (it takes the lock) and the only effect on stat64_min is
> + * that the slow path may be triggered unnecessarily.
> + *
> + * But, we have to set low before high, just like stat64_min reads
> + * high before low.
> + */
> + atomic_set(&s->low, (uint32_t)value);
> + smp_wmb();
> + atomic_set(&s->high, value >> 32);
> + }
> + stat64_wrunlock(s);
> + return true;
> +}
> +
> +bool stat64_max_slow(Stat64 *s, uint64_t value)
> +{
> + uint32_t high, low;
> + uint64_t orig;
> +
> + if (!stat64_wrtrylock(s)) {
> + return false;
> + }
> +
> + high = atomic_read(&s->high);
> + low = atomic_read(&s->low);
> +
> + orig = ((uint64_t)high << 32) | low;
> + if (orig > value) {
> + /* The value may become lower temporarily, but stat64_get does not
> + * notice (it takes the lock) and the only effect on stat64_max is
> + * that the slow path may be triggered unnecessarily.
> + *
> + * But, we have to set low before high, just like stat64_max reads
> + * high before low.
> + */
> + atomic_set(&s->low, (uint32_t)value);
> + smp_wmb();
> + atomic_set(&s->high, value >> 32);
> + }
> + stat64_wrunlock(s);
> + return true;
> +}
> +#endif
> --
> 2.9.3
>
>
>
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 09/17] util: add stats64 module
2017-05-04 7:19 ` Fam Zheng
@ 2017-05-04 7:24 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 7:24 UTC (permalink / raw)
To: Fam Zheng; +Cc: qemu-devel, qemu-block
On 04/05/2017 09:19, Fam Zheng wrote:
> On Thu, 04/20 14:00, Paolo Bonzini wrote:
>> This module provides fast paths for 64-bit atomic operations on machines
>> that only have 32-bit atomic access.
>
> Interesting patch!
>
> Out of curiosity: what are the machines here, besides i386? It strikes me as
> they are old and slow anyway, then why this optimization?
I guess ARM too. I figured it might be useful for TCG too in the future
and it was fun to write. :)
Paolo
>
>>
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> ---
>> include/qemu/stats64.h | 210 +++++++++++++++++++++++++++++++++++++++++++++++++
>> util/Makefile.objs | 1 +
>> util/stats64.c | 135 ++++++++++++++++++++++++++++++++
>> 3 files changed, 346 insertions(+)
>> create mode 100644 include/qemu/stats64.h
>> create mode 100644 util/stats64.c
>>
>> diff --git a/include/qemu/stats64.h b/include/qemu/stats64.h
>> new file mode 100644
>> index 0000000..70963f4
>> --- /dev/null
>> +++ b/include/qemu/stats64.h
>> @@ -0,0 +1,210 @@
>> +/*
>> + * Atomic operations on 64-bit quantities.
>> + *
>> + * Copyright (C) 2017 Red Hat, Inc.
>> + *
>> + * Author: Paolo Bonzini <pbonzini@redhat.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + */
>> +
>> +#ifndef QEMU_STATS64_H
>> +#define QEMU_STATS64_H 1
>> +
>> +#include "qemu/atomic.h"
>
> Include qemu/osdep.h first, to honor scripts/clean-includes?
>
>> +
>> +/* FIXME: i386 doesn't need the spinlock. Are there any others? */
>> +#if __SIZEOF_LONG__ < 8
>> +#define STAT64_NEED_SPINLOCK 1
>> +#endif
>> +
>> +/* This provides atomic operations on 64-bit type, using a reader-writer
>> + * spinlock on architectures that do not have 64-bit accesses. However
>> + * it tries hard not to take the lock.
>> + */
>> +
>> +typedef struct Stat64 {
>> +#ifdef STAT64_NEED_SPINLOCK
>> + uint32_t low, high;
>> + uint32_t lock;
>> +#else
>> + uint64_t value;
>> +#endif
>> +} Stat64;
>> +
>> +#ifndef STAT64_NEED_SPINLOCK
>> +static inline void stat64_init(Stat64 *s, uint64_t value)
>> +{
>> + /* This is not guaranteed to be atomic! */
>> + *s = (Stat64) { value };
>> +}
>> +
>> +static inline uint64_t stat64_get(const Stat64 *s)
>> +{
>> + return atomic_read(&s->value);
>> +}
>> +
>> +static inline void stat64_add(Stat64 *s, uint64_t value)
>> +{
>> + atomic_add(&s->value, value);
>> +}
>> +
>> +static inline void stat64_min(Stat64 *s, uint32_t value)
>> +{
>> + for (;;) {
>> + uint64_t orig = atomic_read(&s->value);
>> + if (orig <= value) {
>> + break;
>> + }
>> + orig = atomic_cmpxchg(&s->value, orig, value);
>> + if (orig <= value) {
>> + break;
>> + }
>> + }
>> +}
>> +
>> +static inline void stat64_max(Stat64 *s, uint32_t value)
>> +{
>> + for (;;) {
>> + uint64_t orig = atomic_read(&s->value);
>> + if (orig >= value) {
>> + break;
>> + }
>> + orig = atomic_cmpxchg(&s->value, orig, value);
>> + if (orig >= value) {
>> + break;
>> + }
>> + }
>> +}
>> +#else
>> +uint64_t stat64_get(const Stat64 *s);
>> +bool stat64_min_slow(Stat64 *s, uint64_t value);
>> +bool stat64_max_slow(Stat64 *s, uint64_t value);
>> +bool stat64_add32_carry(Stat64 *s, uint32_t low, uint32_t high);
>> +
>> +static inline void stat64_init(Stat64 *s, uint64_t value)
>> +{
>> + /* This is not guaranteed to be atomic! */
>> + *s = (Stat64) { .low = value, .high = value >> 32, .lock = 0 };
>> +}
>> +
>> +static inline void stat64_add(Stat64 *s, uint64_t value)
>> +{
>> + uint32_t low, high;
>> + high = value >> 32;
>> + low = (uint32_t) value;
>> + if (!low) {
>> + if (high) {
>> + atomic_add(&s->high, high);
>> + }
>> + return;
>> + }
>> +
>> + for (;;) {
>> + uint32_t orig = s->low;
>> + uint32_t result = orig + low;
>> + uint32_t old;
>> +
>> + if (result < low || high) {
>> + /* If the high part is affected, take the lock. */
>> + if (stat64_add32_carry(s, low, high)) {
>> + return;
>> + }
>> + continue;
>> + }
>> +
>> + /* No carry, try with a 32-bit cmpxchg. The result is independent of
>> + * the high 32 bits, so it can race just fine with stat64_add32_carry
>> + * and even stat64_get!
>> + */
>> + old = atomic_cmpxchg(&s->low, orig, result);
>> + if (orig == old) {
>> + return;
>> + }
>> + }
>> +}
>> +
>> +static inline void stat64_min(Stat64 *s, uint64_t value)
>> +{
>> + uint32_t low, high;
>> + uint32_t orig_low, orig_high;
>> +
>> + high = value >> 32;
>> + low = (uint32_t) value;
>> + do {
>> + orig_high = atomic_read(&s->high);
>> + if (orig_high < high) {
>> + return;
>> + }
>> +
>> + if (orig_high == high) {
>> + /* High 32 bits are equal. Read low after high, otherwise we
>> + * can get a false positive (e.g. 0x1235,0x0000 changes to
>> + * 0x1234,0x8000 and we read it as 0x1234,0x0000). Pairs with
>> + * the write barrier in stat64_min_slow.
>> + */
>> + smp_rmb();
>> + orig_low = atomic_read(&s->low);
>> + if (orig_low <= low) {
>> + return;
>> + }
>> +
>> + /* See if we were lucky and a writer raced against us. The
>> + * barrier is theoretically unnecessary, but if we remove it
>> + * we may miss being lucky.
>> + */
>> + smp_rmb();
>> + orig_high = atomic_read(&s->high);
>> + if (orig_high < high) {
>> + return;
>> + }
>> + }
>> +
>> + /* If the value changes in any way, we have to take the lock. */
>> + } while (!stat64_min_slow(s, value));
>> +}
>> +
>> +static inline void stat64_max(Stat64 *s, uint64_t value)
>> +{
>> + uint32_t low, high;
>> + uint32_t orig_low, orig_high;
>> +
>> + high = value >> 32;
>> + low = (uint32_t) value;
>> + do {
>> + orig_high = atomic_read(&s->high);
>> + if (orig_high > high) {
>> + return;
>> + }
>> +
>> + if (orig_high == high) {
>> + /* High 32 bits are equal. Read low after high, otherwise we
>> + * can get a false positive (e.g. 0x1234,0x8000 changes to
>> + * 0x1235,0x0000 and we read it as 0x1235,0x8000). Pairs with
>> + * the write barrier in stat64_max_slow.
>> + */
>> + smp_rmb();
>> + orig_low = atomic_read(&s->low);
>> + if (orig_low >= low) {
>> + return;
>> + }
>> +
>> + /* See if we were lucky and a writer raced against us. The
>> + * barrier is theoretically unnecessary, but if we remove it
>> + * we may miss being lucky.
>> + */
>> + smp_rmb();
>> + orig_high = atomic_read(&s->high);
>> + if (orig_high > high) {
>> + return;
>> + }
>> + }
>> +
>> + /* If the value changes in any way, we have to take the lock. */
>> + } while (!stat64_max_slow(s, value));
>> +}
>> +
>> +#endif
>> +
>> +#endif
>> diff --git a/util/Makefile.objs b/util/Makefile.objs
>> index c6205eb..8a333d3 100644
>> --- a/util/Makefile.objs
>> +++ b/util/Makefile.objs
>> @@ -42,4 +42,5 @@ util-obj-y += log.o
>> util-obj-y += qdist.o
>> util-obj-y += qht.o
>> util-obj-y += range.o
>> +util-obj-y += stats64.o
>> util-obj-y += systemd.o
>> diff --git a/util/stats64.c b/util/stats64.c
>> new file mode 100644
>> index 0000000..b9238d7
>> --- /dev/null
>> +++ b/util/stats64.c
>> @@ -0,0 +1,135 @@
>> +/*
>> + * Atomic operations on 64-bit quantities.
>> + *
>> + * Copyright (C) 2017 Red Hat, Inc.
>> + *
>> + * Author: Paolo Bonzini <pbonzini@redhat.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
>> + * See the COPYING file in the top-level directory.
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "qemu/atomic.h"
>> +#include "qemu/stats64.h"
>> +
>> +#ifdef STAT64_NEED_SPINLOCK
>> +static inline void stat64_rdlock(Stat64 *s)
>> +{
>> + /* Keep out incoming writers to avoid them starving us. */
>> + atomic_add(&s->lock, 2);
>> +
>> + /* If there is a concurrent writer, wait for it. */
>> + while (atomic_read(&s->lock) & 1) {
>> + g_usleep(5);
>> + }
>> +}
>> +
>> +static inline void stat64_rdunlock(Stat64 *s)
>> +{
>> + atomic_sub(&s->lock, 2);
>> +}
>> +
>> +static inline bool stat64_wrtrylock(Stat64 *s)
>> +{
>> + return atomic_cmpxchg(&s->lock, 0, 1) == 0;
>> +}
>> +
>> +static inline void stat64_wrunlock(Stat64 *s)
>> +{
>> + atomic_dec(&s->lock);
>> +}
>> +
>> +uint64_t stat64_get(const Stat64 *s)
>> +{
>> + uint32_t high, low;
>> +
>> + stat64_rdlock((Stat64 *)s);
>> +
>> + /* 64-bit writes always take the lock, so we can read in
>> + * any order.
>> + */
>> + high = atomic_read(&s->high);
>> + low = atomic_read(&s->low);
>> + stat64_rdunlock((Stat64 *)s);
>> +
>> + return ((uint64_t)high << 32) | low;
>> +}
>> +
>> +bool stat64_add32_carry(Stat64 *s, uint32_t low, uint32_t high)
>
> Maybe add "try" in the name too, for this, and the two below?
>
>> +{
>> + uint32_t old;
>> +
>> + if (!stat64_wrtrylock(s)) {
>> + return false;
>> + }
>> +
>> + /* 64-bit reads always take the lock, so they don't care about the
>> + * order of our update. By updating s->low first, we can check
>> + * whether we have to carry into s->high.
>> + */
>> + old = atomic_fetch_add(&s->low, value);
>> + high += (old + value < old);
>> + atomic_add(&s->high, high);
>> + stat64_wrunlock(s);
>> + return true;
>> +}
>> +
>> +bool stat64_min_slow(Stat64 *s, uint64_t value)
>> +{
>> + uint32_t high, low;
>> + uint64_t orig;
>> +
>> + if (!stat64_wrtrylock(s)) {
>> + return false;
>> + }
>> +
>> + high = atomic_read(&s->high);
>> + low = atomic_read(&s->low);
>> +
>> + orig = ((uint64_t)high << 32) | low;
>> + if (orig < value) {
>> + /* The value may become higher temporarily, but stat64_get does not
>> + * notice (it takes the lock) and the only effect on stat64_min is
>> + * that the slow path may be triggered unnecessarily.
>> + *
>> + * But, we have to set low before high, just like stat64_min reads
>> + * high before low.
>> + */
>> + atomic_set(&s->low, (uint32_t)value);
>> + smp_wmb();
>> + atomic_set(&s->high, value >> 32);
>> + }
>> + stat64_wrunlock(s);
>> + return true;
>> +}
>> +
>> +bool stat64_max_slow(Stat64 *s, uint64_t value)
>> +{
>> + uint32_t high, low;
>> + uint64_t orig;
>> +
>> + if (!stat64_wrtrylock(s)) {
>> + return false;
>> + }
>> +
>> + high = atomic_read(&s->high);
>> + low = atomic_read(&s->low);
>> +
>> + orig = ((uint64_t)high << 32) | low;
>> + if (orig > value) {
>> + /* The value may become lower temporarily, but stat64_get does not
>> + * notice (it takes the lock) and the only effect on stat64_max is
>> + * that the slow path may be triggered unnecessarily.
>> + *
>> + * But, we have to set low before high, just like stat64_max reads
>> + * high before low.
>> + */
>> + atomic_set(&s->low, (uint32_t)value);
>> + smp_wmb();
>> + atomic_set(&s->high, value >> 32);
>> + }
>> + stat64_wrunlock(s);
>> + return true;
>> +}
>> +#endif
>> --
>> 2.9.3
>>
>>
>>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [PATCH 09/17] util: add stats64 module
2017-04-20 12:00 ` [Qemu-devel] [PATCH 09/17] util: add stats64 module Paolo Bonzini
2017-05-04 7:19 ` Fam Zheng
@ 2017-05-04 7:36 ` Fam Zheng
2017-05-04 7:38 ` Paolo Bonzini
2017-05-04 8:55 ` [Qemu-devel] [Qemu-block] " Roman Kagan
2 siblings, 1 reply; 65+ messages in thread
From: Fam Zheng @ 2017-05-04 7:36 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, 04/20 14:00, Paolo Bonzini wrote:
> +static inline void stat64_rdlock(Stat64 *s)
> +{
> + /* Keep out incoming writers to avoid them starving us. */
> + atomic_add(&s->lock, 2);
> +
> + /* If there is a concurrent writer, wait for it. */
> + while (atomic_read(&s->lock) & 1) {
> + g_usleep(5);
What's the difference of g_usleep() from cpu_relax() in
qemu_co_mutex_lock_unlock?
> + }
> +}
Fam
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 09/17] util: add stats64 module
2017-05-04 7:36 ` Fam Zheng
@ 2017-05-04 7:38 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 7:38 UTC (permalink / raw)
To: Fam Zheng; +Cc: qemu-devel, qemu-block
On 04/05/2017 09:36, Fam Zheng wrote:
>> +
>> + /* If there is a concurrent writer, wait for it. */
>> + while (atomic_read(&s->lock) & 1) {
>> + g_usleep(5);
> What's the difference of g_usleep() from cpu_relax() in
> qemu_co_mutex_lock_unlock?
cpu_relax() did not exist when I wrote this patch. :) I'll change it.
Paolo
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH 09/17] util: add stats64 module
2017-04-20 12:00 ` [Qemu-devel] [PATCH 09/17] util: add stats64 module Paolo Bonzini
2017-05-04 7:19 ` Fam Zheng
2017-05-04 7:36 ` Fam Zheng
@ 2017-05-04 8:55 ` Roman Kagan
2017-05-04 9:46 ` Paolo Bonzini
2 siblings, 1 reply; 65+ messages in thread
From: Roman Kagan @ 2017-05-04 8:55 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, Apr 20, 2017 at 02:00:50PM +0200, Paolo Bonzini wrote:
> This module provides fast paths for 64-bit atomic operations on machines
> that only have 32-bit atomic access.
> --- /dev/null
> +++ b/include/qemu/stats64.h
[...]
> +#ifndef STAT64_NEED_SPINLOCK
[...]
> +static inline void stat64_min(Stat64 *s, uint32_t value)
^^
typo or intentional?
Roman.
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH 09/17] util: add stats64 module
2017-05-04 8:55 ` [Qemu-devel] [Qemu-block] " Roman Kagan
@ 2017-05-04 9:46 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 9:46 UTC (permalink / raw)
To: Roman Kagan, qemu-devel, qemu-block
On 04/05/2017 10:55, Roman Kagan wrote:
> On Thu, Apr 20, 2017 at 02:00:50PM +0200, Paolo Bonzini wrote:
>> This module provides fast paths for 64-bit atomic operations on machines
>> that only have 32-bit atomic access.
>
>> --- /dev/null
>> +++ b/include/qemu/stats64.h
> [...]
>> +#ifndef STAT64_NEED_SPINLOCK
> [...]
>> +static inline void stat64_min(Stat64 *s, uint32_t value)
> ^^
> typo or intentional?
Typo.
Thanks,
Paolo
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 10/17] block: use Stat64 for wr_highest_offset
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (8 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 09/17] util: add stats64 module Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 14:02 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 11/17] block: access write_gen with atomics Paolo Bonzini
` (11 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/io.c | 4 +---
block/qapi.c | 2 +-
include/block/block_int.h | 7 ++++---
3 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/block/io.c b/block/io.c
index 3ab9476..fa8abff 100644
--- a/block/io.c
+++ b/block/io.c
@@ -1413,9 +1413,7 @@ static int coroutine_fn bdrv_aligned_pwritev(BdrvChild *child,
++bs->write_gen;
bdrv_set_dirty(bs, start_sector, end_sector - start_sector);
- if (bs->wr_highest_offset < offset + bytes) {
- bs->wr_highest_offset = offset + bytes;
- }
+ stat64_max(&bs->wr_highest_offset, offset + bytes);
if (ret >= 0) {
bs->total_sectors = MAX(bs->total_sectors, end_sector);
diff --git a/block/qapi.c b/block/qapi.c
index a40922e..14b60ae 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -441,7 +441,7 @@ static BlockStats *bdrv_query_bds_stats(const BlockDriverState *bs,
s->node_name = g_strdup(bdrv_get_node_name(bs));
}
- s->stats->wr_highest_offset = bs->wr_highest_offset;
+ s->stats->wr_highest_offset = stat64_get(&bs->wr_highest_offset);
if (bs->file) {
s->has_parent = true;
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 283d079..505c271 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -29,6 +29,7 @@
#include "qemu/option.h"
#include "qemu/queue.h"
#include "qemu/coroutine.h"
+#include "qemu/stats64.h"
#include "qemu/timer.h"
#include "qapi-types.h"
#include "qemu/hbitmap.h"
@@ -590,9 +591,6 @@ struct BlockDriverState {
/* Callback before write request is processed */
NotifierWithReturnList before_write_notifiers;
- /* Offset after the highest byte written to */
- uint64_t wr_highest_offset;
-
/* threshold limit for writes, in bytes. "High water mark". */
uint64_t write_threshold_offset;
NotifierWithReturn write_threshold_notifier;
@@ -605,6 +603,9 @@ struct BlockDriverState {
QLIST_HEAD(, BdrvDirtyBitmap) dirty_bitmaps;
+ /* Offset after the highest byte written to */
+ Stat64 wr_highest_offset;
+
/* If true, copy read backing sectors into image. Can be >1 if more
* than one client has requested copy-on-read. Accessed with atomic
* ops.
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* [Qemu-devel] [PATCH 11/17] block: access write_gen with atomics
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (9 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 10/17] block: use Stat64 for wr_highest_offset Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 14:04 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 12/17] block: protect tracked_requests and flush_queue with reqs_lock Paolo Bonzini
` (10 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block.c | 2 +-
block/io.c | 6 +++---
include/block/block_int.h | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/block.c b/block.c
index a48fdd8..f1aec36 100644
--- a/block.c
+++ b/block.c
@@ -3294,7 +3294,7 @@ int bdrv_truncate(BdrvChild *child, int64_t offset)
ret = refresh_total_sectors(bs, offset >> BDRV_SECTOR_BITS);
bdrv_dirty_bitmap_truncate(bs);
bdrv_parent_cb_resize(bs);
- ++bs->write_gen;
+ atomic_inc(&bs->write_gen);
}
return ret;
}
diff --git a/block/io.c b/block/io.c
index fa8abff..d17564b 100644
--- a/block/io.c
+++ b/block/io.c
@@ -1410,7 +1410,7 @@ static int coroutine_fn bdrv_aligned_pwritev(BdrvChild *child,
}
bdrv_debug_event(bs, BLKDBG_PWRITEV_DONE);
- ++bs->write_gen;
+ atomic_inc(&bs->write_gen);
bdrv_set_dirty(bs, start_sector, end_sector - start_sector);
stat64_max(&bs->wr_highest_offset, offset + bytes);
@@ -2299,7 +2299,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
goto early_exit;
}
- current_gen = bs->write_gen;
+ current_gen = atomic_read(&bs->write_gen);
/* Wait until any previous flushes are completed */
while (bs->active_flush_req) {
@@ -2524,7 +2524,7 @@ int coroutine_fn bdrv_co_pdiscard(BlockDriverState *bs, int64_t offset,
}
ret = 0;
out:
- ++bs->write_gen;
+ atomic_inc(&bs->write_gen);
bdrv_set_dirty(bs, req.offset >> BDRV_SECTOR_BITS,
req.bytes >> BDRV_SECTOR_BITS);
tracked_request_end(&req);
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 505c271..552680c 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -598,7 +598,6 @@ struct BlockDriverState {
QLIST_HEAD(, BdrvTrackedRequest) tracked_requests;
CoQueue flush_queue; /* Serializing flush queue */
bool active_flush_req; /* Flush request in flight? */
- unsigned int write_gen; /* Current data generation */
unsigned int flushed_gen; /* Flushed write generation */
QLIST_HEAD(, BdrvDirtyBitmap) dirty_bitmaps;
@@ -630,6 +629,7 @@ struct BlockDriverState {
/* Accessed with atomic ops. */
int quiesce_counter;
+ unsigned int write_gen; /* Current data generation */
};
struct BlockBackendRootState {
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* [Qemu-devel] [PATCH 12/17] block: protect tracked_requests and flush_queue with reqs_lock
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (10 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 11/17] block: access write_gen with atomics Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 7:30 ` Fam Zheng
2017-04-20 12:00 ` [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock Paolo Bonzini
` (9 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block.c | 1 +
block/io.c | 20 +++++++++++++++++---
include/block/block_int.h | 12 +++++++-----
3 files changed, 25 insertions(+), 8 deletions(-)
diff --git a/block.c b/block.c
index f1aec36..3b2ed29 100644
--- a/block.c
+++ b/block.c
@@ -234,6 +234,7 @@ BlockDriverState *bdrv_new(void)
QLIST_INIT(&bs->op_blockers[i]);
}
notifier_with_return_list_init(&bs->before_write_notifiers);
+ qemu_co_mutex_init(&bs->reqs_lock);
bs->refcnt = 1;
bs->aio_context = qemu_get_aio_context();
diff --git a/block/io.c b/block/io.c
index d17564b..7af9d47 100644
--- a/block/io.c
+++ b/block/io.c
@@ -378,8 +378,10 @@ static void tracked_request_end(BdrvTrackedRequest *req)
atomic_dec(&req->bs->serialising_in_flight);
}
+ qemu_co_mutex_lock(&req->bs->reqs_lock);
QLIST_REMOVE(req, list);
qemu_co_queue_restart_all(&req->wait_queue);
+ qemu_co_mutex_unlock(&req->bs->reqs_lock);
}
/**
@@ -404,7 +406,9 @@ static void tracked_request_begin(BdrvTrackedRequest *req,
qemu_co_queue_init(&req->wait_queue);
+ qemu_co_mutex_lock(&bs->reqs_lock);
QLIST_INSERT_HEAD(&bs->tracked_requests, req, list);
+ qemu_co_mutex_unlock(&bs->reqs_lock);
}
static void mark_request_serialising(BdrvTrackedRequest *req, uint64_t align)
@@ -526,6 +530,7 @@ static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self)
do {
retry = false;
+ qemu_co_mutex_lock(&bs->reqs_lock);
QLIST_FOREACH(req, &bs->tracked_requests, list) {
if (req == self || (!req->serialising && !self->serialising)) {
continue;
@@ -544,7 +549,7 @@ static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self)
* (instead of producing a deadlock in the former case). */
if (!req->waiting_for) {
self->waiting_for = req;
- qemu_co_queue_wait(&req->wait_queue, NULL);
+ qemu_co_queue_wait(&req->wait_queue, &bs->reqs_lock);
self->waiting_for = NULL;
retry = true;
waited = true;
@@ -552,6 +557,7 @@ static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self)
}
}
}
+ qemu_co_mutex_unlock(&bs->reqs_lock);
} while (retry);
return waited;
@@ -2302,11 +2308,13 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
current_gen = atomic_read(&bs->write_gen);
/* Wait until any previous flushes are completed */
+ qemu_co_mutex_lock(&bs->reqs_lock);
while (bs->active_flush_req) {
- qemu_co_queue_wait(&bs->flush_queue, NULL);
+ qemu_co_queue_wait(&bs->flush_queue, &bs->reqs_lock);
}
bs->active_flush_req = true;
+ qemu_co_mutex_unlock(&bs->reqs_lock);
/* Write back all layers by calling one driver function */
if (bs->drv->bdrv_co_flush) {
@@ -2328,10 +2336,14 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
goto flush_parent;
}
- /* Check if we really need to flush anything */
+ /* Check if we really need to flush anything
+ * TODO: use int and atomic access */
+ qemu_co_mutex_lock(&bs->reqs_lock);
if (bs->flushed_gen == current_gen) {
+ qemu_co_mutex_unlock(&bs->reqs_lock);
goto flush_parent;
}
+ qemu_co_mutex_unlock(&bs->reqs_lock);
BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
if (bs->drv->bdrv_co_flush_to_disk) {
@@ -2375,12 +2387,14 @@ flush_parent:
ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
out:
/* Notify any pending flushes that we have completed */
+ qemu_co_mutex_lock(&bs->reqs_lock);
if (ret == 0) {
bs->flushed_gen = current_gen;
}
bs->active_flush_req = false;
/* Return value is ignored - it's ok if wait queue is empty */
qemu_co_queue_next(&bs->flush_queue);
+ qemu_co_mutex_unlock(&bs->reqs_lock);
early_exit:
bdrv_dec_in_flight(bs);
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 552680c..42b49f5 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -595,11 +595,6 @@ struct BlockDriverState {
uint64_t write_threshold_offset;
NotifierWithReturn write_threshold_notifier;
- QLIST_HEAD(, BdrvTrackedRequest) tracked_requests;
- CoQueue flush_queue; /* Serializing flush queue */
- bool active_flush_req; /* Flush request in flight? */
- unsigned int flushed_gen; /* Flushed write generation */
-
QLIST_HEAD(, BdrvDirtyBitmap) dirty_bitmaps;
/* Offset after the highest byte written to */
@@ -630,6 +625,13 @@ struct BlockDriverState {
/* Accessed with atomic ops. */
int quiesce_counter;
unsigned int write_gen; /* Current data generation */
+
+ /* Protected by reqs_lock. */
+ QLIST_HEAD(, BdrvTrackedRequest) tracked_requests;
+ CoQueue flush_queue; /* Serializing flush queue */
+ bool active_flush_req; /* Flush request in flight? */
+ unsigned int flushed_gen; /* Flushed write generation */
+ CoMutex reqs_lock;
};
struct BlockBackendRootState {
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 12/17] block: protect tracked_requests and flush_queue with reqs_lock
2017-04-20 12:00 ` [Qemu-devel] [PATCH 12/17] block: protect tracked_requests and flush_queue with reqs_lock Paolo Bonzini
@ 2017-05-04 7:30 ` Fam Zheng
2017-05-04 8:35 ` Paolo Bonzini
0 siblings, 1 reply; 65+ messages in thread
From: Fam Zheng @ 2017-05-04 7:30 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, 04/20 14:00, Paolo Bonzini wrote:
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> block.c | 1 +
> block/io.c | 20 +++++++++++++++++---
> include/block/block_int.h | 12 +++++++-----
> 3 files changed, 25 insertions(+), 8 deletions(-)
>
> diff --git a/block.c b/block.c
> index f1aec36..3b2ed29 100644
> --- a/block.c
> +++ b/block.c
> @@ -234,6 +234,7 @@ BlockDriverState *bdrv_new(void)
> QLIST_INIT(&bs->op_blockers[i]);
> }
> notifier_with_return_list_init(&bs->before_write_notifiers);
> + qemu_co_mutex_init(&bs->reqs_lock);
> bs->refcnt = 1;
> bs->aio_context = qemu_get_aio_context();
>
> diff --git a/block/io.c b/block/io.c
> index d17564b..7af9d47 100644
> --- a/block/io.c
> +++ b/block/io.c
> @@ -378,8 +378,10 @@ static void tracked_request_end(BdrvTrackedRequest *req)
> atomic_dec(&req->bs->serialising_in_flight);
> }
>
> + qemu_co_mutex_lock(&req->bs->reqs_lock);
> QLIST_REMOVE(req, list);
> qemu_co_queue_restart_all(&req->wait_queue);
> + qemu_co_mutex_unlock(&req->bs->reqs_lock);
> }
>
> /**
> @@ -404,7 +406,9 @@ static void tracked_request_begin(BdrvTrackedRequest *req,
>
> qemu_co_queue_init(&req->wait_queue);
>
> + qemu_co_mutex_lock(&bs->reqs_lock);
> QLIST_INSERT_HEAD(&bs->tracked_requests, req, list);
> + qemu_co_mutex_unlock(&bs->reqs_lock);
> }
>
> static void mark_request_serialising(BdrvTrackedRequest *req, uint64_t align)
> @@ -526,6 +530,7 @@ static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self)
>
> do {
> retry = false;
> + qemu_co_mutex_lock(&bs->reqs_lock);
> QLIST_FOREACH(req, &bs->tracked_requests, list) {
> if (req == self || (!req->serialising && !self->serialising)) {
> continue;
> @@ -544,7 +549,7 @@ static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self)
> * (instead of producing a deadlock in the former case). */
> if (!req->waiting_for) {
> self->waiting_for = req;
> - qemu_co_queue_wait(&req->wait_queue, NULL);
> + qemu_co_queue_wait(&req->wait_queue, &bs->reqs_lock);
> self->waiting_for = NULL;
> retry = true;
> waited = true;
> @@ -552,6 +557,7 @@ static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self)
> }
> }
> }
> + qemu_co_mutex_unlock(&bs->reqs_lock);
> } while (retry);
>
> return waited;
> @@ -2302,11 +2308,13 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
> current_gen = atomic_read(&bs->write_gen);
>
> /* Wait until any previous flushes are completed */
> + qemu_co_mutex_lock(&bs->reqs_lock);
> while (bs->active_flush_req) {
> - qemu_co_queue_wait(&bs->flush_queue, NULL);
> + qemu_co_queue_wait(&bs->flush_queue, &bs->reqs_lock);
> }
>
> bs->active_flush_req = true;
> + qemu_co_mutex_unlock(&bs->reqs_lock);
>
> /* Write back all layers by calling one driver function */
> if (bs->drv->bdrv_co_flush) {
> @@ -2328,10 +2336,14 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
> goto flush_parent;
> }
>
> - /* Check if we really need to flush anything */
> + /* Check if we really need to flush anything
> + * TODO: use int and atomic access */
> + qemu_co_mutex_lock(&bs->reqs_lock);
> if (bs->flushed_gen == current_gen) {
Should the atomic reading of current_gen be moved down here, to avoid TOCTOU?
> + qemu_co_mutex_unlock(&bs->reqs_lock);
> goto flush_parent;
> }
> + qemu_co_mutex_unlock(&bs->reqs_lock);
>
> BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
> if (bs->drv->bdrv_co_flush_to_disk) {
> @@ -2375,12 +2387,14 @@ flush_parent:
> ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
> out:
> /* Notify any pending flushes that we have completed */
> + qemu_co_mutex_lock(&bs->reqs_lock);
> if (ret == 0) {
> bs->flushed_gen = current_gen;
> }
> bs->active_flush_req = false;
> /* Return value is ignored - it's ok if wait queue is empty */
> qemu_co_queue_next(&bs->flush_queue);
> + qemu_co_mutex_unlock(&bs->reqs_lock);
>
> early_exit:
> bdrv_dec_in_flight(bs);
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 12/17] block: protect tracked_requests and flush_queue with reqs_lock
2017-05-04 7:30 ` Fam Zheng
@ 2017-05-04 8:35 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 8:35 UTC (permalink / raw)
To: Fam Zheng; +Cc: qemu-devel, qemu-block
On 04/05/2017 09:30, Fam Zheng wrote:
>> @@ -2302,11 +2308,13 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
>> current_gen = atomic_read(&bs->write_gen);
>>
>> /* Wait until any previous flushes are completed */
>> + qemu_co_mutex_lock(&bs->reqs_lock);
>> while (bs->active_flush_req) {
>> - qemu_co_queue_wait(&bs->flush_queue, NULL);
>> + qemu_co_queue_wait(&bs->flush_queue, &bs->reqs_lock);
>> }
>>
>> bs->active_flush_req = true;
>> + qemu_co_mutex_unlock(&bs->reqs_lock);
>>
>> /* Write back all layers by calling one driver function */
>> if (bs->drv->bdrv_co_flush) {
>> @@ -2328,10 +2336,14 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
>> goto flush_parent;
>> }
>>
>> - /* Check if we really need to flush anything */
>> + /* Check if we really need to flush anything
>> + * TODO: use int and atomic access */
>> + qemu_co_mutex_lock(&bs->reqs_lock);
>> if (bs->flushed_gen == current_gen) {
>
> Should the atomic reading of current_gen be moved down here, to avoid TOCTOU?
No, but another change is needed; current_gen needs to be read under the
lock, to ensure that flushes are processed in increasing generation order.
In addition, this access to flushed_gen does not need the lock;
bs->active_flush_req itself acts as a "lock" for bs->flushed_gen, only
the coroutine that set it to true will read it or write it. I'll adjust
this patch accordingly.
Paolo
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (11 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 12/17] block: protect tracked_requests and flush_queue with reqs_lock Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 7:39 ` Fam Zheng
2017-05-04 14:12 ` Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 14/17] block: optimize access to reqs_lock Paolo Bonzini
` (8 subsequent siblings)
21 siblings, 2 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
This primitive lets you lock/unlock a CoMutex, guaranteeing neither
blocking nor cacheline bouncing if there is no qemu_co_mutex_lock
critical section.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
include/qemu/coroutine.h | 6 ++++++
util/qemu-coroutine-lock.c | 36 ++++++++++++++++++++++++++++++++++++
2 files changed, 42 insertions(+)
diff --git a/include/qemu/coroutine.h b/include/qemu/coroutine.h
index a4509bd..8d4416c 100644
--- a/include/qemu/coroutine.h
+++ b/include/qemu/coroutine.h
@@ -157,6 +157,12 @@ void qemu_co_mutex_init(CoMutex *mutex);
void coroutine_fn qemu_co_mutex_lock(CoMutex *mutex);
/**
+ * Locks the mutex and immediately unlock it. This is faster than back-to-back
+ * lock/unlock if the mutex is not taken by anyone.
+ */
+void coroutine_fn qemu_co_mutex_lock_unlock(CoMutex *mutex);
+
+/**
* Unlocks the mutex and schedules the next coroutine that was waiting for this
* lock to be run.
*/
diff --git a/util/qemu-coroutine-lock.c b/util/qemu-coroutine-lock.c
index 6328eed..86f56cd 100644
--- a/util/qemu-coroutine-lock.c
+++ b/util/qemu-coroutine-lock.c
@@ -287,6 +287,42 @@ retry_fast_path:
self->locks_held++;
}
+void coroutine_fn qemu_co_mutex_lock_unlock(CoMutex *mutex)
+{
+ AioContext *ctx = qemu_get_current_aio_context();
+ int waiters, i;
+
+retry_fast_path:
+ waiters = atomic_read(&mutex->locked);
+ if (waiters == 0) {
+ /* Provide same memory ordering semantics as mutex lock/unlock. */
+ smp_mb_acquire();
+ smp_mb_release();
+ return;
+ }
+
+ i = 0;
+ while (waiters == 1 && ++i < 1000) {
+ if (atomic_read(&mutex->ctx) == ctx) {
+ break;
+ }
+ waiters = atomic_read(&mutex->locked);
+ if (waiters == 0) {
+ smp_mb_acquire();
+ smp_mb_release();
+ return;
+ }
+ cpu_relax();
+ }
+
+ if (atomic_cmpxchg(&mutex->locked, waiters, waiters + 1) != waiters) {
+ goto retry_fast_path;
+ }
+
+ qemu_co_mutex_lock_slowpath(ctx, mutex);
+ qemu_co_mutex_unlock(mutex);
+}
+
void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex)
{
Coroutine *self = qemu_coroutine_self();
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock
2017-04-20 12:00 ` [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock Paolo Bonzini
@ 2017-05-04 7:39 ` Fam Zheng
2017-05-04 9:47 ` Paolo Bonzini
2017-05-04 14:12 ` Stefan Hajnoczi
1 sibling, 1 reply; 65+ messages in thread
From: Fam Zheng @ 2017-05-04 7:39 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, 04/20 14:00, Paolo Bonzini wrote:
> + if (atomic_cmpxchg(&mutex->locked, waiters, waiters + 1) != waiters) {
Is it still useful to try the fast path again if there are now even more
waiters, i.e. "atomic_cmpxchg(...) > waiters"?
> + goto retry_fast_path;
> + }
> +
> + qemu_co_mutex_lock_slowpath(ctx, mutex);
> + qemu_co_mutex_unlock(mutex);
> +}
> +
> void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex)
> {
> Coroutine *self = qemu_coroutine_self();
> --
> 2.9.3
>
>
>
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock
2017-05-04 7:39 ` Fam Zheng
@ 2017-05-04 9:47 ` Paolo Bonzini
2017-05-04 9:52 ` Paolo Bonzini
0 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 9:47 UTC (permalink / raw)
To: Fam Zheng; +Cc: qemu-devel, qemu-block
On 04/05/2017 09:39, Fam Zheng wrote:
> On Thu, 04/20 14:00, Paolo Bonzini wrote:
>> + if (atomic_cmpxchg(&mutex->locked, waiters, waiters + 1) != waiters) {
>
> Is it still useful to try the fast path again if there are now even more
> waiters, i.e. "atomic_cmpxchg(...) > waiters"?
Probably not.
Paolo
>> + goto retry_fast_path;
>> + }
>> +
>> + qemu_co_mutex_lock_slowpath(ctx, mutex);
>> + qemu_co_mutex_unlock(mutex);
>> +}
>> +
>> void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex)
>> {
>> Coroutine *self = qemu_coroutine_self();
>> --
>> 2.9.3
>>
>>
>>
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock
2017-05-04 9:47 ` Paolo Bonzini
@ 2017-05-04 9:52 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 9:52 UTC (permalink / raw)
To: Fam Zheng; +Cc: qemu-devel, qemu-block
On 04/05/2017 11:47, Paolo Bonzini wrote:
>
>
> On 04/05/2017 09:39, Fam Zheng wrote:
>> On Thu, 04/20 14:00, Paolo Bonzini wrote:
>>> + if (atomic_cmpxchg(&mutex->locked, waiters, waiters + 1) != waiters) {
>>
>> Is it still useful to try the fast path again if there are now even more
>> waiters, i.e. "atomic_cmpxchg(...) > waiters"?
>
> Probably not.
... but when this happens, we don't enter the fast path loop:
retry_fast_path:
waiters = atomic_read(&mutex->locked);
if (waiters == 0) {
/* Provide same memory ordering semantics as mutex lock/unlock. */
smp_mb_acquire();
smp_mb_release();
return;
}
i = 0;
while (waiters == 1 && ++i < 1000) {
if (atomic_read(&mutex->ctx) == ctx) {
break;
}
waiters = atomic_read(&mutex->locked);
if (waiters == 0) {
smp_mb_acquire();
smp_mb_release();
return;
}
cpu_relax();
}
if (atomic_cmpxchg(&mutex->locked, waiters, waiters + 1) != waiters) {
goto retry_fast_path;
}
qemu_co_mutex_lock_slowpath(ctx, mutex);
qemu_co_mutex_unlock(mutex);
The "if (waiters == 0)" fails, and the "while (waiters == 1 && ...)" won't
happen either if atomic_cmpxchg returns > 1. So really what you get is
a retry of the atomic_cmpxchg.
We should introduce atomic_try_cmpxchg. Linux added it recently, too,
and it simplifies the code because you don't need to redo the atomic_read.
Like this:
waiters = atomic_read(&mutex->locked);
retry_fast_path:
if (waiters == 0) {
...
}
i = 0;
while (waiters == 1 && ++i < 1000) {
...
}
if (!atomic_cmpxchg(&mutex->locked, &waiters, waiters + 1)) {
goto retry_fast_path;
}
qemu_co_mutex_lock_slowpath(ctx, mutex);
qemu_co_mutex_unlock(mutex);
Paolo
> Paolo
>
>>> + goto retry_fast_path;
>>> + }
>>> +
>>> + qemu_co_mutex_lock_slowpath(ctx, mutex);
>>> + qemu_co_mutex_unlock(mutex);
>>> +}
>>> +
>>> void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex)
>>> {
>>> Coroutine *self = qemu_coroutine_self();
>>> --
>>> 2.9.3
>>>
>>>
>>>
>
>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock
2017-04-20 12:00 ` [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock Paolo Bonzini
2017-05-04 7:39 ` Fam Zheng
@ 2017-05-04 14:12 ` Stefan Hajnoczi
2017-05-04 16:17 ` Paolo Bonzini
1 sibling, 1 reply; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-04 14:12 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 614 bytes --]
On Thu, Apr 20, 2017 at 02:00:54PM +0200, Paolo Bonzini wrote:
> This primitive lets you lock/unlock a CoMutex, guaranteeing neither
> blocking nor cacheline bouncing if there is no qemu_co_mutex_lock
> critical section.
"Guaranteeing neither X nor Y" means "does not guarantee X or Y". Did
you mean "Guarantees not X and not Y"?
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> include/qemu/coroutine.h | 6 ++++++
> util/qemu-coroutine-lock.c | 36 ++++++++++++++++++++++++++++++++++++
> 2 files changed, 42 insertions(+)
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock
2017-05-04 14:12 ` Stefan Hajnoczi
@ 2017-05-04 16:17 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 16:17 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: qemu-devel, qemu-block
On 04/05/2017 16:12, Stefan Hajnoczi wrote:
>> This primitive lets you lock/unlock a CoMutex, guaranteeing neither
>> blocking nor cacheline bouncing if there is no qemu_co_mutex_lock
>> critical section.
> "Guaranteeing neither X nor Y" means "does not guarantee X or Y". Did
> you mean "Guarantees not X and not Y"?
Oops, thanks---I didn't know this! "Guaranteeing no blocking and no
cacheline bouncing if there is no concurrent critical section".
Paolo
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 14/17] block: optimize access to reqs_lock
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (12 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 13/17] coroutine-lock: introduce qemu_co_mutex_lock_unlock Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 14:59 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:00 ` [Qemu-devel] [PATCH 15/17] block: introduce dirty_bitmap_mutex Paolo Bonzini
` (7 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Hot path reqs_lock critical sections are very small; the only large critical
sections happen when a request waits for serialising requests, and these
should never happen in usual circumstances.
We do not want these small critical sections to yield in any case,
which calls for using a spinlock while writing the list. The reqs_lock
is still used to protect the individual requests' CoQueue. For this
purpose, serializing removals against concurrent walks of the request
list can use lock_unlock for efficiency and determinism.
The reqs_lock is also used to protect the flush generation counts, but
that's unrelated.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block.c | 1 +
block/io.c | 25 ++++++++++++++++++++-----
include/block/block_int.h | 11 ++++++++---
3 files changed, 29 insertions(+), 8 deletions(-)
diff --git a/block.c b/block.c
index 3b2ed29..7ba6afe 100644
--- a/block.c
+++ b/block.c
@@ -234,6 +234,7 @@ BlockDriverState *bdrv_new(void)
QLIST_INIT(&bs->op_blockers[i]);
}
notifier_with_return_list_init(&bs->before_write_notifiers);
+ qemu_spin_init(&bs->reqs_list_write_lock);
qemu_co_mutex_init(&bs->reqs_lock);
bs->refcnt = 1;
bs->aio_context = qemu_get_aio_context();
diff --git a/block/io.c b/block/io.c
index 7af9d47..476807d 100644
--- a/block/io.c
+++ b/block/io.c
@@ -374,14 +374,29 @@ void bdrv_drain_all(void)
*/
static void tracked_request_end(BdrvTrackedRequest *req)
{
+ BlockDriverState *bs = req->bs;
+
if (req->serialising) {
- atomic_dec(&req->bs->serialising_in_flight);
+ atomic_dec(&bs->serialising_in_flight);
}
- qemu_co_mutex_lock(&req->bs->reqs_lock);
+ /* Note that there can be a concurrent visit while we remove the list,
+ * so we need to...
+ */
+ qemu_spin_lock(&bs->reqs_list_write_lock);
QLIST_REMOVE(req, list);
+ qemu_spin_unlock(&bs->reqs_list_write_lock);
+
+ /* ... wait for it to end before we leave. qemu_co_mutex_lock_unlock
+ * avoids cacheline bouncing in the common case of no concurrent
+ * reader.
+ */
+ qemu_co_mutex_lock_unlock(&bs->reqs_lock);
+
+ /* Now no coroutine can add itself to the wait queue, so it is
+ * safe to call qemu_co_queue_restart_all outside the reqs_lock.
+ */
qemu_co_queue_restart_all(&req->wait_queue);
- qemu_co_mutex_unlock(&req->bs->reqs_lock);
}
/**
@@ -406,9 +421,9 @@ static void tracked_request_begin(BdrvTrackedRequest *req,
qemu_co_queue_init(&req->wait_queue);
- qemu_co_mutex_lock(&bs->reqs_lock);
+ qemu_spin_lock(&bs->reqs_list_write_lock);
QLIST_INSERT_HEAD(&bs->tracked_requests, req, list);
- qemu_co_mutex_unlock(&bs->reqs_lock);
+ qemu_spin_unlock(&bs->reqs_list_write_lock);
}
static void mark_request_serialising(BdrvTrackedRequest *req, uint64_t align)
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 42b49f5..b298de8 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -78,9 +78,10 @@ typedef struct BdrvTrackedRequest {
QLIST_ENTRY(BdrvTrackedRequest) list;
Coroutine *co; /* owner, used for deadlock detection */
- CoQueue wait_queue; /* coroutines blocked on this request */
-
struct BdrvTrackedRequest *waiting_for;
+
+ /* Protected by BlockDriverState's reqs_lock. */
+ CoQueue wait_queue; /* coroutines blocked on this request */
} BdrvTrackedRequest;
struct BlockDriver {
@@ -626,11 +627,15 @@ struct BlockDriverState {
int quiesce_counter;
unsigned int write_gen; /* Current data generation */
- /* Protected by reqs_lock. */
+ /* Writes are protected by reqs_list_write_lock. Reads take
+ * reqs_lock so that removals can easily synchronize with walks.
+ */
QLIST_HEAD(, BdrvTrackedRequest) tracked_requests;
CoQueue flush_queue; /* Serializing flush queue */
bool active_flush_req; /* Flush request in flight? */
unsigned int flushed_gen; /* Flushed write generation */
+
+ QemuSpin reqs_list_write_lock;
CoMutex reqs_lock;
};
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH 14/17] block: optimize access to reqs_lock
2017-04-20 12:00 ` [Qemu-devel] [PATCH 14/17] block: optimize access to reqs_lock Paolo Bonzini
@ 2017-05-04 14:59 ` Stefan Hajnoczi
2017-05-04 16:06 ` Paolo Bonzini
0 siblings, 1 reply; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-04 14:59 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 927 bytes --]
On Thu, Apr 20, 2017 at 02:00:55PM +0200, Paolo Bonzini wrote:
> Hot path reqs_lock critical sections are very small; the only large critical
> sections happen when a request waits for serialising requests, and these
> should never happen in usual circumstances.
>
> We do not want these small critical sections to yield in any case,
> which calls for using a spinlock while writing the list.
Is this patch purely an optimization?
I'm hesitant about using spinlocks in userspace. There are cases where
the thread is descheduled that are beyond our control. Nested virt will
probably make things worse. People have been optimizing and trying
paravirt approaches to kernel spinlocks for these reasons for years.
Isn't a futex-based lock efficient enough? That way we don't hog the
CPU when there is contention.
Also, there are no performance results included in this patch that
justify the spinlock.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH 14/17] block: optimize access to reqs_lock
2017-05-04 14:59 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
@ 2017-05-04 16:06 ` Paolo Bonzini
2017-05-05 10:25 ` Stefan Hajnoczi
0 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 16:06 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: qemu-devel, qemu-block
On 04/05/2017 16:59, Stefan Hajnoczi wrote:
> On Thu, Apr 20, 2017 at 02:00:55PM +0200, Paolo Bonzini wrote:
>> Hot path reqs_lock critical sections are very small; the only large critical
>> sections happen when a request waits for serialising requests, and these
>> should never happen in usual circumstances.
>>
>> We do not want these small critical sections to yield in any case,
>> which calls for using a spinlock while writing the list.
>
> Is this patch purely an optimization?
Yes, it is, and pretty much a no-op until we have true multiqueue. But
I expect it to have a significant effect for multiqueue.
> I'm hesitant about using spinlocks in userspace. There are cases where
> the thread is descheduled that are beyond our control. Nested virt will
> probably make things worse. People have been optimizing and trying
> paravirt approaches to kernel spinlocks for these reasons for years.
This is true, but here we're talking about a 5-10 instruction window for
preemption; it matches the usage of spinlocks in other parts of QEMU.
The long critical sections, which only happen with combination with
copy-on-read or RMW (large logical block sizes on the host), take the
CoMutex.
On one hand it's true that the more you nest, the more things get worse.
On the other hand there can only ever be contention with multiqueue,
and the multiqueue scenarios are going to use pinning.
> Isn't a futex-based lock efficient enough? That way we don't hog the
> CPU when there is contention.
It is efficient when there is no contention, but when there is, the
latency goes up by several orders of magnitude.
Paolo
> Also, there are no performance results included in this patch that
> justify the spinlock.
>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH 14/17] block: optimize access to reqs_lock
2017-05-04 16:06 ` Paolo Bonzini
@ 2017-05-05 10:25 ` Stefan Hajnoczi
2017-05-05 10:45 ` Paolo Bonzini
0 siblings, 1 reply; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-05 10:25 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]
On Thu, May 04, 2017 at 06:06:39PM +0200, Paolo Bonzini wrote:
> On 04/05/2017 16:59, Stefan Hajnoczi wrote:
> > On Thu, Apr 20, 2017 at 02:00:55PM +0200, Paolo Bonzini wrote:
> >> Hot path reqs_lock critical sections are very small; the only large critical
> >> sections happen when a request waits for serialising requests, and these
> >> should never happen in usual circumstances.
> >>
> >> We do not want these small critical sections to yield in any case,
> >> which calls for using a spinlock while writing the list.
> >
> > Is this patch purely an optimization?
>
> Yes, it is, and pretty much a no-op until we have true multiqueue. But
> I expect it to have a significant effect for multiqueue.
>
> > I'm hesitant about using spinlocks in userspace. There are cases where
> > the thread is descheduled that are beyond our control. Nested virt will
> > probably make things worse. People have been optimizing and trying
> > paravirt approaches to kernel spinlocks for these reasons for years.
>
> This is true, but here we're talking about a 5-10 instruction window for
> preemption; it matches the usage of spinlocks in other parts of QEMU.
Only util/qht.c uses spinlocks, it's not a widely used primitive.
> The long critical sections, which only happen with combination with
> copy-on-read or RMW (large logical block sizes on the host), take the
> CoMutex.
>
> On one hand it's true that the more you nest, the more things get worse.
> On the other hand there can only ever be contention with multiqueue,
> and the multiqueue scenarios are going to use pinning.
>
> > Isn't a futex-based lock efficient enough? That way we don't hog the
> > CPU when there is contention.
>
> It is efficient when there is no contention, but when there is, the
> latency goes up by several orders of magnitude.
Doesn't glibc spin for a while before waiting on the futex? i.e. the
best of both worlds.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH 14/17] block: optimize access to reqs_lock
2017-05-05 10:25 ` Stefan Hajnoczi
@ 2017-05-05 10:45 ` Paolo Bonzini
2017-05-08 16:21 ` Stefan Hajnoczi
0 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-05 10:45 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: qemu-devel, qemu-block
On 05/05/2017 12:25, Stefan Hajnoczi wrote:
> On Thu, May 04, 2017 at 06:06:39PM +0200, Paolo Bonzini wrote:
>> On 04/05/2017 16:59, Stefan Hajnoczi wrote:
>>> On Thu, Apr 20, 2017 at 02:00:55PM +0200, Paolo Bonzini wrote:
>>>> Hot path reqs_lock critical sections are very small; the only large critical
>>>> sections happen when a request waits for serialising requests, and these
>>>> should never happen in usual circumstances.
>>>>
>>>> We do not want these small critical sections to yield in any case,
>>>> which calls for using a spinlock while writing the list.
>>>
>>> Is this patch purely an optimization?
>>
>> Yes, it is, and pretty much a no-op until we have true multiqueue. But
>> I expect it to have a significant effect for multiqueue.
>>
>>> I'm hesitant about using spinlocks in userspace. There are cases where
>>> the thread is descheduled that are beyond our control. Nested virt will
>>> probably make things worse. People have been optimizing and trying
>>> paravirt approaches to kernel spinlocks for these reasons for years.
>>
>> This is true, but here we're talking about a 5-10 instruction window for
>> preemption; it matches the usage of spinlocks in other parts of QEMU.
>
> Only util/qht.c uses spinlocks, it's not a widely used primitive.
Right, but the idea is the same---very short, heavy and
performance-critical cases use spinlocks. (util/qht.c is used heavily
in TCG mode).
>> It is efficient when there is no contention, but when there is, the
>> latency goes up by several orders of magnitude.
>
> Doesn't glibc spin for a while before waiting on the futex? i.e. the
> best of both worlds.
You have to specify that manually with pthread_mutexattr_settype(...,
PTHRED_MUTEX_ADAPTIVE_NP). It is not enabled by default because IIUC
the adaptive one doesn't support pthread_mutex_timedlock.
Paolo
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH 14/17] block: optimize access to reqs_lock
2017-05-05 10:45 ` Paolo Bonzini
@ 2017-05-08 16:21 ` Stefan Hajnoczi
2017-05-08 16:30 ` Paolo Bonzini
0 siblings, 1 reply; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-08 16:21 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 2421 bytes --]
On Fri, May 05, 2017 at 12:45:38PM +0200, Paolo Bonzini wrote:
>
>
> On 05/05/2017 12:25, Stefan Hajnoczi wrote:
> > On Thu, May 04, 2017 at 06:06:39PM +0200, Paolo Bonzini wrote:
> >> On 04/05/2017 16:59, Stefan Hajnoczi wrote:
> >>> On Thu, Apr 20, 2017 at 02:00:55PM +0200, Paolo Bonzini wrote:
> >>>> Hot path reqs_lock critical sections are very small; the only large critical
> >>>> sections happen when a request waits for serialising requests, and these
> >>>> should never happen in usual circumstances.
> >>>>
> >>>> We do not want these small critical sections to yield in any case,
> >>>> which calls for using a spinlock while writing the list.
> >>>
> >>> Is this patch purely an optimization?
> >>
> >> Yes, it is, and pretty much a no-op until we have true multiqueue. But
> >> I expect it to have a significant effect for multiqueue.
> >>
> >>> I'm hesitant about using spinlocks in userspace. There are cases where
> >>> the thread is descheduled that are beyond our control. Nested virt will
> >>> probably make things worse. People have been optimizing and trying
> >>> paravirt approaches to kernel spinlocks for these reasons for years.
> >>
> >> This is true, but here we're talking about a 5-10 instruction window for
> >> preemption; it matches the usage of spinlocks in other parts of QEMU.
> >
> > Only util/qht.c uses spinlocks, it's not a widely used primitive.
>
> Right, but the idea is the same---very short, heavy and
> performance-critical cases use spinlocks. (util/qht.c is used heavily
> in TCG mode).
>
> >> It is efficient when there is no contention, but when there is, the
> >> latency goes up by several orders of magnitude.
> >
> > Doesn't glibc spin for a while before waiting on the futex? i.e. the
> > best of both worlds.
>
> You have to specify that manually with pthread_mutexattr_settype(...,
> PTHRED_MUTEX_ADAPTIVE_NP). It is not enabled by default because IIUC
> the adaptive one doesn't support pthread_mutex_timedlock.
If you want to use spinlocks in QEMU please document strict rules that
ensure they will be used safely. For example, I'd be comfortable with:
Spinlock regions may only call functions in the same source file or
functions explicitly documented as spinlock-safe. Only memory accesses
and computation may be performed; do not make system calls or invoke
library functions.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH 14/17] block: optimize access to reqs_lock
2017-05-08 16:21 ` Stefan Hajnoczi
@ 2017-05-08 16:30 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-08 16:30 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: qemu-devel, qemu-block
On 08/05/2017 18:21, Stefan Hajnoczi wrote:
> If you want to use spinlocks in QEMU please document strict rules that
> ensure they will be used safely. For example, I'd be comfortable with:
>
> Spinlock regions may only call functions in the same source file or
> functions explicitly documented as spinlock-safe. Only memory accesses
> and computation may be performed; do not make system calls or invoke
> library functions.
Sounds good. But for now I'll just drop these patches in the interest
of simplicity.
Paolo
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 15/17] block: introduce dirty_bitmap_mutex
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (13 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 14/17] block: optimize access to reqs_lock Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-04 7:55 ` Fam Zheng
2017-04-20 12:00 ` [Qemu-devel] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex Paolo Bonzini
` (6 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
It protects only the list of dirty bitmaps; in the next patch we will
also protect their content.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/dirty-bitmap.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++-
block/mirror.c | 3 ++-
blockdev.c | 44 +++++++---------------------------------
include/block/block_int.h | 5 +++++
migration/block.c | 6 ------
5 files changed, 64 insertions(+), 45 deletions(-)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 519737c..e13718e 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -52,6 +52,24 @@ struct BdrvDirtyBitmapIter {
BdrvDirtyBitmap *bitmap;
};
+static QemuMutex dirty_bitmap_mutex;
+
+static void __attribute__((__constructor__)) bdrv_dirty_bitmaps_init_lock(void)
+{
+ qemu_mutex_init(&dirty_bitmap_mutex);
+}
+
+static inline void bdrv_dirty_bitmaps_lock(BlockDriverState *bs)
+{
+ qemu_mutex_lock(&dirty_bitmap_mutex);
+}
+
+static inline void bdrv_dirty_bitmaps_unlock(BlockDriverState *bs)
+{
+ qemu_mutex_unlock(&dirty_bitmap_mutex);
+}
+
+/* Called with BQL or dirty_bitmap lock taken. */
BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs, const char *name)
{
BdrvDirtyBitmap *bm;
@@ -65,6 +83,7 @@ BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs, const char *name)
return NULL;
}
+/* Called with BQL taken. */
void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)
{
assert(!bdrv_dirty_bitmap_frozen(bitmap));
@@ -72,6 +91,7 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)
bitmap->name = NULL;
}
+/* Called with BQL taken. */
BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
uint32_t granularity,
const char *name,
@@ -100,7 +120,9 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
bitmap->size = bitmap_size;
bitmap->name = g_strdup(name);
bitmap->disabled = false;
+ bdrv_dirty_bitmaps_lock(bs);
QLIST_INSERT_HEAD(&bs->dirty_bitmaps, bitmap, list);
+ bdrv_dirty_bitmaps_unlock(bs);
return bitmap;
}
@@ -164,16 +186,19 @@ const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap)
return bitmap->name;
}
+/* Called with BQL taken. */
bool bdrv_dirty_bitmap_frozen(BdrvDirtyBitmap *bitmap)
{
return bitmap->successor;
}
+/* Called with BQL taken. */
bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap)
{
return !(bitmap->disabled || bitmap->successor);
}
+/* Called with BQL taken. */
DirtyBitmapStatus bdrv_dirty_bitmap_status(BdrvDirtyBitmap *bitmap)
{
if (bdrv_dirty_bitmap_frozen(bitmap)) {
@@ -188,6 +213,7 @@ DirtyBitmapStatus bdrv_dirty_bitmap_status(BdrvDirtyBitmap *bitmap)
/**
* Create a successor bitmap destined to replace this bitmap after an operation.
* Requires that the bitmap is not frozen and has no successor.
+ * Called with BQL taken.
*/
int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap, Error **errp)
@@ -220,6 +246,7 @@ int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
/**
* For a bitmap with a successor, yield our name to the successor,
* delete the old bitmap, and return a handle to the new bitmap.
+ * Called with BQL taken.
*/
BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap,
@@ -247,6 +274,7 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
* In cases of failure where we can no longer safely delete the parent,
* we may wish to re-join the parent and child/successor.
* The merged parent will be un-frozen, but not explicitly re-enabled.
+ * Called with BQL taken.
*/
BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
BdrvDirtyBitmap *parent,
@@ -271,25 +299,30 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
/**
* Truncates _all_ bitmaps attached to a BDS.
+ * Called with BQL taken.
*/
void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
{
BdrvDirtyBitmap *bitmap;
uint64_t size = bdrv_nb_sectors(bs);
+ bdrv_dirty_bitmaps_lock(bs);
QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
assert(!bdrv_dirty_bitmap_frozen(bitmap));
assert(!bitmap->active_iterators);
hbitmap_truncate(bitmap->bitmap, size);
bitmap->size = size;
}
+ bdrv_dirty_bitmaps_unlock(bs);
}
+/* Called with BQL taken. */
static void bdrv_do_release_matching_dirty_bitmap(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap,
bool only_named)
{
BdrvDirtyBitmap *bm, *next;
+ bdrv_dirty_bitmaps_lock(bs);
QLIST_FOREACH_SAFE(bm, &bs->dirty_bitmaps, list, next) {
if ((!bitmap || bm == bitmap) && (!only_named || bm->name)) {
assert(!bm->active_iterators);
@@ -301,15 +334,19 @@ static void bdrv_do_release_matching_dirty_bitmap(BlockDriverState *bs,
g_free(bm);
if (bitmap) {
- return;
+ goto out;
}
}
}
if (bitmap) {
abort();
}
+
+out:
+ bdrv_dirty_bitmaps_unlock(bs);
}
+/* Called with BQL taken. */
void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
{
bdrv_do_release_matching_dirty_bitmap(bs, bitmap, false);
@@ -318,18 +355,21 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
/**
* Release all named dirty bitmaps attached to a BDS (for use in bdrv_close()).
* There must not be any frozen bitmaps attached.
+ * Called with BQL taken.
*/
void bdrv_release_named_dirty_bitmaps(BlockDriverState *bs)
{
bdrv_do_release_matching_dirty_bitmap(bs, NULL, true);
}
+/* Called with BQL taken. */
void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
{
assert(!bdrv_dirty_bitmap_frozen(bitmap));
bitmap->disabled = true;
}
+/* Called with BQL taken. */
void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
{
assert(!bdrv_dirty_bitmap_frozen(bitmap));
@@ -342,6 +382,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
BlockDirtyInfoList *list = NULL;
BlockDirtyInfoList **plist = &list;
+ bdrv_dirty_bitmaps_lock(bs);
QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) {
BlockDirtyInfo *info = g_new0(BlockDirtyInfo, 1);
BlockDirtyInfoList *entry = g_new0(BlockDirtyInfoList, 1);
@@ -354,6 +395,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
*plist = entry;
plist = &entry->next;
}
+ bdrv_dirty_bitmaps_unlock(bs);
return list;
}
@@ -508,12 +550,19 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
int64_t nr_sectors)
{
BdrvDirtyBitmap *bitmap;
+
+ if (QLIST_EMPTY(&bs->dirty_bitmaps)) {
+ return;
+ }
+
+ bdrv_dirty_bitmaps_lock(bs);
QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
if (!bdrv_dirty_bitmap_enabled(bitmap)) {
continue;
}
hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
}
+ bdrv_dirty_bitmaps_unlock(bs);
}
/**
diff --git a/block/mirror.c b/block/mirror.c
index 2e05dac..dc227a2 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -506,6 +506,8 @@ static void mirror_exit(BlockJob *job, void *opaque)
BlockDriverState *mirror_top_bs = s->mirror_top_bs;
Error *local_err = NULL;
+ bdrv_release_dirty_bitmap(src, s->dirty_bitmap);
+
/* Make sure that the source BDS doesn't go away before we called
* block_job_completed(). */
bdrv_ref(src);
@@ -899,7 +901,6 @@ immediate_exit:
g_free(s->cow_bitmap);
g_free(s->in_flight_bitmap);
bdrv_dirty_iter_free(s->dbi);
- bdrv_release_dirty_bitmap(bs, s->dirty_bitmap);
data = g_malloc(sizeof(*data));
data->ret = ret;
diff --git a/blockdev.c b/blockdev.c
index e9b5717..09325f2 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1364,12 +1364,10 @@ out_aio_context:
static BdrvDirtyBitmap *block_dirty_bitmap_lookup(const char *node,
const char *name,
BlockDriverState **pbs,
- AioContext **paio,
Error **errp)
{
BlockDriverState *bs;
BdrvDirtyBitmap *bitmap;
- AioContext *aio_context;
if (!node) {
error_setg(errp, "Node cannot be NULL");
@@ -1385,29 +1383,17 @@ static BdrvDirtyBitmap *block_dirty_bitmap_lookup(const char *node,
return NULL;
}
- aio_context = bdrv_get_aio_context(bs);
- aio_context_acquire(aio_context);
-
bitmap = bdrv_find_dirty_bitmap(bs, name);
if (!bitmap) {
error_setg(errp, "Dirty bitmap '%s' not found", name);
- goto fail;
+ return NULL;
}
if (pbs) {
*pbs = bs;
}
- if (paio) {
- *paio = aio_context;
- } else {
- aio_context_release(aio_context);
- }
return bitmap;
-
- fail:
- aio_context_release(aio_context);
- return NULL;
}
/* New and old BlockDriverState structs for atomic group operations */
@@ -2024,7 +2010,6 @@ static void block_dirty_bitmap_clear_prepare(BlkActionState *common,
state->bitmap = block_dirty_bitmap_lookup(action->node,
action->name,
&state->bs,
- &state->aio_context,
errp);
if (!state->bitmap) {
return;
@@ -2733,7 +2718,6 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
bool has_granularity, uint32_t granularity,
Error **errp)
{
- AioContext *aio_context;
BlockDriverState *bs;
if (!name || name[0] == '\0') {
@@ -2746,14 +2730,11 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
return;
}
- aio_context = bdrv_get_aio_context(bs);
- aio_context_acquire(aio_context);
-
if (has_granularity) {
if (granularity < 512 || !is_power_of_2(granularity)) {
error_setg(errp, "Granularity must be power of 2 "
"and at least 512");
- goto out;
+ return;
}
} else {
/* Default to cluster size, if available: */
@@ -2761,19 +2742,15 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
}
bdrv_create_dirty_bitmap(bs, granularity, name, errp);
-
- out:
- aio_context_release(aio_context);
}
void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
Error **errp)
{
- AioContext *aio_context;
BlockDriverState *bs;
BdrvDirtyBitmap *bitmap;
- bitmap = block_dirty_bitmap_lookup(node, name, &bs, &aio_context, errp);
+ bitmap = block_dirty_bitmap_lookup(node, name, &bs, errp);
if (!bitmap || !bs) {
return;
}
@@ -2782,13 +2759,10 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
error_setg(errp,
"Bitmap '%s' is currently frozen and cannot be removed",
name);
- goto out;
+ return;
}
bdrv_dirty_bitmap_make_anon(bitmap);
bdrv_release_dirty_bitmap(bs, bitmap);
-
- out:
- aio_context_release(aio_context);
}
/**
@@ -2798,11 +2772,10 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
void qmp_block_dirty_bitmap_clear(const char *node, const char *name,
Error **errp)
{
- AioContext *aio_context;
BdrvDirtyBitmap *bitmap;
BlockDriverState *bs;
- bitmap = block_dirty_bitmap_lookup(node, name, &bs, &aio_context, errp);
+ bitmap = block_dirty_bitmap_lookup(node, name, &bs, errp);
if (!bitmap || !bs) {
return;
}
@@ -2811,18 +2784,15 @@ void qmp_block_dirty_bitmap_clear(const char *node, const char *name,
error_setg(errp,
"Bitmap '%s' is currently frozen and cannot be modified",
name);
- goto out;
+ return;
} else if (!bdrv_dirty_bitmap_enabled(bitmap)) {
error_setg(errp,
"Bitmap '%s' is currently disabled and cannot be cleared",
name);
- goto out;
+ return;
}
bdrv_clear_dirty_bitmap(bitmap, NULL);
-
- out:
- aio_context_release(aio_context);
}
void hmp_drive_del(Monitor *mon, const QDict *qdict)
diff --git a/include/block/block_int.h b/include/block/block_int.h
index b298de8..03db2cf 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -596,6 +596,11 @@ struct BlockDriverState {
uint64_t write_threshold_offset;
NotifierWithReturn write_threshold_notifier;
+ /* Writing to the list requires the BQL _and_ the dirty_bitmap_mutex.
+ * Reading from the list can be done with either the BQL or the
+ * dirty_bitmap_mutex. Modifying a bitmap requires the AioContext
+ * lock. */
+ QemuMutex dirty_bitmap_mutex;
QLIST_HEAD(, BdrvDirtyBitmap) dirty_bitmaps;
/* Offset after the highest byte written to */
diff --git a/migration/block.c b/migration/block.c
index 7734ff7..c33b522 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -346,10 +346,8 @@ static int set_dirty_tracking(void)
int ret;
QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {
- aio_context_acquire(blk_get_aio_context(bmds->blk));
bmds->dirty_bitmap = bdrv_create_dirty_bitmap(blk_bs(bmds->blk),
BLOCK_SIZE, NULL, NULL);
- aio_context_release(blk_get_aio_context(bmds->blk));
if (!bmds->dirty_bitmap) {
ret = -errno;
goto fail;
@@ -360,9 +358,7 @@ static int set_dirty_tracking(void)
fail:
QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {
if (bmds->dirty_bitmap) {
- aio_context_acquire(blk_get_aio_context(bmds->blk));
bdrv_release_dirty_bitmap(blk_bs(bmds->blk), bmds->dirty_bitmap);
- aio_context_release(blk_get_aio_context(bmds->blk));
}
}
return ret;
@@ -375,9 +371,7 @@ static void unset_dirty_tracking(void)
BlkMigDevState *bmds;
QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) {
- aio_context_acquire(blk_get_aio_context(bmds->blk));
bdrv_release_dirty_bitmap(blk_bs(bmds->blk), bmds->dirty_bitmap);
- aio_context_release(blk_get_aio_context(bmds->blk));
}
}
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 15/17] block: introduce dirty_bitmap_mutex
2017-04-20 12:00 ` [Qemu-devel] [PATCH 15/17] block: introduce dirty_bitmap_mutex Paolo Bonzini
@ 2017-05-04 7:55 ` Fam Zheng
2017-05-04 9:57 ` Paolo Bonzini
0 siblings, 1 reply; 65+ messages in thread
From: Fam Zheng @ 2017-05-04 7:55 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, 04/20 14:00, Paolo Bonzini wrote:
> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
> index 519737c..e13718e 100644
> --- a/block/dirty-bitmap.c
> +++ b/block/dirty-bitmap.c
> @@ -52,6 +52,24 @@ struct BdrvDirtyBitmapIter {
> BdrvDirtyBitmap *bitmap;
> };
>
> +static QemuMutex dirty_bitmap_mutex;
> +
> +static void __attribute__((__constructor__)) bdrv_dirty_bitmaps_init_lock(void)
> +{
> + qemu_mutex_init(&dirty_bitmap_mutex);
> +}
> +
> +static inline void bdrv_dirty_bitmaps_lock(BlockDriverState *bs)
> +{
> + qemu_mutex_lock(&dirty_bitmap_mutex);
> +}
> +
> +static inline void bdrv_dirty_bitmaps_unlock(BlockDriverState *bs)
> +{
> + qemu_mutex_unlock(&dirty_bitmap_mutex);
> +}
Why a global lock instead of a per-BDS one? The contention can be heavy if a
block job is made to run on a different thread than the one processing guest
I/O.
> @@ -508,12 +550,19 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
> int64_t nr_sectors)
> {
> BdrvDirtyBitmap *bitmap;
> +
> + if (QLIST_EMPTY(&bs->dirty_bitmaps)) {
> + return;
> + }
Should this check be protected by lock/unlock? Or simply removed?
> +
> + bdrv_dirty_bitmaps_lock(bs);
> QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
> if (!bdrv_dirty_bitmap_enabled(bitmap)) {
> continue;
> }
> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
> }
> + bdrv_dirty_bitmaps_unlock(bs);
> }
>
> /**
Fam
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 15/17] block: introduce dirty_bitmap_mutex
2017-05-04 7:55 ` Fam Zheng
@ 2017-05-04 9:57 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 9:57 UTC (permalink / raw)
To: Fam Zheng; +Cc: qemu-devel, qemu-block
On 04/05/2017 09:55, Fam Zheng wrote:
> On Thu, 04/20 14:00, Paolo Bonzini wrote:
>> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
>> index 519737c..e13718e 100644
>> --- a/block/dirty-bitmap.c
>> +++ b/block/dirty-bitmap.c
>> @@ -52,6 +52,24 @@ struct BdrvDirtyBitmapIter {
>> BdrvDirtyBitmap *bitmap;
>> };
>>
>> +static QemuMutex dirty_bitmap_mutex;
>> +
>> +static void __attribute__((__constructor__)) bdrv_dirty_bitmaps_init_lock(void)
>> +{
>> + qemu_mutex_init(&dirty_bitmap_mutex);
>> +}
>> +
>> +static inline void bdrv_dirty_bitmaps_lock(BlockDriverState *bs)
>> +{
>> + qemu_mutex_lock(&dirty_bitmap_mutex);
>> +}
>> +
>> +static inline void bdrv_dirty_bitmaps_unlock(BlockDriverState *bs)
>> +{
>> + qemu_mutex_unlock(&dirty_bitmap_mutex);
>> +}
>
> Why a global lock instead of a per-BDS one? The contention can be heavy if a
> block job is made to run on a different thread than the one processing guest
> I/O.
Yes, I'll introduce bs->dirty_bitmap_mutex in this patch already.
>> @@ -508,12 +550,19 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
>> int64_t nr_sectors)
>> {
>> BdrvDirtyBitmap *bitmap;
>> +
>> + if (QLIST_EMPTY(&bs->dirty_bitmaps)) {
>> + return;
>> + }
>
> Should this check be protected by lock/unlock? Or simply removed?
The check avoids taking the lock in the common case of having no dirty
bitmap.
Paolo
>> +
>> + bdrv_dirty_bitmaps_lock(bs);
>> QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
>> if (!bdrv_dirty_bitmap_enabled(bitmap)) {
>> continue;
>> }
>> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
>> }
>> + bdrv_dirty_bitmaps_unlock(bs);
>> }
>>
>> /**
>
> Fam
>
>
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (14 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 15/17] block: introduce dirty_bitmap_mutex Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-04-20 14:42 ` Eric Blake
` (2 more replies)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 17/17] block: make accounting thread-safe Paolo Bonzini
` (5 subsequent siblings)
21 siblings, 3 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/dirty-bitmap.c | 74 +++++++++++++++++++++++++++++++++++++++++---
block/mirror.c | 11 +++++--
include/block/block_int.h | 4 +--
include/block/dirty-bitmap.h | 23 +++++++++++---
4 files changed, 97 insertions(+), 15 deletions(-)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index e13718e..b854077 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -37,6 +37,7 @@
* or enabled. A frozen bitmap can only abdicate() or reclaim().
*/
struct BdrvDirtyBitmap {
+ QemuMutex *mutex;
HBitmap *bitmap; /* Dirty sector bitmap implementation */
HBitmap *meta; /* Meta dirty bitmap */
BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */
@@ -69,6 +70,16 @@ static inline void bdrv_dirty_bitmaps_unlock(BlockDriverState *bs)
qemu_mutex_unlock(&dirty_bitmap_mutex);
}
+void bdrv_dirty_bitmap_lock(BdrvDirtyBitmap *bitmap)
+{
+ qemu_mutex_lock(bitmap->mutex);
+}
+
+void bdrv_dirty_bitmap_unlock(BdrvDirtyBitmap *bitmap)
+{
+ qemu_mutex_unlock(bitmap->mutex);
+}
+
/* Called with BQL or dirty_bitmap lock taken. */
BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs, const char *name)
{
@@ -116,6 +127,7 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
return NULL;
}
bitmap = g_new0(BdrvDirtyBitmap, 1);
+ bitmap->mutex = &bs->dirty_bitmap_mutex;
bitmap->bitmap = hbitmap_alloc(bitmap_size, ctz32(sector_granularity));
bitmap->size = bitmap_size;
bitmap->name = g_strdup(name);
@@ -141,18 +153,22 @@ void bdrv_create_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int chunk_size)
{
assert(!bitmap->meta);
+ qemu_mutex_lock(bitmap->mutex);
bitmap->meta = hbitmap_create_meta(bitmap->bitmap,
chunk_size * BITS_PER_BYTE);
+ qemu_mutex_unlock(bitmap->mutex);
}
void bdrv_release_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap)
{
assert(bitmap->meta);
+ qemu_mutex_lock(bitmap->mutex);
hbitmap_free_meta(bitmap->bitmap);
bitmap->meta = NULL;
+ qemu_mutex_unlock(bitmap->mutex);
}
-int bdrv_dirty_bitmap_get_meta(BlockDriverState *bs,
+int bdrv_dirty_bitmap_get_meta_locked(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap, int64_t sector,
int nb_sectors)
{
@@ -169,11 +185,26 @@ int bdrv_dirty_bitmap_get_meta(BlockDriverState *bs,
return false;
}
+int bdrv_dirty_bitmap_get_meta(BlockDriverState *bs,
+ BdrvDirtyBitmap *bitmap, int64_t sector,
+ int nb_sectors)
+{
+ bool dirty;
+
+ qemu_mutex_lock(bitmap->mutex);
+ dirty = bdrv_dirty_bitmap_get_meta_locked(bs, bitmap, sector, nb_sectors);
+ qemu_mutex_unlock(bitmap->mutex);
+
+ return dirty;
+}
+
void bdrv_dirty_bitmap_reset_meta(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap, int64_t sector,
int nb_sectors)
{
+ qemu_mutex_lock(bitmap->mutex);
hbitmap_reset(bitmap->meta, sector, nb_sectors);
+ qemu_mutex_unlock(bitmap->mutex);
}
int64_t bdrv_dirty_bitmap_size(const BdrvDirtyBitmap *bitmap)
@@ -400,7 +431,8 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
return list;
}
-int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+/* Called within bdrv_dirty_bitmap_lock..unlock */
+int bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
int64_t sector)
{
if (bitmap) {
@@ -410,6 +442,18 @@ int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
}
}
+int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+ int64_t sector)
+{
+ bool dirty;
+
+ bdrv_dirty_bitmap_lock(bitmap);
+ dirty = bdrv_get_dirty_locked(bs, bitmap, sector);
+ bdrv_dirty_bitmap_unlock(bitmap);
+
+ return dirty;
+}
+
/**
* Chooses a default granularity based on the existing cluster size,
* but clamped between [4K, 64K]. Defaults to 64K in the case that there
@@ -474,23 +518,42 @@ int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter)
return hbitmap_iter_next(&iter->hbi);
}
+/* Called within bdrv_dirty_bitmap_lock..unlock */
+void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
+ int64_t cur_sector, int64_t nr_sectors)
+{
+ assert(bdrv_dirty_bitmap_enabled(bitmap));
+ hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
+}
+
void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int64_t cur_sector, int64_t nr_sectors)
{
+ bdrv_dirty_bitmap_lock(bitmap);
+ bdrv_set_dirty_bitmap_locked(bitmap, cur_sector, nr_sectors);
+ bdrv_dirty_bitmap_unlock(bitmap);
+}
+
+/* Called within bdrv_dirty_bitmap_lock..unlock */
+void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
+ int64_t cur_sector, int64_t nr_sectors)
+{
assert(bdrv_dirty_bitmap_enabled(bitmap));
- hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
+ hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
}
void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int64_t cur_sector, int64_t nr_sectors)
{
- assert(bdrv_dirty_bitmap_enabled(bitmap));
- hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
+ bdrv_dirty_bitmap_lock(bitmap);
+ bdrv_reset_dirty_bitmap_locked(bitmap, cur_sector, nr_sectors);
+ bdrv_dirty_bitmap_unlock(bitmap);
}
void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
{
assert(bdrv_dirty_bitmap_enabled(bitmap));
+ bdrv_dirty_bitmap_lock(bitmap);
if (!out) {
hbitmap_reset_all(bitmap->bitmap);
} else {
@@ -499,6 +562,7 @@ void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
hbitmap_granularity(backup));
*out = backup;
}
+ bdrv_dirty_bitmap_unlock(bitmap);
}
void bdrv_undo_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap *in)
diff --git a/block/mirror.c b/block/mirror.c
index dc227a2..6a5b0f8 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -344,10 +344,12 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
sector_num = bdrv_dirty_iter_next(s->dbi);
if (sector_num < 0) {
+ bdrv_dirty_bitmap_lock(s->dirty_bitmap);
bdrv_set_dirty_iter(s->dbi, 0);
sector_num = bdrv_dirty_iter_next(s->dbi);
trace_mirror_restart_iter(s, bdrv_get_dirty_count(s->dirty_bitmap));
assert(sector_num >= 0);
+ bdrv_dirty_bitmap_unlock(s->dirty_bitmap);
}
first_chunk = sector_num / sectors_per_chunk;
@@ -360,12 +362,13 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
/* Find the number of consective dirty chunks following the first dirty
* one, and wait for in flight requests in them. */
+ bdrv_dirty_bitmap_lock(s->dirty_bitmap);
while (nb_chunks * sectors_per_chunk < (s->buf_size >> BDRV_SECTOR_BITS)) {
int64_t next_dirty;
int64_t next_sector = sector_num + nb_chunks * sectors_per_chunk;
int64_t next_chunk = next_sector / sectors_per_chunk;
if (next_sector >= end ||
- !bdrv_get_dirty(source, s->dirty_bitmap, next_sector)) {
+ !bdrv_get_dirty_locked(source, s->dirty_bitmap, next_sector)) {
break;
}
if (test_bit(next_chunk, s->in_flight_bitmap)) {
@@ -386,8 +389,10 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
* calling bdrv_get_block_status_above could yield - if some blocks are
* marked dirty in this window, we need to know.
*/
- bdrv_reset_dirty_bitmap(s->dirty_bitmap, sector_num,
- nb_chunks * sectors_per_chunk);
+ bdrv_reset_dirty_bitmap_locked(s->dirty_bitmap, sector_num,
+ nb_chunks * sectors_per_chunk);
+ bdrv_dirty_bitmap_unlock(s->dirty_bitmap);
+
bitmap_set(s->in_flight_bitmap, sector_num / sectors_per_chunk, nb_chunks);
while (nb_chunks > 0 && sector_num < end) {
int64_t ret;
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 03db2cf..c264ead 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -598,8 +598,8 @@ struct BlockDriverState {
/* Writing to the list requires the BQL _and_ the dirty_bitmap_mutex.
* Reading from the list can be done with either the BQL or the
- * dirty_bitmap_mutex. Modifying a bitmap requires the AioContext
- * lock. */
+ * dirty_bitmap_mutex. Modifying a bitmap only requires
+ * dirty_bitmap_mutex. */
QemuMutex dirty_bitmap_mutex;
QLIST_HEAD(, BdrvDirtyBitmap) dirty_bitmaps;
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 9dea14b..b6fc35b 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -45,6 +45,9 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int bdrv_dirty_bitmap_get_meta(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap, int64_t sector,
int nb_sectors);
+int bdrv_dirty_bitmap_get_meta_locked(BlockDriverState *bs,
+ BdrvDirtyBitmap *bitmap, int64_t sector,
+ int nb_sectors);
void bdrv_dirty_bitmap_reset_meta(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap, int64_t sector,
int nb_sectors);
@@ -52,11 +55,6 @@ BdrvDirtyBitmapIter *bdrv_dirty_meta_iter_new(BdrvDirtyBitmap *bitmap);
BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap,
uint64_t first_sector);
void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter);
-int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter);
-void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t sector_num);
-int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap);
-int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap);
-void bdrv_dirty_bitmap_truncate(BlockDriverState *bs);
uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap,
uint64_t start, uint64_t count);
@@ -72,4 +70,19 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
bool finish);
void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
+/* Functions that require manual locking. */
+void bdrv_dirty_bitmap_lock(BdrvDirtyBitmap *bitmap);
+void bdrv_dirty_bitmap_unlock(BdrvDirtyBitmap *bitmap);
+int bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+ int64_t sector);
+void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
+ int64_t cur_sector, int64_t nr_sectors);
+void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
+ int64_t cur_sector, int64_t nr_sectors);
+int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter);
+void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t sector_num);
+int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap);
+int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap);
+void bdrv_dirty_bitmap_truncate(BlockDriverState *bs);
+
#endif
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex
2017-04-20 12:00 ` [Qemu-devel] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex Paolo Bonzini
@ 2017-04-20 14:42 ` Eric Blake
2017-05-04 8:05 ` Fam Zheng
2017-05-05 10:36 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2 siblings, 0 replies; 65+ messages in thread
From: Eric Blake @ 2017-04-20 14:42 UTC (permalink / raw)
To: Paolo Bonzini, qemu-devel; +Cc: qemu-block
[-- Attachment #1: Type: text/plain, Size: 849 bytes --]
On 04/20/2017 07:00 AM, Paolo Bonzini wrote:
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> block/dirty-bitmap.c | 74 +++++++++++++++++++++++++++++++++++++++++---
> block/mirror.c | 11 +++++--
> include/block/block_int.h | 4 +--
> include/block/dirty-bitmap.h | 23 +++++++++++---
> 4 files changed, 97 insertions(+), 15 deletions(-)
Fun conflicts with my pending patches to switch dirty-bitmap to be
byte-based instead of sector-based. I doubt the computer will be able to
resolve nicely, but I think rebasing by hand should be pretty doable, no
matter which of our patches lands first.
https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg02163.html
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex
2017-04-20 12:00 ` [Qemu-devel] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex Paolo Bonzini
2017-04-20 14:42 ` Eric Blake
@ 2017-05-04 8:05 ` Fam Zheng
2017-05-04 10:05 ` Paolo Bonzini
2017-05-05 10:36 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2 siblings, 1 reply; 65+ messages in thread
From: Fam Zheng @ 2017-05-04 8:05 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, 04/20 14:00, Paolo Bonzini wrote:
> @@ -400,7 +431,8 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
> return list;
> }
>
> -int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
> +/* Called within bdrv_dirty_bitmap_lock..unlock */
> +int bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
> int64_t sector)
Parameter indentation is off now.
> {
> if (bitmap) {
> @@ -410,6 +442,18 @@ int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
> }
> }
>
> diff --git a/include/block/block_int.h b/include/block/block_int.h
> index 03db2cf..c264ead 100644
> --- a/include/block/block_int.h
> +++ b/include/block/block_int.h
> @@ -598,8 +598,8 @@ struct BlockDriverState {
>
> /* Writing to the list requires the BQL _and_ the dirty_bitmap_mutex.
> * Reading from the list can be done with either the BQL or the
> - * dirty_bitmap_mutex. Modifying a bitmap requires the AioContext
> - * lock. */
> + * dirty_bitmap_mutex. Modifying a bitmap only requires
> + * dirty_bitmap_mutex. */
I'm confused by this comment. What's added in this patch is bitmap->mutex, not
dirty_bitmap_mutex. Is it a mistake?
> QemuMutex dirty_bitmap_mutex;
> QLIST_HEAD(, BdrvDirtyBitmap) dirty_bitmaps;
>
Fam
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex
2017-05-04 8:05 ` Fam Zheng
@ 2017-05-04 10:05 ` Paolo Bonzini
0 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-04 10:05 UTC (permalink / raw)
To: Fam Zheng; +Cc: qemu-devel, qemu-block
On 04/05/2017 10:05, Fam Zheng wrote:
>> /* Writing to the list requires the BQL _and_ the dirty_bitmap_mutex.
>> * Reading from the list can be done with either the BQL or the
>> - * dirty_bitmap_mutex. Modifying a bitmap requires the AioContext
>> - * lock. */
>> + * dirty_bitmap_mutex. Modifying a bitmap only requires
>> + * dirty_bitmap_mutex. */
> I'm confused by this comment. What's added in this patch is bitmap->mutex, not
> dirty_bitmap_mutex. Is it a mistake?
bitmap->mutex is a pointer that points to the (single) per-BDS mutex.
This patch changes the modification of bitmaps to require bitmap->mutex
(which points to bs->dirty_bitmap_mutex) instead of the AioContext lock.
Paolo
>> QemuMutex dirty_bitmap_mutex;
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex
2017-04-20 12:00 ` [Qemu-devel] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex Paolo Bonzini
2017-04-20 14:42 ` Eric Blake
2017-05-04 8:05 ` Fam Zheng
@ 2017-05-05 10:36 ` Stefan Hajnoczi
2017-05-05 10:47 ` Paolo Bonzini
2 siblings, 1 reply; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-05 10:36 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 1110 bytes --]
On Thu, Apr 20, 2017 at 02:00:57PM +0200, Paolo Bonzini wrote:
> @@ -410,6 +442,18 @@ int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
> }
> }
>
> +int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
> + int64_t sector)
Is it a good idea to offer an unlocked bdrv_get_dirty() API? It
encourages non-atomic access to the bitmap, e.g.
if (bdrv_get_dirty()) {
...do something outside the lock...
bdrv_reset_dirty_bitmap();
}
The unlocked API should be test-and-set/clear instead so that callers
automatically avoid race conditions.
> diff --git a/block/mirror.c b/block/mirror.c
> index dc227a2..6a5b0f8 100644
> --- a/block/mirror.c
> +++ b/block/mirror.c
> @@ -344,10 +344,12 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
>
> sector_num = bdrv_dirty_iter_next(s->dbi);
> if (sector_num < 0) {
> + bdrv_dirty_bitmap_lock(s->dirty_bitmap);
bdrv_dirty_iter_next() is listed under "functions that require manual
locking" but it's being called outside of the lock.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex
2017-05-05 10:36 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
@ 2017-05-05 10:47 ` Paolo Bonzini
2017-05-08 16:17 ` Stefan Hajnoczi
0 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-05 10:47 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: qemu-devel, qemu-block
On 05/05/2017 12:36, Stefan Hajnoczi wrote:
> On Thu, Apr 20, 2017 at 02:00:57PM +0200, Paolo Bonzini wrote:
>> @@ -410,6 +442,18 @@ int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
>> }
>> }
>>
>> +int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
>> + int64_t sector)
>
> Is it a good idea to offer an unlocked bdrv_get_dirty() API? It
> encourages non-atomic access to the bitmap, e.g.
>
> if (bdrv_get_dirty()) {
> ...do something outside the lock...
> bdrv_reset_dirty_bitmap();
> }
>
> The unlocked API should be test-and-set/clear instead so that callers
> automatically avoid race conditions.
I'm not sure it's possible to implement atomic test and clear for
HBitmap. But I can look into removing unlocked bdrv_get_dirty, the only
user is block migration.
>> diff --git a/block/mirror.c b/block/mirror.c
>> index dc227a2..6a5b0f8 100644
>> --- a/block/mirror.c
>> +++ b/block/mirror.c
>> @@ -344,10 +344,12 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
>>
>> sector_num = bdrv_dirty_iter_next(s->dbi);
>> if (sector_num < 0) {
>> + bdrv_dirty_bitmap_lock(s->dirty_bitmap);
>
> bdrv_dirty_iter_next() is listed under "functions that require manual
> locking" but it's being called outside of the lock.
Thanks, will fix.
Paolo
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex
2017-05-05 10:47 ` Paolo Bonzini
@ 2017-05-08 16:17 ` Stefan Hajnoczi
0 siblings, 0 replies; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-08 16:17 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]
On Fri, May 05, 2017 at 12:47:35PM +0200, Paolo Bonzini wrote:
>
>
> On 05/05/2017 12:36, Stefan Hajnoczi wrote:
> > On Thu, Apr 20, 2017 at 02:00:57PM +0200, Paolo Bonzini wrote:
> >> @@ -410,6 +442,18 @@ int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
> >> }
> >> }
> >>
> >> +int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
> >> + int64_t sector)
> >
> > Is it a good idea to offer an unlocked bdrv_get_dirty() API? It
> > encourages non-atomic access to the bitmap, e.g.
> >
> > if (bdrv_get_dirty()) {
> > ...do something outside the lock...
> > bdrv_reset_dirty_bitmap();
> > }
> >
> > The unlocked API should be test-and-set/clear instead so that callers
> > automatically avoid race conditions.
>
> I'm not sure it's possible to implement atomic test and clear for
> HBitmap. But I can look into removing unlocked bdrv_get_dirty, the only
> user is block migration.
Removing unlocked bdrv_get_dirty() is good.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* [Qemu-devel] [PATCH 17/17] block: make accounting thread-safe
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (15 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 16/17] block: protect modification of dirty bitmaps with a mutex Paolo Bonzini
@ 2017-04-20 12:00 ` Paolo Bonzini
2017-05-05 12:56 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-04-20 12:40 ` [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 no-reply
` (4 subsequent siblings)
21 siblings, 1 reply; 65+ messages in thread
From: Paolo Bonzini @ 2017-04-20 12:00 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
I'm not trying too hard yet. Later, with multiqueue support,
this may cause cacheline bouncing.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
block/accounting.c | 15 +++++++++++++++
include/block/accounting.h | 8 ++++++--
2 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/block/accounting.c b/block/accounting.c
index 3f457c4..dc10855 100644
--- a/block/accounting.c
+++ b/block/accounting.c
@@ -35,6 +35,7 @@ static const int qtest_latency_ns = NANOSECONDS_PER_SECOND / 1000;
void block_acct_init(BlockAcctStats *stats, bool account_invalid,
bool account_failed)
{
+ qemu_spin_init(&stats->spin);
stats->account_invalid = account_invalid;
stats->account_failed = account_failed;
@@ -58,12 +59,15 @@ void block_acct_add_interval(BlockAcctStats *stats, unsigned interval_length)
s = g_new0(BlockAcctTimedStats, 1);
s->interval_length = interval_length;
+ s->stats = stats;
+ qemu_spin_lock(&stats->spin);
QSLIST_INSERT_HEAD(&stats->intervals, s, entries);
for (i = 0; i < BLOCK_MAX_IOTYPE; i++) {
timed_average_init(&s->latency[i], clock_type,
(uint64_t) interval_length * NANOSECONDS_PER_SECOND);
}
+ qemu_spin_unlock(&stats->spin);
}
BlockAcctTimedStats *block_acct_interval_next(BlockAcctStats *stats,
@@ -98,6 +102,7 @@ void block_acct_done(BlockAcctStats *stats, BlockAcctCookie *cookie)
assert(cookie->type < BLOCK_MAX_IOTYPE);
+ qemu_spin_lock(&stats->spin);
stats->nr_bytes[cookie->type] += cookie->bytes;
stats->nr_ops[cookie->type]++;
stats->total_time_ns[cookie->type] += latency_ns;
@@ -106,12 +111,14 @@ void block_acct_done(BlockAcctStats *stats, BlockAcctCookie *cookie)
QSLIST_FOREACH(s, &stats->intervals, entries) {
timed_average_account(&s->latency[cookie->type], latency_ns);
}
+ qemu_spin_unlock(&stats->spin);
}
void block_acct_failed(BlockAcctStats *stats, BlockAcctCookie *cookie)
{
assert(cookie->type < BLOCK_MAX_IOTYPE);
+ qemu_spin_lock(&stats->spin);
stats->failed_ops[cookie->type]++;
if (stats->account_failed) {
@@ -130,6 +137,7 @@ void block_acct_failed(BlockAcctStats *stats, BlockAcctCookie *cookie)
timed_average_account(&s->latency[cookie->type], latency_ns);
}
}
+ qemu_spin_unlock(&stats->spin);
}
void block_acct_invalid(BlockAcctStats *stats, enum BlockAcctType type)
@@ -141,18 +149,23 @@ void block_acct_invalid(BlockAcctStats *stats, enum BlockAcctType type)
* invalid requests are accounted during their submission,
* therefore there's no actual I/O involved. */
+ qemu_spin_lock(&stats->spin);
stats->invalid_ops[type]++;
if (stats->account_invalid) {
stats->last_access_time_ns = qemu_clock_get_ns(clock_type);
}
+ qemu_spin_unlock(&stats->spin);
}
void block_acct_merge_done(BlockAcctStats *stats, enum BlockAcctType type,
int num_requests)
{
assert(type < BLOCK_MAX_IOTYPE);
+
+ qemu_spin_lock(&stats->spin);
stats->merged[type] += num_requests;
+ qemu_spin_unlock(&stats->spin);
}
int64_t block_acct_idle_time_ns(BlockAcctStats *stats)
@@ -167,7 +180,9 @@ double block_acct_queue_depth(BlockAcctTimedStats *stats,
assert(type < BLOCK_MAX_IOTYPE);
+ qemu_spin_lock(&stats->stats->spin);
sum = timed_average_sum(&stats->latency[type], &elapsed);
+ qemu_spin_unlock(&stats->stats->spin);
return (double) sum / elapsed;
}
diff --git a/include/block/accounting.h b/include/block/accounting.h
index 2089163..90b7a1d 100644
--- a/include/block/accounting.h
+++ b/include/block/accounting.h
@@ -26,8 +26,10 @@
#define BLOCK_ACCOUNTING_H
#include "qemu/timed-average.h"
+#include "qemu/thread.h"
typedef struct BlockAcctTimedStats BlockAcctTimedStats;
+typedef struct BlockAcctStats BlockAcctStats;
enum BlockAcctType {
BLOCK_ACCT_READ,
@@ -37,12 +39,14 @@ enum BlockAcctType {
};
struct BlockAcctTimedStats {
+ BlockAcctStats *stats;
TimedAverage latency[BLOCK_MAX_IOTYPE];
unsigned interval_length; /* in seconds */
QSLIST_ENTRY(BlockAcctTimedStats) entries;
};
-typedef struct BlockAcctStats {
+struct BlockAcctStats {
+ QemuSpin spin;
uint64_t nr_bytes[BLOCK_MAX_IOTYPE];
uint64_t nr_ops[BLOCK_MAX_IOTYPE];
uint64_t invalid_ops[BLOCK_MAX_IOTYPE];
@@ -53,7 +57,7 @@ typedef struct BlockAcctStats {
QSLIST_HEAD(, BlockAcctTimedStats) intervals;
bool account_invalid;
bool account_failed;
-} BlockAcctStats;
+};
typedef struct BlockAcctCookie {
int64_t bytes;
--
2.9.3
^ permalink raw reply related [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH 17/17] block: make accounting thread-safe
2017-04-20 12:00 ` [Qemu-devel] [PATCH 17/17] block: make accounting thread-safe Paolo Bonzini
@ 2017-05-05 12:56 ` Stefan Hajnoczi
0 siblings, 0 replies; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-05 12:56 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 438 bytes --]
On Thu, Apr 20, 2017 at 02:00:58PM +0200, Paolo Bonzini wrote:
> I'm not trying too hard yet. Later, with multiqueue support,
> this may cause cacheline bouncing.
This patch calls qemu_clock_get_ns() within a spinlock region. It's
probably just a memory load, rdtsc, and some math but more heavy-weight
code paths are possible too.
Taking the clock reading outside the lock region should be fine and
reduces the chance of contention.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (16 preceding siblings ...)
2017-04-20 12:00 ` [Qemu-devel] [PATCH 17/17] block: make accounting thread-safe Paolo Bonzini
@ 2017-04-20 12:40 ` no-reply
2017-04-20 12:42 ` no-reply
` (3 subsequent siblings)
21 siblings, 0 replies; 65+ messages in thread
From: no-reply @ 2017-04-20 12:40 UTC (permalink / raw)
To: pbonzini; +Cc: famz, qemu-devel, qemu-block
Hi,
This series failed automatic build test. Please find the testing commands and
their output below. If you have docker installed, you can probably reproduce it
locally.
Type: series
Message-id: 20170420120058.28404-1-pbonzini@redhat.com
Subject: [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1
=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
make docker-test-quick@centos6
make docker-test-mingw@fedora
make docker-test-build@min-glib
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
* [new tag] patchew/20170420120058.28404-1-pbonzini@redhat.com -> patchew/20170420120058.28404-1-pbonzini@redhat.com
Switched to a new branch 'test'
c716264 block: make accounting thread-safe
7d0704f block: protect modification of dirty bitmaps with a mutex
2259950 block: introduce dirty_bitmap_mutex
0532009 block: optimize access to reqs_lock
862fd98 coroutine-lock: introduce qemu_co_mutex_lock_unlock
5d9d699 block: protect tracked_requests and flush_queue with reqs_lock
a65d4b0 block: access write_gen with atomics
01ac75b block: use Stat64 for wr_highest_offset
628d1ba util: add stats64 module
ea6bb59 throttle-groups: protect throttled requests with a CoMutex
895a11d throttle-groups: do not use qemu_co_enter_next
42ce4b3 block: access io_plugged with atomic ops
bc2deb1 block: access wakeup with atomic ops
18d2d96 block: access serialising_in_flight with atomic ops
a11151d block: access io_limits_disabled with atomic ops
c4d7323 block: access quiesce_counter with atomic ops
9c192bd block: access copy_on_read with atomic ops
=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-f1fvdzwe/src/dtc'...
Submodule path 'dtc': checked out '558cd81bdd432769b59bff01240c44f82cfb1a9d'
BUILD centos6
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-f1fvdzwe/src'
ARCHIVE qemu.tgz
ARCHIVE dtc.tgz
COPY RUNNER
RUN test-quick in qemu:centos6
Packages installed:
SDL-devel-1.2.14-7.el6_7.1.x86_64
ccache-3.1.6-2.el6.x86_64
epel-release-6-8.noarch
gcc-4.4.7-17.el6.x86_64
git-1.7.1-4.el6_7.1.x86_64
glib2-devel-2.28.8-5.el6.x86_64
libfdt-devel-1.4.0-1.el6.x86_64
make-3.81-23.el6.x86_64
package g++ is not installed
pixman-devel-0.32.8-1.el6.x86_64
tar-1.23-15.el6_8.x86_64
zlib-devel-1.2.3-29.el6.x86_64
Environment variables:
PACKAGES=libfdt-devel ccache tar git make gcc g++ zlib-devel glib2-devel SDL-devel pixman-devel epel-release
HOSTNAME=6ae6c396f9d6
TERM=xterm
MAKEFLAGS= -j8
HISTSIZE=1000
J=8
USER=root
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
LANG=en_US.UTF-8
TARGET_LIST=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES= dtc
DEBUG=
G_BROKEN_FILENAMES=1
CCACHE_HASHDIR=
_=/usr/bin/env
Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/var/tmp/qemu-build/install
No C++ compiler available; disabling C++ specific optional code
Install prefix /var/tmp/qemu-build/install
BIOS directory /var/tmp/qemu-build/install/share/qemu
binary directory /var/tmp/qemu-build/install/bin
library directory /var/tmp/qemu-build/install/lib
module directory /var/tmp/qemu-build/install/lib/qemu
libexec directory /var/tmp/qemu-build/install/libexec
include directory /var/tmp/qemu-build/install/include
config directory /var/tmp/qemu-build/install/etc
local state directory /var/tmp/qemu-build/install/var
Manual directory /var/tmp/qemu-build/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path /tmp/qemu-test/src
C compiler cc
Host C compiler cc
C++ compiler
Objective-C compiler cc
ARFLAGS rv
CFLAGS -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g
QEMU_CFLAGS -I/usr/include/pixman-1 -I$(SRC_PATH)/dtc/libfdt -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -Wendif-labels -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all
LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g
make make
install install
python python -B
smbd /usr/sbin/smbd
module support no
host CPU x86_64
host big endian no
target list x86_64-softmmu aarch64-softmmu
tcg debug enabled no
gprof enabled no
sparse enabled no
strip binaries yes
profiler no
static build no
pixman system
SDL support yes (1.2.14)
GTK support no
GTK GL support no
VTE support no
TLS priority NORMAL
GNUTLS support no
GNUTLS rnd no
libgcrypt no
libgcrypt kdf no
nettle no
nettle kdf no
libtasn1 no
curses support no
virgl support no
curl support no
mingw32 support no
Audio drivers oss
Block whitelist (rw)
Block whitelist (ro)
VirtFS support no
VNC support yes
VNC SASL support no
VNC JPEG support no
VNC PNG support no
xen support no
brlapi support no
bluez support no
Documentation no
PIE yes
vde support no
netmap support no
Linux AIO support no
ATTR/XATTR support yes
Install blobs yes
KVM support yes
HAX support no
RDMA support no
TCG interpreter no
fdt support yes
preadv support yes
fdatasync yes
madvise yes
posix_madvise yes
libcap-ng support no
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
Trace backends log
spice support no
rbd support no
xfsctl support no
smartcard support no
libusb no
usb net redir no
OpenGL support no
OpenGL dmabufs no
libiscsi support no
libnfs support no
build guest agent yes
QGA VSS support no
QGA w32 disk info no
QGA MSI support no
seccomp support no
coroutine backend ucontext
coroutine pool yes
debug stack usage no
GlusterFS support no
gcov gcov
gcov enabled no
TPM support yes
libssh2 support no
TPM passthrough yes
QOM debugging yes
lzo support no
snappy support no
bzip2 support no
NUMA host support no
tcmalloc support no
jemalloc support no
avx2 optimization no
replication support yes
mkdir -p dtc/libfdt
GEN x86_64-softmmu/config-devices.mak.tmp
GEN aarch64-softmmu/config-devices.mak.tmp
GEN config-host.h
mkdir -p dtc/tests
GEN qemu-options.def
GEN qmp-commands.h
GEN qapi-visit.h
GEN qapi-types.h
GEN qapi-event.h
GEN x86_64-softmmu/config-devices.mak
GEN aarch64-softmmu/config-devices.mak
GEN qmp-marshal.c
GEN qapi-types.c
GEN qapi-visit.c
GEN qapi-event.c
GEN qmp-introspect.h
GEN qmp-introspect.c
GEN trace/generated-tcg-tracers.h
GEN trace/generated-helpers-wrappers.h
GEN trace/generated-helpers.h
GEN trace/generated-helpers.c
GEN module_block.h
GEN tests/test-qapi-types.h
GEN tests/test-qapi-visit.h
GEN tests/test-qmp-commands.h
GEN tests/test-qapi-event.h
GEN tests/test-qmp-introspect.h
GEN trace-root.h
GEN util/trace.h
GEN crypto/trace.h
GEN io/trace.h
GEN migration/trace.h
GEN block/trace.h
GEN backends/trace.h
GEN hw/block/trace.h
GEN hw/block/dataplane/trace.h
GEN hw/char/trace.h
GEN hw/intc/trace.h
GEN hw/net/trace.h
GEN hw/virtio/trace.h
GEN hw/audio/trace.h
GEN hw/misc/trace.h
GEN hw/usb/trace.h
GEN hw/scsi/trace.h
GEN hw/nvram/trace.h
GEN hw/display/trace.h
GEN hw/input/trace.h
GEN hw/timer/trace.h
GEN hw/dma/trace.h
GEN hw/sparc/trace.h
GEN hw/sd/trace.h
GEN hw/isa/trace.h
GEN hw/mem/trace.h
GEN hw/i386/trace.h
GEN hw/i386/xen/trace.h
GEN hw/9pfs/trace.h
GEN hw/ppc/trace.h
GEN hw/pci/trace.h
GEN hw/s390x/trace.h
GEN hw/vfio/trace.h
GEN hw/acpi/trace.h
GEN hw/arm/trace.h
GEN hw/alpha/trace.h
GEN hw/xen/trace.h
GEN ui/trace.h
GEN audio/trace.h
GEN net/trace.h
GEN target/arm/trace.h
GEN target/i386/trace.h
GEN target/mips/trace.h
GEN target/sparc/trace.h
GEN target/s390x/trace.h
GEN target/ppc/trace.h
GEN qom/trace.h
GEN linux-user/trace.h
GEN qapi/trace.h
GEN trace-root.c
GEN util/trace.c
GEN crypto/trace.c
GEN io/trace.c
GEN migration/trace.c
GEN block/trace.c
GEN backends/trace.c
GEN hw/block/trace.c
GEN hw/block/dataplane/trace.c
GEN hw/char/trace.c
GEN hw/intc/trace.c
GEN hw/net/trace.c
GEN hw/virtio/trace.c
GEN hw/audio/trace.c
GEN hw/misc/trace.c
GEN hw/usb/trace.c
GEN hw/scsi/trace.c
GEN hw/nvram/trace.c
GEN hw/display/trace.c
GEN hw/input/trace.c
GEN hw/timer/trace.c
GEN hw/dma/trace.c
GEN hw/sparc/trace.c
GEN hw/sd/trace.c
GEN hw/isa/trace.c
GEN hw/mem/trace.c
GEN hw/i386/trace.c
GEN hw/i386/xen/trace.c
GEN hw/9pfs/trace.c
GEN hw/ppc/trace.c
GEN hw/pci/trace.c
GEN hw/s390x/trace.c
GEN hw/vfio/trace.c
GEN hw/acpi/trace.c
GEN hw/arm/trace.c
GEN hw/alpha/trace.c
GEN hw/xen/trace.c
GEN ui/trace.c
GEN audio/trace.c
GEN net/trace.c
GEN target/arm/trace.c
GEN target/i386/trace.c
GEN target/mips/trace.c
GEN target/sparc/trace.c
GEN target/s390x/trace.c
GEN target/ppc/trace.c
GEN qom/trace.c
GEN linux-user/trace.c
GEN qapi/trace.c
GEN config-all-devices.mak
DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
DEP /tmp/qemu-test/src/dtc/tests/testutils.c
DEP /tmp/qemu-test/src/dtc/tests/trees.S
DEP /tmp/qemu-test/src/dtc/tests/value-labels.c
DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c
DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
DEP /tmp/qemu-test/src/dtc/tests/check_path.c
DEP /tmp/qemu-test/src/dtc/tests/overlay_bad_fixup.c
DEP /tmp/qemu-test/src/dtc/tests/overlay.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/property_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c
DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c
DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c
DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c
DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c
DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
DEP /tmp/qemu-test/src/dtc/tests/incbin.c
DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c
DEP /tmp/qemu-test/src/dtc/tests/path-references.c
DEP /tmp/qemu-test/src/dtc/tests/references.c
DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c
DEP /tmp/qemu-test/src/dtc/tests/del_node.c
DEP /tmp/qemu-test/src/dtc/tests/del_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop.c
DEP /tmp/qemu-test/src/dtc/tests/set_name.c
DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c
DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c
DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
DEP /tmp/qemu-test/src/dtc/tests/nop_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c
DEP /tmp/qemu-test/src/dtc/tests/stringlist.c
DEP /tmp/qemu-test/src/dtc/tests/addr_size_cells.c
DEP /tmp/qemu-test/src/dtc/tests/notfound.c
DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c
DEP /tmp/qemu-test/src/dtc/tests/char_literal.c
DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c
DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c
DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c
DEP /tmp/qemu-test/src/dtc/tests/get_path.c
DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/getprop.c
DEP /tmp/qemu-test/src/dtc/tests/get_name.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c
DEP /tmp/qemu-test/src/dtc/tests/find_property.c
DEP /tmp/qemu-test/src/dtc/tests/root_node.c
DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_overlay.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_addresses.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
DEP /tmp/qemu-test/src/dtc/util.c
DEP /tmp/qemu-test/src/dtc/fdtput.c
DEP /tmp/qemu-test/src/dtc/fdtget.c
DEP /tmp/qemu-test/src/dtc/fdtdump.c
LEX convert-dtsv0-lexer.lex.c
make[1]: flex: Command not found
DEP /tmp/qemu-test/src/dtc/srcpos.c
BISON dtc-parser.tab.c
make[1]: bison: Command not found
LEX dtc-lexer.lex.c
make[1]: flex: Command not found
DEP /tmp/qemu-test/src/dtc/treesource.c
DEP /tmp/qemu-test/src/dtc/livetree.c
DEP /tmp/qemu-test/src/dtc/fstree.c
DEP /tmp/qemu-test/src/dtc/flattree.c
DEP /tmp/qemu-test/src/dtc/dtc.c
DEP /tmp/qemu-test/src/dtc/data.c
DEP /tmp/qemu-test/src/dtc/checks.c
CHK version_gen.h
LEX convert-dtsv0-lexer.lex.c
make[1]: flex: Command not found BISON dtc-parser.tab.c
LEX dtc-lexer.lex.c
make[1]: bison: Command not found
make[1]: flex: Command not found
UPD version_gen.h
DEP /tmp/qemu-test/src/dtc/util.c
LEX convert-dtsv0-lexer.lex.c
BISON dtc-parser.tab.c
make[1]: flex: Command not found
LEX dtc-lexer.lex.c
make[1]: bison: Command not found
make[1]: flex: Command not found
CC libfdt/fdt.o
CC libfdt/fdt_ro.o
CC libfdt/fdt_wip.o
CC libfdt/fdt_rw.o
CC libfdt/fdt_strerror.o
CC libfdt/fdt_sw.o
CC libfdt/fdt_empty_tree.o
CC libfdt/fdt_addresses.o
CC libfdt/fdt_overlay.o
AR libfdt/libfdt.a
ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
LEX convert-dtsv0-lexer.lex.c
make[1]: flex: Command not found
BISON dtc-parser.tab.c
make[1]: bison: Command not found
LEX dtc-lexer.lex.c
make[1]: flex: Command not found
CC tests/qemu-iotests/socket_scm_helper.o
GEN qga/qapi-generated/qga-qapi-types.h
GEN qga/qapi-generated/qga-qapi-visit.h
GEN qga/qapi-generated/qga-qapi-visit.c
GEN qga/qapi-generated/qga-qmp-commands.h
GEN qga/qapi-generated/qga-qapi-types.c
CC trace-root.o
GEN qga/qapi-generated/qga-qmp-marshal.c
CC util/trace.o
CC crypto/trace.o
CC io/trace.o
CC migration/trace.o
CC block/trace.o
CC backends/trace.o
CC hw/block/trace.o
CC hw/block/dataplane/trace.o
CC hw/char/trace.o
CC hw/intc/trace.o
CC hw/net/trace.o
CC hw/virtio/trace.o
CC hw/audio/trace.o
CC hw/misc/trace.o
CC hw/usb/trace.o
CC hw/scsi/trace.o
CC hw/nvram/trace.o
CC hw/display/trace.o
CC hw/input/trace.o
CC hw/timer/trace.o
CC hw/dma/trace.o
CC hw/sparc/trace.o
CC hw/sd/trace.o
CC hw/isa/trace.o
CC hw/mem/trace.o
CC hw/i386/trace.o
CC hw/i386/xen/trace.o
CC hw/9pfs/trace.o
CC hw/ppc/trace.o
CC hw/pci/trace.o
CC hw/s390x/trace.o
CC hw/vfio/trace.o
CC hw/acpi/trace.o
CC hw/arm/trace.o
CC hw/alpha/trace.o
CC hw/xen/trace.o
CC ui/trace.o
CC audio/trace.o
CC net/trace.o
CC target/arm/trace.o
CC target/i386/trace.o
CC target/mips/trace.o
CC target/sparc/trace.o
CC target/s390x/trace.o
CC target/ppc/trace.o
CC qom/trace.o
CC linux-user/trace.o
CC qapi/trace.o
CC qmp-introspect.o
CC qapi-types.o
CC qapi-visit.o
CC qapi-event.o
CC qapi/qapi-visit-core.o
CC qapi/qapi-dealloc-visitor.o
CC qapi/qobject-input-visitor.o
CC qapi/qobject-output-visitor.o
CC qapi/qmp-registry.o
CC qapi/qmp-dispatch.o
CC qapi/string-input-visitor.o
CC qapi/string-output-visitor.o
CC qapi/opts-visitor.o
CC qapi/qapi-clone-visitor.o
CC qapi/qmp-event.o
CC qapi/qapi-util.o
CC qobject/qnull.o
CC qobject/qint.o
CC qobject/qstring.o
CC qobject/qdict.o
CC qobject/qfloat.o
CC qobject/qlist.o
CC qobject/qbool.o
CC qobject/qjson.o
CC qobject/qobject.o
CC qobject/json-lexer.o
CC qobject/json-streamer.o
CC qobject/json-parser.o
CC trace/control.o
CC trace/qmp.o
CC util/osdep.o
CC util/cutils.o
CC util/unicode.o
CC util/qemu-timer-common.o
CC util/bufferiszero.o
CC util/lockcnt.o
CC util/aiocb.o
CC util/async.o
CC util/thread-pool.o
CC util/qemu-timer.o
CC util/main-loop.o
CC util/iohandler.o
CC util/aio-posix.o
CC util/compatfd.o
CC util/event_notifier-posix.o
CC util/mmap-alloc.o
CC util/oslib-posix.o
CC util/qemu-openpty.o
CC util/qemu-thread-posix.o
CC util/memfd.o
CC util/envlist.o
CC util/path.o
CC util/module.o
CC util/host-utils.o
CC util/bitmap.o
CC util/bitops.o
CC util/hbitmap.o
CC util/fifo8.o
CC util/acl.o
CC util/error.o
CC util/qemu-error.o
CC util/id.o
CC util/iov.o
CC util/qemu-config.o
CC util/qemu-sockets.o
CC util/uri.o
CC util/notify.o
CC util/qemu-option.o
CC util/qemu-progress.o
CC util/keyval.o
CC util/hexdump.o
CC util/crc32c.o
CC util/uuid.o
CC util/throttle.o
CC util/readline.o
CC util/getauxval.o
CC util/rcu.o
CC util/qemu-coroutine.o
CC util/qemu-coroutine-lock.o
CC util/qemu-coroutine-io.o
CC util/qemu-coroutine-sleep.o
CC util/coroutine-ucontext.o
CC util/buffer.o
CC util/timed-average.o
CC util/base64.o
CC util/log.o
CC util/qdist.o
CC util/qht.o
CC util/range.o
CC util/stats64.o
CC util/systemd.o
CC crypto/pbkdf-stub.o
CC stubs/arch-query-cpu-def.o
CC stubs/arch-query-cpu-model-comparison.o
CC stubs/arch-query-cpu-model-expansion.o
CC stubs/arch-query-cpu-model-baseline.o
CC stubs/bdrv-next-monitor-owned.o
CC stubs/blk-commit-all.o
CC stubs/blockdev-close-all-bdrv-states.o
CC stubs/clock-warp.o
CC stubs/cpu-get-clock.o
CC stubs/cpu-get-icount.o
CC stubs/dump.o
CC stubs/error-printf.o
CC stubs/fdset.o
CC stubs/gdbstub.o
CC stubs/get-vm-name.o
CC stubs/iothread.o
CC stubs/iothread-lock.o
CC stubs/is-daemonized.o
CC stubs/machine-init-done.o
CC stubs/migr-blocker.o
CC stubs/monitor.o
CC stubs/qtest.o
CC stubs/notify-event.o
CC stubs/replay.o
CC stubs/runstate-check.o
CC stubs/set-fd-handler.o
CC stubs/slirp.o
CC stubs/sysbus.o
CC stubs/trace-control.o
CC stubs/uuid.o
CC stubs/vm-stop.o
CC stubs/vmstate.o
CC stubs/qmp_pc_dimm_device_list.o
CC stubs/target-monitor-defs.o
CC stubs/target-get-monitor-def.o
CC stubs/pc_madt_cpu_entry.o
CC stubs/vmgenid.o
CC contrib/ivshmem-client/ivshmem-client.o
CC contrib/ivshmem-client/main.o
CC contrib/ivshmem-server/ivshmem-server.o
CC contrib/ivshmem-server/main.o
CC qemu-nbd.o
CC block.o
CC blockjob.o
CC replication.o
CC qemu-io-cmds.o
CC block/raw-format.o
CC block/qcow.o
CC block/vdi.o
CC block/vmdk.o
CC block/cloop.o
CC block/bochs.o
CC block/vpc.o
CC block/vvfat.o
CC block/dmg.o
CC block/qcow2.o
CC block/qcow2-refcount.o
CC block/qcow2-cluster.o
CC block/qcow2-snapshot.o
CC block/qcow2-cache.o
CC block/qed.o
CC block/qed-gencb.o
CC block/qed-l2-cache.o
CC block/qed-table.o
CC block/qed-cluster.o
CC block/qed-check.o
CC block/vhdx.o
CC block/vhdx-endian.o
CC block/vhdx-log.o
CC block/quorum.o
CC block/parallels.o
CC block/blkdebug.o
CC block/blkverify.o
CC block/blkreplay.o
CC block/block-backend.o
CC block/snapshot.o
CC block/qapi.o
CC block/file-posix.o
CC block/null.o
CC block/mirror.o
CC block/commit.o
CC block/io.o
CC block/throttle-groups.o
CC block/nbd.o
CC block/nbd-client.o
CC block/sheepdog.o
CC block/accounting.o
CC block/dirty-bitmap.o
CC block/write-threshold.o
CC block/backup.o
CC block/replication.o
CC block/crypto.o
CC nbd/server.o
CC nbd/client.o
CC nbd/common.o
CC crypto/init.o
CC crypto/hash.o
CC crypto/hash-glib.o
CC crypto/hmac.o
CC crypto/hmac-glib.o
CC crypto/aes.o
CC crypto/desrfb.o
CC crypto/cipher.o
CC crypto/tlscreds.o
CC crypto/tlscredsanon.o
CC crypto/tlscredsx509.o
CC crypto/secret.o
CC crypto/tlssession.o
CC crypto/random-platform.o
CC crypto/pbkdf.o
CC crypto/ivgen.o
CC crypto/ivgen-essiv.o
CC crypto/ivgen-plain.o
CC crypto/ivgen-plain64.o
CC crypto/afsplit.o
CC crypto/xts.o
CC crypto/block.o
CC crypto/block-qcow.o
CC crypto/block-luks.o
CC io/channel.o
CC io/channel-buffer.o
CC io/channel-command.o
CC io/channel-socket.o
CC io/channel-file.o
CC io/channel-tls.o
CC io/channel-watch.o
CC io/channel-websock.o
CC io/channel-util.o
CC io/dns-resolver.o
CC io/task.o
CC qom/object.o
CC qom/container.o
CC qom/qom-qobject.o
CC qom/object_interfaces.o
GEN qemu-img-cmds.h
CC qemu-io.o
CC qemu-bridge-helper.o
CC blockdev.o
CC blockdev-nbd.o
CC iothread.o
CC qdev-monitor.o
CC device-hotplug.o
CC os-posix.o
CC page_cache.o
CC accel.o
CC bt-host.o
CC bt-vhci.o
CC dma-helpers.o
CC vl.o
CC tpm.o
CC device_tree.o
CC qmp-marshal.o
CC qmp.o
CC hmp.o
CC cpus-common.o
CC audio/audio.o
CC audio/noaudio.o
CC audio/wavaudio.o
CC audio/mixeng.o
CC audio/sdlaudio.o
CC audio/ossaudio.o
CC audio/wavcapture.o
CC backends/rng.o
CC backends/rng-egd.o
CC backends/rng-random.o
CC backends/msmouse.o
CC backends/wctablet.o
CC backends/testdev.o
CC backends/tpm.o
CC backends/hostmem.o
CC backends/hostmem-ram.o
CC backends/hostmem-file.o
CC backends/cryptodev.o
CC backends/cryptodev-builtin.o
CC block/stream.o
CC disas/arm.o
CC disas/i386.o
CC fsdev/qemu-fsdev-dummy.o
CC fsdev/qemu-fsdev-opts.o
CC fsdev/qemu-fsdev-throttle.o
CC hw/acpi/core.o
CC hw/acpi/piix4.o
CC hw/acpi/pcihp.o
CC hw/acpi/ich9.o
CC hw/acpi/tco.o
CC hw/acpi/cpu_hotplug.o
CC hw/acpi/memory_hotplug.o
CC hw/acpi/cpu.o
CC hw/acpi/nvdimm.o
CC hw/acpi/vmgenid.o
CC hw/acpi/acpi_interface.o
CC hw/acpi/bios-linker-loader.o
CC hw/acpi/aml-build.o
CC hw/acpi/ipmi.o
CC hw/acpi/acpi-stub.o
CC hw/acpi/ipmi-stub.o
CC hw/audio/sb16.o
CC hw/audio/es1370.o
CC hw/audio/ac97.o
CC hw/audio/fmopl.o
CC hw/audio/adlib.o
CC hw/audio/gus.o
CC hw/audio/gusemu_hal.o
CC hw/audio/gusemu_mixer.o
CC hw/audio/cs4231a.o
CC hw/audio/intel-hda.o
CC hw/audio/hda-codec.o
CC hw/audio/pcspk.o
CC hw/audio/wm8750.o
CC hw/audio/pl041.o
CC hw/audio/lm4549.o
CC hw/audio/marvell_88w8618.o
CC hw/block/block.o
CC hw/block/cdrom.o
CC hw/block/hd-geometry.o
CC hw/block/fdc.o
CC hw/block/m25p80.o
CC hw/block/nand.o
CC hw/block/pflash_cfi01.o
CC hw/block/pflash_cfi02.o
CC hw/block/ecc.o
CC hw/block/onenand.o
CC hw/block/nvme.o
CC hw/bt/core.o
CC hw/bt/l2cap.o
CC hw/bt/sdp.o
CC hw/bt/hci.o
CC hw/bt/hid.o
CC hw/bt/hci-csr.o
CC hw/char/ipoctal232.o
CC hw/char/parallel.o
CC hw/char/pl011.o
CC hw/char/serial.o
CC hw/char/serial-isa.o
CC hw/char/serial-pci.o
CC hw/char/virtio-console.o
CC hw/char/cadence_uart.o
CC hw/char/imx_serial.o
CC hw/char/debugcon.o
CC hw/core/qdev.o
CC hw/core/qdev-properties.o
CC hw/core/bus.o
CC hw/core/reset.o
CC hw/core/fw-path-provider.o
CC hw/core/irq.o
CC hw/core/hotplug.o
CC hw/core/ptimer.o
CC hw/core/sysbus.o
CC hw/core/machine.o
CC hw/core/loader.o
CC hw/core/qdev-properties-system.o
CC hw/core/register.o
CC hw/core/or-irq.o
CC hw/core/platform-bus.o
CC hw/display/ads7846.o
CC hw/display/cirrus_vga.o
CC hw/display/pl110.o
CC hw/display/ssd0303.o
CC hw/display/ssd0323.o
CC hw/display/vga-pci.o
CC hw/display/vga-isa.o
CC hw/display/vmware_vga.o
CC hw/display/blizzard.o
CC hw/display/exynos4210_fimd.o
CC hw/display/framebuffer.o
CC hw/display/tc6393xb.o
CC hw/dma/pl330.o
CC hw/dma/i8257.o
CC hw/dma/pl080.o
CC hw/dma/xlnx-zynq-devcfg.o
CC hw/gpio/max7310.o
CC hw/gpio/pl061.o
CC hw/gpio/zaurus.o
CC hw/gpio/gpio_key.o
CC hw/i2c/core.o
CC hw/i2c/smbus.o
CC hw/i2c/smbus_eeprom.o
CC hw/i2c/i2c-ddc.o
CC hw/i2c/versatile_i2c.o
CC hw/i2c/smbus_ich9.o
CC hw/i2c/pm_smbus.o
CC hw/i2c/bitbang_i2c.o
CC hw/i2c/exynos4210_i2c.o
CC hw/i2c/imx_i2c.o
CC hw/i2c/aspeed_i2c.o
CC hw/ide/core.o
CC hw/ide/atapi.o
CC hw/ide/qdev.o
CC hw/ide/pci.o
CC hw/ide/isa.o
CC hw/ide/piix.o
CC hw/ide/microdrive.o
CC hw/ide/ahci.o
CC hw/ide/ich.o
CC hw/input/hid.o
CC hw/input/lm832x.o
CC hw/input/pckbd.o
CC hw/input/pl050.o
CC hw/input/ps2.o
CC hw/input/stellaris_input.o
CC hw/input/tsc2005.o
CC hw/input/vmmouse.o
CC hw/input/virtio-input.o
CC hw/input/virtio-input-hid.o
CC hw/input/virtio-input-host.o
CC hw/intc/i8259_common.o
CC hw/intc/i8259.o
CC hw/intc/pl190.o
CC hw/intc/imx_avic.o
CC hw/intc/realview_gic.o
CC hw/intc/ioapic_common.o
CC hw/intc/arm_gic_common.o
CC hw/intc/arm_gic.o
CC hw/intc/arm_gicv2m.o
CC hw/intc/arm_gicv3_common.o
CC hw/intc/arm_gicv3.o
CC hw/intc/arm_gicv3_dist.o
CC hw/intc/arm_gicv3_redist.o
CC hw/intc/arm_gicv3_its_common.o
CC hw/intc/intc.o
CC hw/ipack/ipack.o
CC hw/ipack/tpci200.o
CC hw/ipmi/ipmi.o
CC hw/ipmi/ipmi_bmc_sim.o
CC hw/ipmi/ipmi_bmc_extern.o
CC hw/ipmi/isa_ipmi_kcs.o
CC hw/isa/isa-bus.o
CC hw/ipmi/isa_ipmi_bt.o
CC hw/isa/apm.o
CC hw/mem/pc-dimm.o
CC hw/mem/nvdimm.o
CC hw/misc/applesmc.o
CC hw/misc/max111x.o
CC hw/misc/tmp105.o
CC hw/misc/debugexit.o
CC hw/misc/sga.o
CC hw/misc/pc-testdev.o
CC hw/misc/pci-testdev.o
CC hw/misc/unimp.o
CC hw/misc/arm_l2x0.o
CC hw/misc/arm_integrator_debug.o
CC hw/misc/a9scu.o
CC hw/misc/arm11scu.o
CC hw/net/ne2000.o
CC hw/net/eepro100.o
CC hw/net/pcnet.o
CC hw/net/pcnet-pci.o
CC hw/net/e1000.o
CC hw/net/e1000x_common.o
CC hw/net/net_tx_pkt.o
CC hw/net/net_rx_pkt.o
CC hw/net/e1000e_core.o
CC hw/net/e1000e.o
CC hw/net/rtl8139.o
CC hw/net/vmxnet3.o
CC hw/net/smc91c111.o
CC hw/net/lan9118.o
CC hw/net/ne2000-isa.o
CC hw/net/xgmac.o
CC hw/net/allwinner_emac.o
CC hw/net/imx_fec.o
CC hw/net/cadence_gem.o
CC hw/net/stellaris_enet.o
CC hw/net/rocker/rocker.o
CC hw/net/rocker/rocker_fp.o
CC hw/net/rocker/rocker_desc.o
CC hw/net/rocker/rocker_world.o
CC hw/net/rocker/rocker_of_dpa.o
CC hw/nvram/eeprom93xx.o
CC hw/nvram/fw_cfg.o
CC hw/nvram/chrp_nvram.o
CC hw/pci-bridge/pci_bridge_dev.o
CC hw/pci-bridge/pcie_root_port.o
CC hw/pci-bridge/gen_pcie_root_port.o
CC hw/pci-bridge/pci_expander_bridge.o
CC hw/pci-bridge/xio3130_upstream.o
CC hw/pci-bridge/xio3130_downstream.o
CC hw/pci-bridge/ioh3420.o
CC hw/pci-bridge/i82801b11.o
CC hw/pci-host/pam.o
CC hw/pci-host/versatile.o
CC hw/pci-host/piix.o
CC hw/pci-host/q35.o
CC hw/pci-host/gpex.o
CC hw/pci/pci.o
CC hw/pci/pci_bridge.o
CC hw/pci/msix.o
CC hw/pci/msi.o
CC hw/pci/shpc.o
CC hw/pci/slotid_cap.o
CC hw/pci/pci_host.o
CC hw/pci/pcie_host.o
CC hw/pci/pcie.o
CC hw/pci/pcie_aer.o
CC hw/pci/pcie_port.o
CC hw/pci/pci-stub.o
CC hw/pcmcia/pcmcia.o
CC hw/scsi/scsi-disk.o
CC hw/scsi/scsi-generic.o
CC hw/scsi/scsi-bus.o
CC hw/scsi/lsi53c895a.o
CC hw/scsi/mptsas.o
CC hw/scsi/mptconfig.o
CC hw/scsi/mptendian.o
CC hw/scsi/megasas.o
CC hw/scsi/vmw_pvscsi.o
CC hw/scsi/esp.o
CC hw/scsi/esp-pci.o
CC hw/sd/pl181.o
CC hw/sd/ssi-sd.o
CC hw/sd/sd.o
CC hw/sd/core.o
CC hw/sd/sdhci.o
CC hw/smbios/smbios.o
CC hw/smbios/smbios_type_38.o
CC hw/smbios/smbios-stub.o
CC hw/smbios/smbios_type_38-stub.o
CC hw/ssi/pl022.o
CC hw/ssi/ssi.o
CC hw/ssi/xilinx_spips.o
CC hw/ssi/aspeed_smc.o
CC hw/ssi/stm32f2xx_spi.o
CC hw/timer/arm_timer.o
CC hw/timer/arm_mptimer.o
CC hw/timer/armv7m_systick.o
CC hw/timer/a9gtimer.o
CC hw/timer/cadence_ttc.o
CC hw/timer/ds1338.o
CC hw/timer/hpet.o
CC hw/timer/i8254_common.o
CC hw/timer/i8254.o
CC hw/timer/pl031.o
CC hw/timer/imx_epit.o
CC hw/timer/twl92230.o
CC hw/timer/imx_gpt.o
CC hw/timer/stm32f2xx_timer.o
CC hw/timer/aspeed_timer.o
CC hw/tpm/tpm_tis.o
CC hw/tpm/tpm_passthrough.o
CC hw/tpm/tpm_util.o
CC hw/usb/core.o
CC hw/usb/combined-packet.o
CC hw/usb/bus.o
CC hw/usb/libhw.o
CC hw/usb/desc.o
CC hw/usb/desc-msos.o
CC hw/usb/hcd-uhci.o
CC hw/usb/hcd-ohci.o
CC hw/usb/hcd-ehci.o
CC hw/usb/hcd-ehci-pci.o
CC hw/usb/hcd-ehci-sysbus.o
CC hw/usb/hcd-xhci.o
CC hw/usb/hcd-musb.o
CC hw/usb/dev-hub.o
CC hw/usb/dev-hid.o
CC hw/usb/dev-wacom.o
CC hw/usb/dev-storage.o
CC hw/usb/dev-uas.o
CC hw/usb/dev-audio.o
CC hw/usb/dev-serial.o
CC hw/usb/dev-bluetooth.o
CC hw/usb/dev-network.o
CC hw/usb/dev-smartcard-reader.o
CC hw/usb/dev-mtp.o
CC hw/usb/host-stub.o
CC hw/virtio/virtio-rng.o
CC hw/virtio/virtio-pci.o
CC hw/virtio/virtio-bus.o
CC hw/virtio/virtio-mmio.o
CC hw/virtio/vhost-stub.o
CC hw/watchdog/watchdog.o
CC hw/watchdog/wdt_i6300esb.o
CC hw/watchdog/wdt_ib700.o
CC hw/watchdog/wdt_aspeed.o
CC migration/migration.o
CC migration/socket.o
CC migration/fd.o
CC migration/exec.o
CC migration/tls.o
CC migration/colo-comm.o
CC migration/colo.o
CC migration/colo-failover.o
CC migration/vmstate.o
CC migration/qemu-file.o
CC migration/qemu-file-channel.o
CC migration/xbzrle.o
CC migration/postcopy-ram.o
CC migration/qjson.o
CC migration/block.o
CC net/net.o
CC net/queue.o
CC net/checksum.o
CC net/util.o
CC net/hub.o
CC net/socket.o
CC net/dump.o
CC net/eth.o
CC net/l2tpv3.o
CC net/vhost-user.o
CC net/tap.o
CC net/tap-linux.o
CC net/slirp.o
CC net/filter.o
CC net/filter-buffer.o
CC net/filter-mirror.o
CC net/colo-compare.o
CC net/colo.o
CC net/filter-rewriter.o
CC net/filter-replay.o
CC qom/cpu.o
CC replay/replay.o
CC replay/replay-internal.o
CC replay/replay-events.o
CC replay/replay-time.o
CC replay/replay-input.o
CC replay/replay-char.o
/tmp/qemu-test/src/replay/replay-internal.c: In function ‘replay_put_array’:
/tmp/qemu-test/src/replay/replay-internal.c:65: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
CC replay/replay-snapshot.o
CC replay/replay-net.o
CC replay/replay-audio.o
CC slirp/cksum.o
CC slirp/if.o
CC slirp/ip_icmp.o
CC slirp/ip6_icmp.o
CC slirp/ip6_input.o
CC slirp/ip6_output.o
CC slirp/ip_input.o
CC slirp/ip_output.o
CC slirp/dnssearch.o
CC slirp/dhcpv6.o
CC slirp/slirp.o
CC slirp/mbuf.o
CC slirp/misc.o
CC slirp/sbuf.o
CC slirp/socket.o
CC slirp/tcp_input.o
CC slirp/tcp_output.o
CC slirp/tcp_subr.o
CC slirp/tcp_timer.o
CC slirp/udp.o
CC slirp/udp6.o
CC slirp/bootp.o
CC slirp/tftp.o
/tmp/qemu-test/src/slirp/tcp_input.c: In function ‘tcp_input’:
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_p’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_len’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_tos’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_id’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_off’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_ttl’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_sum’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_src.s_addr’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_dst.s_addr’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:220: warning: ‘save_ip6.ip_nh’ may be used uninitialized in this function
CC slirp/arp_table.o
CC slirp/ndp_table.o
CC ui/keymaps.o
CC ui/console.o
CC ui/cursor.o
CC ui/qemu-pixman.o
CC ui/input.o
CC ui/input-keymap.o
CC ui/input-legacy.o
CC ui/input-linux.o
CC ui/sdl.o
CC ui/x_keymap.o
CC ui/vnc.o
CC ui/sdl_zoom.o
CC ui/vnc-enc-hextile.o
CC ui/vnc-enc-zlib.o
CC ui/vnc-enc-tight.o
CC ui/vnc-palette.o
CC ui/vnc-enc-zrle.o
CC ui/vnc-auth-vencrypt.o
CC ui/vnc-ws.o
CC ui/vnc-jobs.o
CC chardev/char.o
CC chardev/char-fd.o
CC chardev/char-file.o
CC chardev/char-io.o
CC chardev/char-mux.o
CC chardev/char-null.o
CC chardev/char-parallel.o
CC chardev/char-pipe.o
CC chardev/char-pty.o
CC chardev/char-ringbuf.o
CC chardev/char-serial.o
CC chardev/char-socket.o
CC chardev/char-stdio.o
CC chardev/char-udp.o
LINK tests/qemu-iotests/socket_scm_helper
AS optionrom/multiboot.o
AS optionrom/linuxboot.o
CC optionrom/linuxboot_dma.o
cc: unrecognized option '-no-integrated-as'
cc: unrecognized option '-no-integrated-as'
AS optionrom/kvmvapic.o
CC qga/commands.o
BUILD optionrom/multiboot.img
BUILD optionrom/linuxboot.img
BUILD optionrom/linuxboot_dma.img
BUILD optionrom/kvmvapic.img
BUILD optionrom/multiboot.raw
BUILD optionrom/linuxboot.raw
BUILD optionrom/linuxboot_dma.raw
BUILD optionrom/kvmvapic.raw
CC qga/guest-agent-command-state.o
SIGN optionrom/multiboot.bin
CC qga/main.o
CC qga/commands-posix.o
CC qga/channel-posix.o
SIGN optionrom/linuxboot.bin
CC qga/qapi-generated/qga-qapi-types.o
SIGN optionrom/linuxboot_dma.bin
SIGN optionrom/kvmvapic.bin
CC qga/qapi-generated/qga-qapi-visit.o
CC qga/qapi-generated/qga-qmp-marshal.o
AR libqemuutil.a
AR libqemustub.a
CC qemu-img.o
LINK qemu-io
LINK qemu-bridge-helper
LINK ivshmem-client
LINK ivshmem-server
LINK qemu-nbd
LINK qemu-ga
GEN x86_64-softmmu/config-target.h
GEN x86_64-softmmu/hmp-commands-info.h
GEN x86_64-softmmu/hmp-commands.h
GEN aarch64-softmmu/hmp-commands-info.h
GEN aarch64-softmmu/config-target.h
GEN aarch64-softmmu/hmp-commands.h
CC x86_64-softmmu/translate-common.o
CC x86_64-softmmu/cpu-exec-common.o
CC x86_64-softmmu/cpu-exec.o
CC x86_64-softmmu/translate-all.o
CC x86_64-softmmu/exec.o
CC x86_64-softmmu/tcg/tcg.o
CC x86_64-softmmu/tcg/tcg-op.o
CC aarch64-softmmu/exec.o
CC aarch64-softmmu/translate-all.o
CC aarch64-softmmu/cpu-exec.o
CC aarch64-softmmu/translate-common.o
CC x86_64-softmmu/tcg/optimize.o
LINK qemu-img
CC aarch64-softmmu/cpu-exec-common.o
CC aarch64-softmmu/tcg/tcg.o
CC aarch64-softmmu/tcg/tcg-op.o
CC x86_64-softmmu/tcg/tcg-common.o
CC x86_64-softmmu/fpu/softfloat.o
CC x86_64-softmmu/disas.o
CC x86_64-softmmu/tcg-runtime.o
CC aarch64-softmmu/tcg/optimize.o
CC x86_64-softmmu/hax-stub.o
CC x86_64-softmmu/arch_init.o
CC aarch64-softmmu/tcg/tcg-common.o
CC x86_64-softmmu/cpus.o
CC aarch64-softmmu/fpu/softfloat.o
CC aarch64-softmmu/disas.o
CC aarch64-softmmu/tcg-runtime.o
GEN aarch64-softmmu/gdbstub-xml.c
CC aarch64-softmmu/hax-stub.o
CC aarch64-softmmu/kvm-stub.o
CC x86_64-softmmu/monitor.o
CC x86_64-softmmu/gdbstub.o
CC x86_64-softmmu/balloon.o
CC aarch64-softmmu/arch_init.o
CC aarch64-softmmu/cpus.o
CC aarch64-softmmu/monitor.o
CC x86_64-softmmu/ioport.o
CC aarch64-softmmu/gdbstub.o
CC aarch64-softmmu/balloon.o
CC x86_64-softmmu/numa.o
CC x86_64-softmmu/qtest.o
CC x86_64-softmmu/bootdevice.o
CC aarch64-softmmu/ioport.o
CC aarch64-softmmu/numa.o
CC aarch64-softmmu/qtest.o
CC x86_64-softmmu/kvm-all.o
CC aarch64-softmmu/bootdevice.o
CC x86_64-softmmu/memory.o
CC aarch64-softmmu/memory.o
CC aarch64-softmmu/cputlb.o
CC x86_64-softmmu/cputlb.o
CC aarch64-softmmu/memory_mapping.o
CC aarch64-softmmu/dump.o
CC aarch64-softmmu/migration/ram.o
CC x86_64-softmmu/memory_mapping.o
CC aarch64-softmmu/migration/savevm.o
CC aarch64-softmmu/xen-common-stub.o
CC aarch64-softmmu/hw/adc/stm32f2xx_adc.o
CC aarch64-softmmu/xen-hvm-stub.o
CC x86_64-softmmu/dump.o
CC x86_64-softmmu/migration/ram.o
CC aarch64-softmmu/hw/block/virtio-blk.o
CC aarch64-softmmu/hw/block/dataplane/virtio-blk.o
CC aarch64-softmmu/hw/char/exynos4210_uart.o
CC aarch64-softmmu/hw/char/omap_uart.o
CC aarch64-softmmu/hw/char/digic-uart.o
CC aarch64-softmmu/hw/char/stm32f2xx_usart.o
CC aarch64-softmmu/hw/char/bcm2835_aux.o
CC x86_64-softmmu/migration/savevm.o
CC aarch64-softmmu/hw/char/virtio-serial-bus.o
CC aarch64-softmmu/hw/core/nmi.o
CC aarch64-softmmu/hw/core/generic-loader.o
CC aarch64-softmmu/hw/core/null-machine.o
CC aarch64-softmmu/hw/cpu/arm11mpcore.o
CC aarch64-softmmu/hw/cpu/realview_mpcore.o
CC aarch64-softmmu/hw/cpu/a9mpcore.o
CC aarch64-softmmu/hw/cpu/a15mpcore.o
CC aarch64-softmmu/hw/cpu/core.o
CC x86_64-softmmu/xen-common-stub.o
CC aarch64-softmmu/hw/display/omap_dss.o
CC aarch64-softmmu/hw/display/omap_lcdc.o
CC aarch64-softmmu/hw/display/pxa2xx_lcd.o
CC aarch64-softmmu/hw/display/bcm2835_fb.o
CC aarch64-softmmu/hw/display/vga.o
CC x86_64-softmmu/xen-hvm-stub.o
CC aarch64-softmmu/hw/display/virtio-gpu.o
CC aarch64-softmmu/hw/display/virtio-gpu-3d.o
CC aarch64-softmmu/hw/display/virtio-gpu-pci.o
CC x86_64-softmmu/hw/block/virtio-blk.o
CC x86_64-softmmu/hw/block/dataplane/virtio-blk.o
CC aarch64-softmmu/hw/display/dpcd.o
CC aarch64-softmmu/hw/display/xlnx_dp.o
CC aarch64-softmmu/hw/dma/xlnx_dpdma.o
CC aarch64-softmmu/hw/dma/omap_dma.o
CC aarch64-softmmu/hw/dma/soc_dma.o
CC x86_64-softmmu/hw/char/virtio-serial-bus.o
CC aarch64-softmmu/hw/dma/pxa2xx_dma.o
CC x86_64-softmmu/hw/core/nmi.o
CC aarch64-softmmu/hw/dma/bcm2835_dma.o
CC aarch64-softmmu/hw/gpio/omap_gpio.o
CC aarch64-softmmu/hw/gpio/imx_gpio.o
CC aarch64-softmmu/hw/gpio/bcm2835_gpio.o
CC aarch64-softmmu/hw/i2c/omap_i2c.o
CC aarch64-softmmu/hw/input/pxa2xx_keypad.o
CC x86_64-softmmu/hw/core/generic-loader.o
CC aarch64-softmmu/hw/input/tsc210x.o
CC aarch64-softmmu/hw/intc/armv7m_nvic.o
CC aarch64-softmmu/hw/intc/exynos4210_gic.o
CC x86_64-softmmu/hw/core/null-machine.o
CC aarch64-softmmu/hw/intc/exynos4210_combiner.o
CC x86_64-softmmu/hw/cpu/core.o
CC aarch64-softmmu/hw/intc/omap_intc.o
CC x86_64-softmmu/hw/display/vga.o
CC aarch64-softmmu/hw/intc/bcm2835_ic.o
CC x86_64-softmmu/hw/display/virtio-gpu.o
CC aarch64-softmmu/hw/intc/bcm2836_control.o
CC aarch64-softmmu/hw/intc/allwinner-a10-pic.o
CC aarch64-softmmu/hw/intc/aspeed_vic.o
CC aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
CC aarch64-softmmu/hw/misc/ivshmem.o
CC aarch64-softmmu/hw/misc/arm_sysctl.o
CC aarch64-softmmu/hw/misc/cbus.o
CC aarch64-softmmu/hw/misc/exynos4210_pmu.o
CC aarch64-softmmu/hw/misc/exynos4210_clk.o
CC aarch64-softmmu/hw/misc/imx_ccm.o
CC aarch64-softmmu/hw/misc/imx31_ccm.o
CC x86_64-softmmu/hw/display/virtio-gpu-3d.o
CC aarch64-softmmu/hw/misc/imx25_ccm.o
CC aarch64-softmmu/hw/misc/imx6_ccm.o
CC x86_64-softmmu/hw/display/virtio-gpu-pci.o
CC x86_64-softmmu/hw/display/virtio-vga.o
CC x86_64-softmmu/hw/intc/apic.o
CC x86_64-softmmu/hw/intc/apic_common.o
CC aarch64-softmmu/hw/misc/imx6_src.o
CC aarch64-softmmu/hw/misc/mst_fpga.o
CC aarch64-softmmu/hw/misc/omap_clk.o
CC aarch64-softmmu/hw/misc/omap_gpmc.o
CC x86_64-softmmu/hw/intc/ioapic.o
CC x86_64-softmmu/hw/isa/lpc_ich9.o
CC aarch64-softmmu/hw/misc/omap_l4.o
CC x86_64-softmmu/hw/misc/vmport.o
CC x86_64-softmmu/hw/misc/ivshmem.o
CC x86_64-softmmu/hw/misc/pvpanic.o
CC x86_64-softmmu/hw/misc/edu.o
CC x86_64-softmmu/hw/misc/hyperv_testdev.o
CC aarch64-softmmu/hw/misc/omap_sdrc.o
CC aarch64-softmmu/hw/misc/omap_tap.o
CC x86_64-softmmu/hw/net/virtio-net.o
CC x86_64-softmmu/hw/net/vhost_net.o
CC x86_64-softmmu/hw/scsi/virtio-scsi.o
CC x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
CC x86_64-softmmu/hw/scsi/vhost-scsi.o
CC x86_64-softmmu/hw/timer/mc146818rtc.o
CC aarch64-softmmu/hw/misc/bcm2835_mbox.o
CC x86_64-softmmu/hw/vfio/common.o
CC aarch64-softmmu/hw/misc/bcm2835_property.o
CC aarch64-softmmu/hw/misc/bcm2835_rng.o
CC aarch64-softmmu/hw/misc/zynq_slcr.o
CC aarch64-softmmu/hw/misc/zynq-xadc.o
CC x86_64-softmmu/hw/vfio/pci.o
CC x86_64-softmmu/hw/vfio/pci-quirks.o
CC aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
CC aarch64-softmmu/hw/misc/edu.o
CC aarch64-softmmu/hw/misc/auxbus.o
CC aarch64-softmmu/hw/misc/aspeed_scu.o
CC x86_64-softmmu/hw/vfio/platform.o
CC aarch64-softmmu/hw/misc/aspeed_sdmc.o
CC aarch64-softmmu/hw/net/virtio-net.o
CC aarch64-softmmu/hw/net/vhost_net.o
CC aarch64-softmmu/hw/pcmcia/pxa2xx.o
CC aarch64-softmmu/hw/scsi/virtio-scsi.o
CC x86_64-softmmu/hw/vfio/spapr.o
CC aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
CC x86_64-softmmu/hw/virtio/virtio.o
CC aarch64-softmmu/hw/scsi/vhost-scsi.o
CC aarch64-softmmu/hw/sd/omap_mmc.o
CC x86_64-softmmu/hw/virtio/virtio-balloon.o
CC x86_64-softmmu/hw/virtio/vhost.o
CC x86_64-softmmu/hw/virtio/vhost-backend.o
CC aarch64-softmmu/hw/sd/pxa2xx_mmci.o
CC aarch64-softmmu/hw/sd/bcm2835_sdhost.o
CC x86_64-softmmu/hw/virtio/vhost-user.o
CC x86_64-softmmu/hw/virtio/vhost-vsock.o
CC x86_64-softmmu/hw/virtio/virtio-crypto.o
CC aarch64-softmmu/hw/ssi/omap_spi.o
CC aarch64-softmmu/hw/ssi/imx_spi.o
CC x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
CC x86_64-softmmu/hw/i386/multiboot.o
CC aarch64-softmmu/hw/timer/exynos4210_mct.o
CC x86_64-softmmu/hw/i386/pc.o
CC x86_64-softmmu/hw/i386/pc_piix.o
CC aarch64-softmmu/hw/timer/exynos4210_pwm.o
CC aarch64-softmmu/hw/timer/exynos4210_rtc.o
CC x86_64-softmmu/hw/i386/pc_q35.o
CC x86_64-softmmu/hw/i386/pc_sysfw.o
CC x86_64-softmmu/hw/i386/x86-iommu.o
CC aarch64-softmmu/hw/timer/omap_gptimer.o
CC aarch64-softmmu/hw/timer/omap_synctimer.o
CC aarch64-softmmu/hw/timer/pxa2xx_timer.o
CC x86_64-softmmu/hw/i386/intel_iommu.o
CC x86_64-softmmu/hw/i386/amd_iommu.o
CC x86_64-softmmu/hw/i386/kvmvapic.o
/tmp/qemu-test/src/hw/i386/pc_piix.c: In function ‘igd_passthrough_isa_bridge_create’:
/tmp/qemu-test/src/hw/i386/pc_piix.c:1055: warning: ‘pch_rev_id’ may be used uninitialized in this function
CC x86_64-softmmu/hw/i386/acpi-build.o
CC aarch64-softmmu/hw/timer/digic-timer.o
CC aarch64-softmmu/hw/timer/allwinner-a10-pit.o
CC aarch64-softmmu/hw/usb/tusb6010.o
CC aarch64-softmmu/hw/vfio/common.o
CC aarch64-softmmu/hw/vfio/pci.o
CC aarch64-softmmu/hw/vfio/pci-quirks.o
CC aarch64-softmmu/hw/vfio/platform.o
CC aarch64-softmmu/hw/vfio/calxeda-xgmac.o
CC x86_64-softmmu/hw/i386/pci-assign-load-rom.o
CC aarch64-softmmu/hw/vfio/amd-xgbe.o
CC x86_64-softmmu/hw/i386/kvm/clock.o
CC aarch64-softmmu/hw/vfio/spapr.o
/tmp/qemu-test/src/hw/i386/acpi-build.c: In function ‘build_append_pci_bus_devices’:
/tmp/qemu-test/src/hw/i386/acpi-build.c:496: warning: ‘notify_method’ may be used uninitialized in this function
CC x86_64-softmmu/hw/i386/kvm/apic.o
CC x86_64-softmmu/hw/i386/kvm/i8259.o
CC aarch64-softmmu/hw/virtio/virtio.o
CC aarch64-softmmu/hw/virtio/virtio-balloon.o
CC x86_64-softmmu/hw/i386/kvm/ioapic.o
CC x86_64-softmmu/hw/i386/kvm/i8254.o
CC aarch64-softmmu/hw/virtio/vhost.o
CC aarch64-softmmu/hw/virtio/vhost-backend.o
CC x86_64-softmmu/hw/i386/kvm/pci-assign.o
CC x86_64-softmmu/target/i386/translate.o
CC x86_64-softmmu/target/i386/helper.o
CC x86_64-softmmu/target/i386/cpu.o
CC aarch64-softmmu/hw/virtio/vhost-user.o
CC aarch64-softmmu/hw/virtio/vhost-vsock.o
CC aarch64-softmmu/hw/virtio/virtio-crypto.o
CC aarch64-softmmu/hw/virtio/virtio-crypto-pci.o
CC aarch64-softmmu/hw/arm/boot.o
CC x86_64-softmmu/target/i386/bpt_helper.o
CC x86_64-softmmu/target/i386/excp_helper.o
CC aarch64-softmmu/hw/arm/collie.o
CC aarch64-softmmu/hw/arm/exynos4_boards.o
CC x86_64-softmmu/target/i386/fpu_helper.o
CC aarch64-softmmu/hw/arm/gumstix.o
CC aarch64-softmmu/hw/arm/highbank.o
CC aarch64-softmmu/hw/arm/digic_boards.o
CC aarch64-softmmu/hw/arm/integratorcp.o
CC aarch64-softmmu/hw/arm/mainstone.o
CC aarch64-softmmu/hw/arm/musicpal.o
CC aarch64-softmmu/hw/arm/nseries.o
CC aarch64-softmmu/hw/arm/omap_sx1.o
CC aarch64-softmmu/hw/arm/palm.o
CC aarch64-softmmu/hw/arm/realview.o
CC aarch64-softmmu/hw/arm/spitz.o
CC aarch64-softmmu/hw/arm/stellaris.o
CC aarch64-softmmu/hw/arm/tosa.o
CC x86_64-softmmu/target/i386/cc_helper.o
CC x86_64-softmmu/target/i386/int_helper.o
CC x86_64-softmmu/target/i386/svm_helper.o
CC aarch64-softmmu/hw/arm/versatilepb.o
CC aarch64-softmmu/hw/arm/vexpress.o
CC aarch64-softmmu/hw/arm/virt.o
CC aarch64-softmmu/hw/arm/xilinx_zynq.o
CC x86_64-softmmu/target/i386/smm_helper.o
CC aarch64-softmmu/hw/arm/z2.o
CC aarch64-softmmu/hw/arm/virt-acpi-build.o
CC aarch64-softmmu/hw/arm/netduino2.o
CC aarch64-softmmu/hw/arm/sysbus-fdt.o
CC aarch64-softmmu/hw/arm/armv7m.o
CC aarch64-softmmu/hw/arm/exynos4210.o
CC aarch64-softmmu/hw/arm/pxa2xx.o
CC aarch64-softmmu/hw/arm/pxa2xx_gpio.o
CC aarch64-softmmu/hw/arm/pxa2xx_pic.o
CC aarch64-softmmu/hw/arm/digic.o
CC aarch64-softmmu/hw/arm/omap1.o
CC aarch64-softmmu/hw/arm/omap2.o
CC aarch64-softmmu/hw/arm/strongarm.o
CC aarch64-softmmu/hw/arm/allwinner-a10.o
CC aarch64-softmmu/hw/arm/cubieboard.o
CC aarch64-softmmu/hw/arm/bcm2835_peripherals.o
CC x86_64-softmmu/target/i386/misc_helper.o
CC x86_64-softmmu/target/i386/mem_helper.o
CC aarch64-softmmu/hw/arm/bcm2836.o
CC x86_64-softmmu/target/i386/seg_helper.o
CC aarch64-softmmu/hw/arm/raspi.o
CC x86_64-softmmu/target/i386/mpx_helper.o
CC x86_64-softmmu/target/i386/gdbstub.o
CC aarch64-softmmu/hw/arm/stm32f205_soc.o
CC aarch64-softmmu/hw/arm/xlnx-zynqmp.o
CC aarch64-softmmu/hw/arm/xlnx-ep108.o
CC aarch64-softmmu/hw/arm/fsl-imx25.o
CC aarch64-softmmu/hw/arm/imx25_pdk.o
CC aarch64-softmmu/hw/arm/fsl-imx31.o
CC aarch64-softmmu/hw/arm/kzm.o
CC aarch64-softmmu/hw/arm/fsl-imx6.o
CC x86_64-softmmu/target/i386/machine.o
CC aarch64-softmmu/hw/arm/sabrelite.o
CC x86_64-softmmu/target/i386/arch_memory_mapping.o
CC x86_64-softmmu/target/i386/arch_dump.o
CC aarch64-softmmu/hw/arm/aspeed_soc.o
CC x86_64-softmmu/target/i386/monitor.o
CC aarch64-softmmu/hw/arm/aspeed.o
CC aarch64-softmmu/target/arm/arm-semi.o
CC aarch64-softmmu/target/arm/machine.o
CC x86_64-softmmu/target/i386/kvm.o
CC aarch64-softmmu/target/arm/psci.o
CC x86_64-softmmu/target/i386/hyperv.o
CC aarch64-softmmu/target/arm/arch_dump.o
CC aarch64-softmmu/target/arm/monitor.o
GEN trace/generated-helpers.c
CC aarch64-softmmu/target/arm/kvm-stub.o
CC aarch64-softmmu/target/arm/translate.o
CC aarch64-softmmu/target/arm/op_helper.o
CC aarch64-softmmu/target/arm/helper.o
CC aarch64-softmmu/target/arm/cpu.o
CC aarch64-softmmu/target/arm/neon_helper.o
CC aarch64-softmmu/target/arm/iwmmxt_helper.o
CC aarch64-softmmu/target/arm/gdbstub.o
CC aarch64-softmmu/target/arm/cpu64.o
CC x86_64-softmmu/trace/control-target.o
CC aarch64-softmmu/target/arm/translate-a64.o
/tmp/qemu-test/src/target/arm/translate-a64.c: In function ‘handle_shri_with_rndacc’:
/tmp/qemu-test/src/target/arm/translate-a64.c:6359: warning: ‘tcg_src_hi’ may be used uninitialized in this function
/tmp/qemu-test/src/target/arm/translate-a64.c: In function ‘disas_simd_scalar_two_reg_misc’:
/tmp/qemu-test/src/target/arm/translate-a64.c:8086: warning: ‘rmode’ may be used uninitialized in this function
CC aarch64-softmmu/target/arm/helper-a64.o
CC aarch64-softmmu/target/arm/gdbstub64.o
CC aarch64-softmmu/target/arm/crypto_helper.o
CC aarch64-softmmu/target/arm/arm-powerctl.o
GEN trace/generated-helpers.c
CC aarch64-softmmu/trace/control-target.o
CC aarch64-softmmu/gdbstub-xml.o
CC aarch64-softmmu/trace/generated-helpers.o
LINK aarch64-softmmu/qemu-system-aarch64
CC x86_64-softmmu/trace/generated-helpers.o
LINK x86_64-softmmu/qemu-system-x86_64
LEX convert-dtsv0-lexer.lex.c
BISON dtc-parser.tab.c
make[1]: flex: Command not found
LEX dtc-lexer.lex.c
make[1]: bison: Command not found
make[1]: flex: Command not found
TEST tests/qapi-schema/alternate-any.out
TEST tests/qapi-schema/alternate-conflict-dict.out
TEST tests/qapi-schema/alternate-array.out
TEST tests/qapi-schema/alternate-base.out
TEST tests/qapi-schema/alternate-conflict-string.out
TEST tests/qapi-schema/alternate-clash.out
TEST tests/qapi-schema/alternate-empty.out
TEST tests/qapi-schema/alternate-nested.out
TEST tests/qapi-schema/alternate-unknown.out
TEST tests/qapi-schema/args-alternate.out
TEST tests/qapi-schema/args-any.out
TEST tests/qapi-schema/args-array-empty.out
TEST tests/qapi-schema/args-array-unknown.out
TEST tests/qapi-schema/args-bad-boxed.out
TEST tests/qapi-schema/args-boxed-anon.out
TEST tests/qapi-schema/args-boxed-empty.out
TEST tests/qapi-schema/args-boxed-string.out
TEST tests/qapi-schema/args-int.out
TEST tests/qapi-schema/args-invalid.out
TEST tests/qapi-schema/args-member-array-bad.out
TEST tests/qapi-schema/args-member-case.out
TEST tests/qapi-schema/args-member-unknown.out
TEST tests/qapi-schema/args-name-clash.out
TEST tests/qapi-schema/args-union.out
TEST tests/qapi-schema/args-unknown.out
TEST tests/qapi-schema/bad-base.out
TEST tests/qapi-schema/bad-data.out
TEST tests/qapi-schema/bad-ident.out
TEST tests/qapi-schema/bad-type-bool.out
TEST tests/qapi-schema/bad-type-dict.out
TEST tests/qapi-schema/bad-type-int.out
TEST tests/qapi-schema/base-cycle-direct.out
TEST tests/qapi-schema/base-cycle-indirect.out
TEST tests/qapi-schema/command-int.out
TEST tests/qapi-schema/comments.out
TEST tests/qapi-schema/doc-bad-alternate-member.out
TEST tests/qapi-schema/doc-bad-command-arg.out
TEST tests/qapi-schema/doc-bad-symbol.out
TEST tests/qapi-schema/doc-bad-union-member.out
TEST tests/qapi-schema/doc-before-include.out
TEST tests/qapi-schema/doc-before-pragma.out
TEST tests/qapi-schema/doc-duplicated-return.out
TEST tests/qapi-schema/doc-duplicated-arg.out
TEST tests/qapi-schema/doc-duplicated-since.out
TEST tests/qapi-schema/doc-empty-arg.out
TEST tests/qapi-schema/doc-empty-section.out
TEST tests/qapi-schema/doc-empty-symbol.out
TEST tests/qapi-schema/doc-good.out
TEST tests/qapi-schema/doc-invalid-end.out
TEST tests/qapi-schema/doc-interleaved-section.out
TEST tests/qapi-schema/doc-invalid-end2.out
TEST tests/qapi-schema/doc-invalid-return.out
TEST tests/qapi-schema/doc-invalid-section.out
TEST tests/qapi-schema/doc-invalid-start.out
TEST tests/qapi-schema/doc-missing.out
TEST tests/qapi-schema/doc-missing-colon.out
TEST tests/qapi-schema/doc-missing-expr.out
TEST tests/qapi-schema/doc-missing-space.out
TEST tests/qapi-schema/doc-no-symbol.out
TEST tests/qapi-schema/double-data.out
TEST tests/qapi-schema/double-type.out
TEST tests/qapi-schema/duplicate-key.out
TEST tests/qapi-schema/empty.out
TEST tests/qapi-schema/enum-bad-name.out
TEST tests/qapi-schema/enum-bad-prefix.out
TEST tests/qapi-schema/enum-clash-member.out
TEST tests/qapi-schema/enum-dict-member.out
TEST tests/qapi-schema/enum-int-member.out
TEST tests/qapi-schema/enum-member-case.out
TEST tests/qapi-schema/enum-missing-data.out
TEST tests/qapi-schema/enum-wrong-data.out
TEST tests/qapi-schema/escape-outside-string.out
TEST tests/qapi-schema/escape-too-big.out
TEST tests/qapi-schema/escape-too-short.out
TEST tests/qapi-schema/event-boxed-empty.out
TEST tests/qapi-schema/event-case.out
TEST tests/qapi-schema/event-nest-struct.out
TEST tests/qapi-schema/flat-union-array-branch.out
TEST tests/qapi-schema/flat-union-bad-base.out
TEST tests/qapi-schema/flat-union-bad-discriminator.out
TEST tests/qapi-schema/flat-union-base-any.out
TEST tests/qapi-schema/flat-union-base-union.out
TEST tests/qapi-schema/flat-union-clash-member.out
TEST tests/qapi-schema/flat-union-empty.out
TEST tests/qapi-schema/flat-union-incomplete-branch.out
TEST tests/qapi-schema/flat-union-inline.out
TEST tests/qapi-schema/flat-union-int-branch.out
TEST tests/qapi-schema/flat-union-invalid-branch-key.out
TEST tests/qapi-schema/flat-union-invalid-discriminator.out
TEST tests/qapi-schema/flat-union-no-base.out
TEST tests/qapi-schema/flat-union-optional-discriminator.out
TEST tests/qapi-schema/flat-union-string-discriminator.out
TEST tests/qapi-schema/funny-char.out
TEST tests/qapi-schema/ident-with-escape.out
TEST tests/qapi-schema/include-before-err.out
TEST tests/qapi-schema/include-cycle.out
TEST tests/qapi-schema/include-extra-junk.out
TEST tests/qapi-schema/include-format-err.out
TEST tests/qapi-schema/include-nested-err.out
TEST tests/qapi-schema/include-no-file.out
TEST tests/qapi-schema/include-non-file.out
TEST tests/qapi-schema/include-relpath.out
TEST tests/qapi-schema/include-repetition.out
TEST tests/qapi-schema/include-self-cycle.out
TEST tests/qapi-schema/include-simple.out
TEST tests/qapi-schema/indented-expr.out
TEST tests/qapi-schema/leading-comma-list.out
TEST tests/qapi-schema/leading-comma-object.out
TEST tests/qapi-schema/missing-colon.out
TEST tests/qapi-schema/missing-comma-list.out
TEST tests/qapi-schema/missing-comma-object.out
TEST tests/qapi-schema/missing-type.out
TEST tests/qapi-schema/nested-struct-data.out
TEST tests/qapi-schema/non-objects.out
TEST tests/qapi-schema/pragma-doc-required-crap.out
TEST tests/qapi-schema/pragma-extra-junk.out
TEST tests/qapi-schema/pragma-name-case-whitelist-crap.out
TEST tests/qapi-schema/pragma-non-dict.out
TEST tests/qapi-schema/pragma-returns-whitelist-crap.out
TEST tests/qapi-schema/qapi-schema-test.out
TEST tests/qapi-schema/quoted-structural-chars.out
TEST tests/qapi-schema/redefined-builtin.out
TEST tests/qapi-schema/redefined-command.out
TEST tests/qapi-schema/redefined-event.out
TEST tests/qapi-schema/redefined-type.out
TEST tests/qapi-schema/reserved-command-q.out
TEST tests/qapi-schema/reserved-enum-q.out
TEST tests/qapi-schema/reserved-member-has.out
TEST tests/qapi-schema/reserved-member-q.out
TEST tests/qapi-schema/reserved-member-u.out
TEST tests/qapi-schema/reserved-member-underscore.out
TEST tests/qapi-schema/reserved-type-kind.out
TEST tests/qapi-schema/returns-alternate.out
TEST tests/qapi-schema/reserved-type-list.out
TEST tests/qapi-schema/returns-array-bad.out
TEST tests/qapi-schema/returns-dict.out
TEST tests/qapi-schema/returns-unknown.out
TEST tests/qapi-schema/returns-whitelist.out
TEST tests/qapi-schema/struct-base-clash-deep.out
TEST tests/qapi-schema/struct-base-clash.out
TEST tests/qapi-schema/struct-data-invalid.out
TEST tests/qapi-schema/struct-member-invalid.out
TEST tests/qapi-schema/trailing-comma-list.out
TEST tests/qapi-schema/trailing-comma-object.out
TEST tests/qapi-schema/type-bypass-bad-gen.out
TEST tests/qapi-schema/unclosed-list.out
TEST tests/qapi-schema/unclosed-object.out
TEST tests/qapi-schema/unclosed-string.out
TEST tests/qapi-schema/unicode-str.out
TEST tests/qapi-schema/union-base-empty.out
TEST tests/qapi-schema/union-base-no-discriminator.out
TEST tests/qapi-schema/union-branch-case.out
TEST tests/qapi-schema/union-clash-branches.out
TEST tests/qapi-schema/union-empty.out
TEST tests/qapi-schema/union-invalid-base.out
TEST tests/qapi-schema/union-optional-branch.out
TEST tests/qapi-schema/union-unknown.out
TEST tests/qapi-schema/unknown-escape.out
TEST tests/qapi-schema/unknown-expr-key.out
GEN tests/qapi-schema/doc-good.test.texi
CC tests/check-qdict.o
CC tests/test-char.o
CC tests/check-qfloat.o
CC tests/check-qint.o
CC tests/check-qstring.o
CC tests/check-qlist.o
CC tests/check-qnull.o
CC tests/check-qjson.o
CC tests/test-qobject-output-visitor.o
GEN tests/test-qapi-visit.c
GEN tests/test-qapi-types.c
GEN tests/test-qapi-event.c
GEN tests/test-qmp-introspect.c
CC tests/test-clone-visitor.o
CC tests/test-qobject-input-visitor.o
CC tests/test-qmp-commands.o
GEN tests/test-qmp-marshal.c
CC tests/test-string-input-visitor.o
CC tests/test-string-output-visitor.o
CC tests/test-qmp-event.o
CC tests/test-opts-visitor.o
CC tests/iothread.o
CC tests/test-visitor-serialization.o
CC tests/test-coroutine.o
CC tests/test-iov.o
CC tests/test-aio.o
CC tests/test-aio-multithread.o
CC tests/test-throttle.o
CC tests/test-thread-pool.o
CC tests/test-hbitmap.o
CC tests/test-blockjob.o
CC tests/test-blockjob-txn.o
CC tests/test-x86-cpuid.o
CC tests/test-xbzrle.o
CC tests/test-vmstate.o
CC tests/test-cutils.o
CC tests/test-shift128.o
CC tests/test-mul64.o
CC tests/test-int128.o
CC tests/rcutorture.o
CC tests/test-rcu-list.o
CC tests/test-qdist.o
CC tests/test-qht.o
/tmp/qemu-test/src/tests/test-int128.c:180: warning: ‘__noclone__’ attribute directive ignored
CC tests/test-qht-par.o
CC tests/qht-bench.o
CC tests/test-bitops.o
CC tests/test-bitcnt.o
CC tests/check-qom-interface.o
CC tests/check-qom-proplist.o
CC tests/test-qemu-opts.o
CC tests/test-keyval.o
CC tests/test-write-threshold.o
CC tests/test-crypto-hash.o
CC tests/test-crypto-hmac.o
CC tests/test-crypto-cipher.o
CC tests/test-crypto-secret.o
CC tests/test-qga.o
CC tests/libqtest.o
CC tests/test-timed-average.o
CC tests/test-io-task.o
CC tests/test-io-channel-socket.o
CC tests/io-channel-helpers.o
CC tests/test-io-channel-file.o
CC tests/test-io-channel-command.o
CC tests/test-io-channel-buffer.o
CC tests/test-base64.o
CC tests/test-crypto-ivgen.o
CC tests/test-crypto-afsplit.o
CC tests/test-crypto-xts.o
CC tests/test-crypto-block.o
CC tests/test-logging.o
CC tests/test-replication.o
CC tests/test-bufferiszero.o
CC tests/test-uuid.o
CC tests/ptimer-test.o
CC tests/ptimer-test-stubs.o
CC tests/test-qapi-util.o
CC tests/vhost-user-test.o
CC tests/libqos/pci.o
CC tests/libqos/fw_cfg.o
CC tests/libqos/malloc.o
CC tests/libqos/i2c.o
CC tests/libqos/libqos.o
CC tests/libqos/malloc-spapr.o
CC tests/libqos/rtas.o
CC tests/libqos/libqos-spapr.o
CC tests/libqos/pci-pc.o
CC tests/libqos/pci-spapr.o
CC tests/libqos/malloc-pc.o
CC tests/libqos/libqos-pc.o
CC tests/libqos/ahci.o
CC tests/libqos/virtio.o
CC tests/libqos/virtio-pci.o
CC tests/libqos/malloc-generic.o
CC tests/libqos/virtio-mmio.o
CC tests/endianness-test.o
CC tests/fdc-test.o
CC tests/ide-test.o
CC tests/ahci-test.o
/tmp/qemu-test/src/tests/ide-test.c: In function ‘cdrom_pio_impl’:
/tmp/qemu-test/src/tests/ide-test.c:803: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
/tmp/qemu-test/src/tests/ide-test.c: In function ‘test_cdrom_dma’:
/tmp/qemu-test/src/tests/ide-test.c:899: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
CC tests/hd-geo-test.o
CC tests/boot-order-test.o
CC tests/bios-tables-test.o
CC tests/acpi-utils.o
CC tests/boot-sector.o
CC tests/boot-serial-test.o
CC tests/pxe-test.o
CC tests/rtc-test.o
CC tests/ipmi-kcs-test.o
CC tests/ipmi-bt-test.o
CC tests/i440fx-test.o
CC tests/fw_cfg-test.o
CC tests/drive_del-test.o
CC tests/wdt_ib700-test.o
CC tests/tco-test.o
CC tests/e1000-test.o
CC tests/e1000e-test.o
CC tests/rtl8139-test.o
CC tests/pcnet-test.o
CC tests/eepro100-test.o
CC tests/ne2000-test.o
CC tests/nvme-test.o
CC tests/ac97-test.o
CC tests/es1370-test.o
CC tests/virtio-net-test.o
CC tests/virtio-blk-test.o
CC tests/virtio-balloon-test.o
CC tests/virtio-rng-test.o
CC tests/virtio-scsi-test.o
CC tests/virtio-serial-test.o
CC tests/tpci200-test.o
CC tests/virtio-console-test.o
CC tests/ipoctal232-test.o
CC tests/display-vga-test.o
CC tests/intel-hda-test.o
CC tests/ivshmem-test.o
CC tests/vmxnet3-test.o
CC tests/pvpanic-test.o
CC tests/i82801b11-test.o
CC tests/ioh3420-test.o
CC tests/usb-hcd-ohci-test.o
CC tests/libqos/usb.o
CC tests/usb-hcd-uhci-test.o
CC tests/usb-hcd-ehci-test.o
CC tests/usb-hcd-xhci-test.o
CC tests/pc-cpu-test.o
CC tests/q35-test.o
CC tests/test-netfilter.o
CC tests/test-filter-mirror.o
CC tests/test-filter-redirector.o
CC tests/postcopy-test.o
CC tests/test-x86-cpuid-compat.o
CC tests/qmp-test.o
CC tests/device-introspect-test.o
CC tests/qom-test.o
LINK tests/check-qdict
LINK tests/test-char
LINK tests/check-qfloat
LINK tests/check-qint
LINK tests/check-qstring
LINK tests/check-qlist
LINK tests/check-qnull
LINK tests/check-qjson
CC tests/test-qapi-visit.o
CC tests/test-qapi-types.o
CC tests/test-qapi-event.o
CC tests/test-qmp-marshal.o
CC tests/test-qmp-introspect.o
LINK tests/test-coroutine
LINK tests/test-visitor-serialization
LINK tests/test-iov
LINK tests/test-aio
LINK tests/test-aio-multithread
LINK tests/test-throttle
LINK tests/test-thread-pool
LINK tests/test-hbitmap
LINK tests/test-blockjob
LINK tests/test-blockjob-txn
LINK tests/test-x86-cpuid
LINK tests/test-xbzrle
LINK tests/test-vmstate
LINK tests/test-cutils
LINK tests/test-shift128
LINK tests/test-mul64
LINK tests/test-int128
LINK tests/rcutorture
LINK tests/test-rcu-list
LINK tests/test-qdist
LINK tests/test-qht
LINK tests/qht-bench
LINK tests/test-bitops
LINK tests/test-bitcnt
LINK tests/check-qom-interface
LINK tests/check-qom-proplist
LINK tests/test-qemu-opts
LINK tests/test-keyval
LINK tests/test-write-threshold
LINK tests/test-crypto-hash
LINK tests/test-crypto-hmac
LINK tests/test-crypto-cipher
LINK tests/test-crypto-secret
LINK tests/test-qga
LINK tests/test-timed-average
LINK tests/test-io-task
LINK tests/test-io-channel-socket
LINK tests/test-io-channel-file
LINK tests/test-io-channel-command
LINK tests/test-io-channel-buffer
LINK tests/test-base64
LINK tests/test-crypto-ivgen
LINK tests/test-crypto-afsplit
LINK tests/test-crypto-xts
LINK tests/test-crypto-block
LINK tests/test-logging
LINK tests/test-replication
LINK tests/test-bufferiszero
LINK tests/test-uuid
LINK tests/ptimer-test
LINK tests/test-qapi-util
LINK tests/vhost-user-test
LINK tests/endianness-test
LINK tests/fdc-test
LINK tests/ide-test
LINK tests/ahci-test
LINK tests/hd-geo-test
LINK tests/boot-order-test
LINK tests/bios-tables-test
LINK tests/boot-serial-test
LINK tests/pxe-test
LINK tests/rtc-test
LINK tests/ipmi-kcs-test
LINK tests/ipmi-bt-test
LINK tests/i440fx-test
LINK tests/fw_cfg-test
LINK tests/drive_del-test
LINK tests/wdt_ib700-test
LINK tests/tco-test
LINK tests/e1000-test
LINK tests/e1000e-test
LINK tests/rtl8139-test
LINK tests/pcnet-test
LINK tests/eepro100-test
LINK tests/ne2000-test
LINK tests/nvme-test
LINK tests/ac97-test
LINK tests/es1370-test
LINK tests/virtio-net-test
LINK tests/virtio-balloon-test
LINK tests/virtio-blk-test
LINK tests/virtio-rng-test
LINK tests/virtio-scsi-test
LINK tests/virtio-serial-test
LINK tests/virtio-console-test
LINK tests/tpci200-test
LINK tests/ipoctal232-test
LINK tests/display-vga-test
LINK tests/intel-hda-test
LINK tests/ivshmem-test
LINK tests/vmxnet3-test
LINK tests/pvpanic-test
LINK tests/i82801b11-test
LINK tests/ioh3420-test
LINK tests/usb-hcd-ohci-test
LINK tests/usb-hcd-uhci-test
LINK tests/usb-hcd-ehci-test
LINK tests/usb-hcd-xhci-test
LINK tests/pc-cpu-test
LINK tests/q35-test
LINK tests/test-netfilter
LINK tests/test-filter-mirror
LINK tests/test-filter-redirector
LINK tests/postcopy-test
LINK tests/test-x86-cpuid-compat
LINK tests/qmp-test
LINK tests/device-introspect-test
LINK tests/qom-test
GTESTER tests/check-qdict
GTESTER tests/test-char
GTESTER tests/check-qfloat
GTESTER tests/check-qint
GTESTER tests/check-qstring
GTESTER tests/check-qlist
GTESTER tests/check-qnull
GTESTER tests/check-qjson
LINK tests/test-qobject-output-visitor
LINK tests/test-clone-visitor
LINK tests/test-qobject-input-visitor
LINK tests/test-qmp-commands
LINK tests/test-string-input-visitor
LINK tests/test-string-output-visitor
LINK tests/test-qmp-event
LINK tests/test-opts-visitor
GTESTER tests/test-aio-multithread
GTESTER tests/test-iov
GTESTER tests/test-coroutine
GTESTER tests/test-thread-pool
GTESTER tests/test-throttle
GTESTER tests/test-visitor-serialization
GTESTER tests/test-hbitmap
GTESTER tests/test-aio
GTESTER tests/test-blockjob
GTESTER tests/test-blockjob-txn
GTESTER tests/test-x86-cpuid
GTESTER tests/test-xbzrle
GTESTER tests/test-vmstate
GTESTER tests/test-cutils
Failed to load simple/primitive:b_1
Failed to load simple/primitive:i64_2
Failed to load simple/primitive:i32_1
Failed to load simple/primitive:i32_1
Failed to load test/with_tmp:a
Failed to load test/tmp_child_parent:f
Failed to load test/tmp_child:parent
Failed to load test/with_tmp:tmp
Failed to load test/tmp_child:diff
Failed to load test/with_tmp:tmp
Failed to load test/tmp_child:diff
Failed to load test/with_tmp:tmp
GTESTER tests/test-shift128
GTESTER tests/test-mul64
GTESTER tests/rcutorture
GTESTER tests/test-int128
GTESTER tests/test-rcu-list
GTESTER tests/test-qdist
GTESTER tests/test-qht
LINK tests/test-qht-par
GTESTER tests/test-bitops
GTESTER tests/test-bitcnt
GTESTER tests/check-qom-interface
GTESTER tests/check-qom-proplist
GTESTER tests/test-qemu-opts
GTESTER tests/test-keyval
GTESTER tests/test-write-threshold
GTESTER tests/test-crypto-hash
GTESTER tests/test-crypto-hmac
GTESTER tests/test-crypto-cipher
GTESTER tests/test-crypto-secret
GTESTER tests/test-qga
GTESTER tests/test-timed-average
GTESTER tests/test-io-task
GTESTER tests/test-io-channel-socket
GTESTER tests/test-io-channel-file
GTESTER tests/test-io-channel-command
GTESTER tests/test-io-channel-buffer
GTESTER tests/test-base64
GTESTER tests/test-crypto-ivgen
GTESTER tests/test-crypto-afsplit
GTESTER tests/test-crypto-xts
GTESTER tests/test-crypto-block
GTESTER tests/test-logging
GTESTER tests/test-replication
GTESTER tests/test-bufferiszero
GTESTER tests/test-uuid
GTESTER tests/ptimer-test
GTESTER tests/test-qapi-util
GTESTER check-qtest-x86_64
GTESTER check-qtest-aarch64
GTESTER tests/test-qobject-output-visitor
GTESTER tests/test-clone-visitor
GTESTER tests/test-qobject-input-visitor
GTESTER tests/test-qmp-commands
GTESTER tests/test-string-input-visitor
GTESTER tests/test-string-output-visitor
GTESTER tests/test-qmp-event
GTESTER tests/test-opts-visitor
GTESTER tests/test-qht-par
Vhost user backend fails to broadcast fake RARP
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-f1fvdzwe/src'
BUILD fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-f1fvdzwe/src'
ARCHIVE qemu.tgz
ARCHIVE dtc.tgz
COPY RUNNER
RUN test-mingw in qemu:fedora
Packages installed:
PyYAML-3.11-13.fc25.x86_64
SDL-devel-1.2.15-21.fc24.x86_64
bc-1.06.95-16.fc24.x86_64
bison-3.0.4-4.fc24.x86_64
ccache-3.3.4-1.fc25.x86_64
clang-3.9.1-2.fc25.x86_64
findutils-4.6.0-8.fc25.x86_64
flex-2.6.0-3.fc25.x86_64
gcc-6.3.1-1.fc25.x86_64
gcc-c++-6.3.1-1.fc25.x86_64
git-2.9.3-2.fc25.x86_64
glib2-devel-2.50.3-1.fc25.x86_64
libfdt-devel-1.4.2-1.fc25.x86_64
make-4.1-5.fc24.x86_64
mingw32-SDL-1.2.15-7.fc24.noarch
mingw32-bzip2-1.0.6-7.fc24.noarch
mingw32-curl-7.47.0-1.fc24.noarch
mingw32-glib2-2.50.1-1.fc25.noarch
mingw32-gmp-6.1.1-1.fc25.noarch
mingw32-gnutls-3.5.5-2.fc25.noarch
mingw32-gtk2-2.24.31-2.fc25.noarch
mingw32-gtk3-3.22.2-1.fc25.noarch
mingw32-libjpeg-turbo-1.5.1-1.fc25.noarch
mingw32-libpng-1.6.27-1.fc25.noarch
mingw32-libssh2-1.4.3-5.fc24.noarch
mingw32-libtasn1-4.9-1.fc25.noarch
mingw32-nettle-3.3-1.fc25.noarch
mingw32-pixman-0.34.0-1.fc25.noarch
mingw32-pkg-config-0.28-6.fc24.x86_64
mingw64-SDL-1.2.15-7.fc24.noarch
mingw64-bzip2-1.0.6-7.fc24.noarch
mingw64-curl-7.47.0-1.fc24.noarch
mingw64-glib2-2.50.1-1.fc25.noarch
mingw64-gmp-6.1.1-1.fc25.noarch
mingw64-gnutls-3.5.5-2.fc25.noarch
mingw64-gtk2-2.24.31-2.fc25.noarch
mingw64-gtk3-3.22.2-1.fc25.noarch
mingw64-libjpeg-turbo-1.5.1-1.fc25.noarch
mingw64-libpng-1.6.27-1.fc25.noarch
mingw64-libssh2-1.4.3-5.fc24.noarch
mingw64-libtasn1-4.9-1.fc25.noarch
mingw64-nettle-3.3-1.fc25.noarch
mingw64-pixman-0.34.0-1.fc25.noarch
mingw64-pkg-config-0.28-6.fc24.x86_64
package python2 is not installed
perl-5.24.1-385.fc25.x86_64
pixman-devel-0.34.0-2.fc24.x86_64
sparse-0.5.0-10.fc25.x86_64
tar-1.29-3.fc25.x86_64
which-2.21-1.fc25.x86_64
zlib-devel-1.2.8-10.fc24.x86_64
Environment variables:
FBR=f25
PACKAGES=ccache git tar PyYAML sparse flex bison python2 glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel gcc gcc-c++ clang make perl which bc findutils mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1 mingw32-libjpeg-turbo mingw32-libpng mingw32-curl mingw32-libssh2 mingw32-bzip2 mingw64-pixman mingw64-glib2 mingw64-gmp mingw64-SDL mingw64-pkg-config mingw64-gtk2 mingw64-gtk3 mingw64-gnutls mingw64-nettle mingw64-libtasn1 mingw64-libjpeg-turbo mingw64-libpng mingw64-curl mingw64-libssh2 mingw64-bzip2
HOSTNAME=
TERM=xterm
MAKEFLAGS= -j8
HISTSIZE=1000
J=8
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
TARGET_LIST=
HISTCONTROL=ignoredups
FGC=f25
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
DISTTAG=f25docker
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES=mingw clang pyyaml dtc
DEBUG=
_=/usr/bin/env
Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/var/tmp/qemu-build/install --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-debug --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent --with-sdlabi=1.2 --with-gtkabi=2.0
Install prefix /var/tmp/qemu-build/install
BIOS directory /var/tmp/qemu-build/install
binary directory /var/tmp/qemu-build/install
library directory /var/tmp/qemu-build/install/lib
module directory /var/tmp/qemu-build/install/lib
libexec directory /var/tmp/qemu-build/install/libexec
include directory /var/tmp/qemu-build/install/include
config directory /var/tmp/qemu-build/install
local state directory queried at runtime
Windows SDK no
Source path /tmp/qemu-test/src
C compiler x86_64-w64-mingw32-gcc
Host C compiler cc
C++ compiler x86_64-w64-mingw32-g++
Objective-C compiler clang
ARFLAGS rv
CFLAGS -g
QEMU_CFLAGS -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/pixman-1 -I$(SRC_PATH)/dtc/libfdt -Werror -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -m64 -mcx16 -mthreads -D__USE_MINGW_ANSI_STDIO=1 -DWIN32_LEAN_AND_MEAN -DWINVER=0x501 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16
LDFLAGS -Wl,--nxcompat -Wl,--no-seh -Wl,--dynamicbase -Wl,--warn-common -m64 -g
make make
install install
python python -B
smbd /usr/sbin/smbd
module support no
host CPU x86_64
host big endian no
target list x86_64-softmmu aarch64-softmmu
tcg debug enabled yes
gprof enabled no
sparse enabled no
strip binaries no
profiler no
static build no
pixman system
SDL support yes (1.2.15)
GTK support yes (2.24.31)
GTK GL support no
VTE support no
TLS priority NORMAL
GNUTLS support yes
GNUTLS rnd yes
libgcrypt no
libgcrypt kdf no
nettle yes (3.3)
nettle kdf yes
libtasn1 yes
curses support no
virgl support no
curl support yes
mingw32 support yes
Audio drivers dsound
Block whitelist (rw)
Block whitelist (ro)
VirtFS support no
VNC support yes
VNC SASL support no
VNC JPEG support yes
VNC PNG support yes
xen support no
brlapi support no
bluez support no
Documentation no
PIE no
vde support no
netmap support no
Linux AIO support no
ATTR/XATTR support no
Install blobs yes
KVM support no
HAX support yes
RDMA support no
TCG interpreter no
fdt support yes
preadv support no
fdatasync no
madvise no
posix_madvise no
libcap-ng support no
vhost-net support no
vhost-scsi support no
vhost-vsock support no
Trace backends simple
Trace output file trace-<pid>
spice support no
rbd support no
xfsctl support no
smartcard support no
libusb no
usb net redir no
OpenGL support no
OpenGL dmabufs no
libiscsi support no
libnfs support no
build guest agent yes
QGA VSS support no
QGA w32 disk info yes
QGA MSI support no
seccomp support no
coroutine backend win32
coroutine pool yes
debug stack usage no
GlusterFS support no
gcov gcov
gcov enabled no
TPM support yes
libssh2 support yes
TPM passthrough no
QOM debugging yes
lzo support no
snappy support no
bzip2 support yes
NUMA host support no
tcmalloc support no
jemalloc support no
avx2 optimization yes
replication support yes
mkdir -p dtc/libfdt
mkdir -p dtc/tests
GEN config-host.h
GEN x86_64-softmmu/config-devices.mak.tmp
GEN qapi-types.h
GEN aarch64-softmmu/config-devices.mak.tmp
GEN qemu-options.def
GEN qmp-commands.h
GEN qapi-visit.h
GEN qapi-event.h
GEN x86_64-softmmu/config-devices.mak
GEN aarch64-softmmu/config-devices.mak
GEN qmp-marshal.c
GEN qapi-types.c
GEN qapi-visit.c
GEN qapi-event.c
GEN qmp-introspect.h
GEN qmp-introspect.c
GEN trace/generated-tcg-tracers.h
GEN trace/generated-helpers-wrappers.h
GEN trace/generated-helpers.h
GEN trace/generated-helpers.c
GEN module_block.h
GEN tests/test-qapi-types.h
GEN tests/test-qapi-visit.h
GEN tests/test-qmp-commands.h
GEN tests/test-qapi-event.h
GEN tests/test-qmp-introspect.h
GEN trace-root.h
GEN util/trace.h
GEN crypto/trace.h
GEN io/trace.h
GEN migration/trace.h
GEN block/trace.h
GEN backends/trace.h
GEN hw/block/trace.h
GEN hw/block/dataplane/trace.h
GEN hw/char/trace.h
GEN hw/intc/trace.h
GEN hw/net/trace.h
GEN hw/virtio/trace.h
GEN hw/audio/trace.h
GEN hw/misc/trace.h
GEN hw/usb/trace.h
GEN hw/scsi/trace.h
GEN hw/nvram/trace.h
GEN hw/display/trace.h
GEN hw/input/trace.h
GEN hw/timer/trace.h
GEN hw/dma/trace.h
GEN hw/sparc/trace.h
GEN hw/sd/trace.h
GEN hw/isa/trace.h
GEN hw/mem/trace.h
GEN hw/i386/trace.h
GEN hw/i386/xen/trace.h
GEN hw/9pfs/trace.h
GEN hw/ppc/trace.h
GEN hw/pci/trace.h
GEN hw/s390x/trace.h
GEN hw/vfio/trace.h
GEN hw/acpi/trace.h
GEN hw/arm/trace.h
GEN hw/alpha/trace.h
GEN hw/xen/trace.h
GEN ui/trace.h
GEN audio/trace.h
GEN net/trace.h
GEN target/arm/trace.h
GEN target/i386/trace.h
GEN target/mips/trace.h
GEN target/sparc/trace.h
GEN target/s390x/trace.h
GEN target/ppc/trace.h
GEN qom/trace.h
GEN linux-user/trace.h
GEN qapi/trace.h
GEN trace-root.c
GEN util/trace.c
GEN crypto/trace.c
GEN io/trace.c
GEN migration/trace.c
GEN block/trace.c
GEN backends/trace.c
GEN hw/block/dataplane/trace.c
GEN hw/block/trace.c
GEN hw/char/trace.c
GEN hw/intc/trace.c
GEN hw/net/trace.c
GEN hw/virtio/trace.c
GEN hw/audio/trace.c
GEN hw/misc/trace.c
GEN hw/usb/trace.c
GEN hw/scsi/trace.c
GEN hw/nvram/trace.c
GEN hw/display/trace.c
GEN hw/input/trace.c
GEN hw/timer/trace.c
GEN hw/dma/trace.c
GEN hw/sd/trace.c
GEN hw/sparc/trace.c
GEN hw/isa/trace.c
GEN hw/mem/trace.c
GEN hw/i386/trace.c
GEN hw/i386/xen/trace.c
GEN hw/9pfs/trace.c
GEN hw/ppc/trace.c
GEN hw/pci/trace.c
GEN hw/s390x/trace.c
GEN hw/vfio/trace.c
GEN hw/acpi/trace.c
GEN hw/arm/trace.c
GEN hw/alpha/trace.c
GEN hw/xen/trace.c
GEN ui/trace.c
GEN audio/trace.c
GEN net/trace.c
GEN target/arm/trace.c
GEN target/i386/trace.c
GEN target/mips/trace.c
GEN target/sparc/trace.c
GEN target/s390x/trace.c
GEN target/ppc/trace.c
GEN qom/trace.c
GEN linux-user/trace.c
GEN qapi/trace.c
GEN config-all-devices.mak
DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
DEP /tmp/qemu-test/src/dtc/tests/trees.S
DEP /tmp/qemu-test/src/dtc/tests/testutils.c
DEP /tmp/qemu-test/src/dtc/tests/value-labels.c
DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c
DEP /tmp/qemu-test/src/dtc/tests/check_path.c
DEP /tmp/qemu-test/src/dtc/tests/overlay_bad_fixup.c
DEP /tmp/qemu-test/src/dtc/tests/overlay.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/property_iterate.c
DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c
DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c
DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c
DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c
DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c
DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c
DEP /tmp/qemu-test/src/dtc/tests/incbin.c
DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c
DEP /tmp/qemu-test/src/dtc/tests/path-references.c
DEP /tmp/qemu-test/src/dtc/tests/references.c
DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c
DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c
DEP /tmp/qemu-test/src/dtc/tests/del_node.c
DEP /tmp/qemu-test/src/dtc/tests/del_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop.c
DEP /tmp/qemu-test/src/dtc/tests/set_name.c
DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c
DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c
DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
DEP /tmp/qemu-test/src/dtc/tests/nop_property.c
DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c
DEP /tmp/qemu-test/src/dtc/tests/stringlist.c
DEP /tmp/qemu-test/src/dtc/tests/addr_size_cells.c
DEP /tmp/qemu-test/src/dtc/tests/notfound.c
DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c
DEP /tmp/qemu-test/src/dtc/tests/char_literal.c
DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c
DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c
DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c
DEP /tmp/qemu-test/src/dtc/tests/get_path.c
DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c
DEP /tmp/qemu-test/src/dtc/tests/getprop.c
DEP /tmp/qemu-test/src/dtc/tests/get_name.c
DEP /tmp/qemu-test/src/dtc/tests/path_offset.c
DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c
DEP /tmp/qemu-test/src/dtc/tests/find_property.c
DEP /tmp/qemu-test/src/dtc/tests/root_node.c
DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_overlay.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_addresses.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
DEP /tmp/qemu-test/src/dtc/util.c
DEP /tmp/qemu-test/src/dtc/fdtput.c
DEP /tmp/qemu-test/src/dtc/fdtget.c
DEP /tmp/qemu-test/src/dtc/fdtdump.c
LEX convert-dtsv0-lexer.lex.c
DEP /tmp/qemu-test/src/dtc/srcpos.c
BISON dtc-parser.tab.c
LEX dtc-lexer.lex.c
DEP /tmp/qemu-test/src/dtc/treesource.c
DEP /tmp/qemu-test/src/dtc/livetree.c
DEP /tmp/qemu-test/src/dtc/fstree.c
DEP /tmp/qemu-test/src/dtc/flattree.c
DEP /tmp/qemu-test/src/dtc/dtc.c
DEP /tmp/qemu-test/src/dtc/data.c
DEP /tmp/qemu-test/src/dtc/checks.c
DEP convert-dtsv0-lexer.lex.c
DEP dtc-parser.tab.c
DEP dtc-lexer.lex.c
CHK version_gen.h
UPD version_gen.h
DEP /tmp/qemu-test/src/dtc/util.c
CC libfdt/fdt.o
CC libfdt/fdt_ro.o
CC libfdt/fdt_sw.o
CC libfdt/fdt_rw.o
CC libfdt/fdt_wip.o
CC libfdt/fdt_strerror.o
CC libfdt/fdt_empty_tree.o
CC libfdt/fdt_addresses.o
CC libfdt/fdt_overlay.o
AR libfdt/libfdt.a
x86_64-w64-mingw32-ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
RC version.o
GEN qga/qapi-generated/qga-qapi-types.h
GEN qga/qapi-generated/qga-qapi-visit.h
GEN qga/qapi-generated/qga-qmp-commands.h
GEN qga/qapi-generated/qga-qapi-types.c
GEN qga/qapi-generated/qga-qmp-marshal.c
GEN qga/qapi-generated/qga-qapi-visit.c
CC util/trace.o
CC trace-root.o
CC crypto/trace.o
CC io/trace.o
CC migration/trace.o
CC block/trace.o
CC backends/trace.o
CC hw/block/trace.o
CC hw/block/dataplane/trace.o
CC hw/char/trace.o
CC hw/intc/trace.o
CC hw/net/trace.o
CC hw/virtio/trace.o
CC hw/audio/trace.o
CC hw/misc/trace.o
CC hw/usb/trace.o
CC hw/scsi/trace.o
CC hw/nvram/trace.o
CC hw/display/trace.o
CC hw/input/trace.o
CC hw/timer/trace.o
CC hw/dma/trace.o
CC hw/sparc/trace.o
CC hw/sd/trace.o
CC hw/isa/trace.o
CC hw/mem/trace.o
CC hw/i386/trace.o
CC hw/i386/xen/trace.o
CC hw/9pfs/trace.o
CC hw/ppc/trace.o
CC hw/pci/trace.o
CC hw/s390x/trace.o
CC hw/vfio/trace.o
CC hw/acpi/trace.o
CC hw/arm/trace.o
CC hw/alpha/trace.o
CC hw/xen/trace.o
CC ui/trace.o
CC audio/trace.o
CC net/trace.o
CC target/arm/trace.o
CC target/i386/trace.o
CC target/mips/trace.o
CC target/sparc/trace.o
CC target/s390x/trace.o
CC target/ppc/trace.o
CC qom/trace.o
CC linux-user/trace.o
CC qapi/trace.o
CC qmp-introspect.o
CC qapi-types.o
CC qapi-visit.o
CC qapi-event.o
CC qapi/qapi-visit-core.o
CC qapi/qapi-dealloc-visitor.o
CC qapi/qobject-input-visitor.o
CC qapi/qobject-output-visitor.o
CC qapi/qmp-registry.o
CC qapi/qmp-dispatch.o
CC qapi/string-input-visitor.o
CC qapi/string-output-visitor.o
CC qapi/opts-visitor.o
CC qapi/qapi-clone-visitor.o
CC qapi/qmp-event.o
CC qapi/qapi-util.o
CC qobject/qnull.o
CC qobject/qint.o
CC qobject/qstring.o
CC qobject/qdict.o
CC qobject/qlist.o
CC qobject/qfloat.o
CC qobject/qbool.o
CC qobject/qjson.o
CC qobject/qobject.o
CC qobject/json-lexer.o
CC qobject/json-streamer.o
CC qobject/json-parser.o
CC trace/control.o
CC trace/simple.o
CC trace/qmp.o
CC util/osdep.o
CC util/cutils.o
CC util/unicode.o
CC util/qemu-timer-common.o
CC util/bufferiszero.o
CC util/lockcnt.o
CC util/aiocb.o
CC util/async.o
CC util/thread-pool.o
CC util/qemu-timer.o
CC util/main-loop.o
CC util/iohandler.o
CC util/aio-win32.o
CC util/event_notifier-win32.o
CC util/oslib-win32.o
CC util/qemu-thread-win32.o
CC util/envlist.o
CC util/path.o
CC util/module.o
CC util/host-utils.o
CC util/bitmap.o
CC util/bitops.o
CC util/hbitmap.o
CC util/fifo8.o
CC util/acl.o
CC util/error.o
CC util/qemu-error.o
CC util/id.o
CC util/iov.o
CC util/qemu-config.o
CC util/qemu-sockets.o
CC util/uri.o
CC util/notify.o
CC util/qemu-option.o
CC util/qemu-progress.o
CC util/keyval.o
CC util/hexdump.o
CC util/crc32c.o
CC util/uuid.o
CC util/throttle.o
CC util/getauxval.o
CC util/readline.o
CC util/rcu.o
CC util/qemu-coroutine.o
CC util/qemu-coroutine-lock.o
CC util/qemu-coroutine-io.o
CC util/qemu-coroutine-sleep.o
CC util/coroutine-win32.o
CC util/buffer.o
CC util/timed-average.o
CC util/base64.o
CC util/log.o
CC util/qdist.o
CC util/qht.o
CC util/range.o
CC util/stats64.o
CC util/systemd.o
CC crypto/pbkdf-stub.o
CC stubs/arch-query-cpu-def.o
CC stubs/arch-query-cpu-model-expansion.o
CC stubs/arch-query-cpu-model-comparison.o
CC stubs/arch-query-cpu-model-baseline.o
CC stubs/bdrv-next-monitor-owned.o
CC stubs/blk-commit-all.o
CC stubs/blockdev-close-all-bdrv-states.o
CC stubs/clock-warp.o
CC stubs/cpu-get-clock.o
CC stubs/cpu-get-icount.o
CC stubs/dump.o
CC stubs/error-printf.o
CC stubs/fdset.o
CC stubs/gdbstub.o
CC stubs/get-vm-name.o
CC stubs/iothread.o
CC stubs/iothread-lock.o
CC stubs/is-daemonized.o
In file included from /tmp/qemu-test/src/util/stats64.c:13:0:
/tmp/qemu-test/src/util/stats64.c: In function 'stat64_add32_carry':
/tmp/qemu-test/src/util/stats64.c:71:37: error: 'value' undeclared (first use in this function)
old = atomic_fetch_add(&s->low, value);
^
/tmp/qemu-test/src/include/qemu/atomic.h:185:58: note: in definition of macro 'atomic_fetch_add'
#define atomic_fetch_add(ptr, n) __atomic_fetch_add(ptr, n, __ATOMIC_SEQ_CST)
^
/tmp/qemu-test/src/util/stats64.c:71:37: note: each undeclared identifier is reported only once for each function it appears in
old = atomic_fetch_add(&s->low, value);
^
/tmp/qemu-test/src/include/qemu/atomic.h:185:58: note: in definition of macro 'atomic_fetch_add'
#define atomic_fetch_add(ptr, n) __atomic_fetch_add(ptr, n, __ATOMIC_SEQ_CST)
^
/tmp/qemu-test/src/rules.mak:69: recipe for target 'util/stats64.o' failed
make: *** [util/stats64.o] Error 1
make: *** Waiting for unfinished jobs....
tests/docker/Makefile.include:118: recipe for target 'docker-run' failed
make[1]: *** [docker-run] Error 2
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-f1fvdzwe/src'
tests/docker/Makefile.include:149: recipe for target 'docker-run-test-mingw@fedora' failed
make: *** [docker-run-test-mingw@fedora] Error 2
=== OUTPUT END ===
Test command exited with code: 2
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (17 preceding siblings ...)
2017-04-20 12:40 ` [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 no-reply
@ 2017-04-20 12:42 ` no-reply
2017-05-02 15:42 ` Paolo Bonzini
` (2 subsequent siblings)
21 siblings, 0 replies; 65+ messages in thread
From: no-reply @ 2017-04-20 12:42 UTC (permalink / raw)
To: pbonzini; +Cc: famz, qemu-devel, qemu-block
Hi,
This series seems to have some coding style problems. See output below for
more information:
Type: series
Message-id: 20170420120058.28404-1-pbonzini@redhat.com
Subject: [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1
=== TEST SCRIPT BEGIN ===
#!/bin/bash
BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0
# Useful git options
git config --local diff.renamelimit 0
git config --local diff.renames True
commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done
exit $failed
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
* [new tag] patchew/20170420121639.32685-1-berrange@redhat.com -> patchew/20170420121639.32685-1-berrange@redhat.com
Switched to a new branch 'test'
c716264 block: make accounting thread-safe
7d0704f block: protect modification of dirty bitmaps with a mutex
2259950 block: introduce dirty_bitmap_mutex
0532009 block: optimize access to reqs_lock
862fd98 coroutine-lock: introduce qemu_co_mutex_lock_unlock
5d9d699 block: protect tracked_requests and flush_queue with reqs_lock
a65d4b0 block: access write_gen with atomics
01ac75b block: use Stat64 for wr_highest_offset
628d1ba util: add stats64 module
ea6bb59 throttle-groups: protect throttled requests with a CoMutex
895a11d throttle-groups: do not use qemu_co_enter_next
42ce4b3 block: access io_plugged with atomic ops
bc2deb1 block: access wakeup with atomic ops
18d2d96 block: access serialising_in_flight with atomic ops
a11151d block: access io_limits_disabled with atomic ops
c4d7323 block: access quiesce_counter with atomic ops
9c192bd block: access copy_on_read with atomic ops
=== OUTPUT BEGIN ===
Checking PATCH 1/17: block: access copy_on_read with atomic ops...
Checking PATCH 2/17: block: access quiesce_counter with atomic ops...
Checking PATCH 3/17: block: access io_limits_disabled with atomic ops...
Checking PATCH 4/17: block: access serialising_in_flight with atomic ops...
Checking PATCH 5/17: block: access wakeup with atomic ops...
Checking PATCH 6/17: block: access io_plugged with atomic ops...
Checking PATCH 7/17: throttle-groups: do not use qemu_co_enter_next...
Checking PATCH 8/17: throttle-groups: protect throttled requests with a CoMutex...
Checking PATCH 9/17: util: add stats64 module...
WARNING: architecture specific defines should be avoided
#35: FILE: include/qemu/stats64.h:18:
+#if __SIZEOF_LONG__ < 8
ERROR: memory barrier without comment
#343: FILE: util/stats64.c:100:
+ smp_wmb();
ERROR: memory barrier without comment
#372: FILE: util/stats64.c:129:
+ smp_wmb();
total: 2 errors, 1 warnings, 350 lines checked
Your patch has style problems, please review. If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
Checking PATCH 10/17: block: use Stat64 for wr_highest_offset...
Checking PATCH 11/17: block: access write_gen with atomics...
Checking PATCH 12/17: block: protect tracked_requests and flush_queue with reqs_lock...
Checking PATCH 13/17: coroutine-lock: introduce qemu_co_mutex_lock_unlock...
Checking PATCH 14/17: block: optimize access to reqs_lock...
Checking PATCH 15/17: block: introduce dirty_bitmap_mutex...
Checking PATCH 16/17: block: protect modification of dirty bitmaps with a mutex...
Checking PATCH 17/17: block: make accounting thread-safe...
=== OUTPUT END ===
Test command exited with code: 1
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (18 preceding siblings ...)
2017-04-20 12:42 ` no-reply
@ 2017-05-02 15:42 ` Paolo Bonzini
2017-05-04 8:09 ` Fam Zheng
2017-05-05 13:01 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
21 siblings, 0 replies; 65+ messages in thread
From: Paolo Bonzini @ 2017-05-02 15:42 UTC (permalink / raw)
To: qemu-devel; +Cc: qemu-block
On 20/04/2017 14:00, Paolo Bonzini wrote:
> This series uses mutexes or atomic operations around core block layer
> operations. The remaining parts include:
>
> - drivers, though most of them are already thread safe (part 2, 8 patches,
> depends on Kevin's conversion of QED to coroutines)
>
> - block jobs, before-write notifiers, the write threshold mechanism,
> snapshots, replication, key management (part 3, 16 patches)
>
> - devices (virtio-blk/virtio-scsi, part 4, 5 patches)
>
> Once these four parts are done the AioContext lock can be removed
> together with all temporary workarounds that have accumulated.
Ping?
Paolo
> Paolo
>
> Paolo Bonzini (17):
> block: access copy_on_read with atomic ops
> block: access quiesce_counter with atomic ops
> block: access io_limits_disabled with atomic ops
> block: access serialising_in_flight with atomic ops
> block: access wakeup with atomic ops
> block: access io_plugged with atomic ops
> throttle-groups: do not use qemu_co_enter_next
> throttle-groups: protect throttled requests with a CoMutex
> util: add stats64 module
> block: use Stat64 for wr_highest_offset
> block: access write_gen with atomics
> block: protect tracked_requests and flush_queue with reqs_lock
> coroutine-lock: introduce qemu_co_mutex_lock_unlock
> block: optimize access to reqs_lock
> block: introduce dirty_bitmap_mutex
> block: protect modification of dirty bitmaps with a mutex
> block: make accounting thread-safe
>
> block.c | 10 +-
> block/accounting.c | 15 +++
> block/block-backend.c | 5 +-
> block/dirty-bitmap.c | 125 ++++++++++++++++++++++--
> block/io.c | 70 +++++++++-----
> block/mirror.c | 14 ++-
> block/nfs.c | 4 +-
> block/qapi.c | 2 +-
> block/sheepdog.c | 3 +-
> block/throttle-groups.c | 76 +++++++++++++--
> block/write-threshold.c | 37 ++++----
> blockdev.c | 46 ++-------
> include/block/accounting.h | 8 +-
> include/block/block.h | 5 +-
> include/block/block_int.h | 63 ++++++++-----
> include/block/dirty-bitmap.h | 23 ++++-
> include/qemu/coroutine.h | 6 ++
> include/qemu/stats64.h | 210 +++++++++++++++++++++++++++++++++++++++++
> include/sysemu/block-backend.h | 10 +-
> migration/block.c | 6 --
> util/Makefile.objs | 1 +
> util/qemu-coroutine-lock.c | 36 +++++++
> util/stats64.c | 135 ++++++++++++++++++++++++++
> 23 files changed, 763 insertions(+), 147 deletions(-)
> create mode 100644 include/qemu/stats64.h
> create mode 100644 util/stats64.c
>
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (19 preceding siblings ...)
2017-05-02 15:42 ` Paolo Bonzini
@ 2017-05-04 8:09 ` Fam Zheng
2017-05-05 13:01 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
21 siblings, 0 replies; 65+ messages in thread
From: Fam Zheng @ 2017-05-04 8:09 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
On Thu, 04/20 14:00, Paolo Bonzini wrote:
> This series uses mutexes or atomic operations around core block layer
> operations. The remaining parts include:
>
> - drivers, though most of them are already thread safe (part 2, 8 patches,
> depends on Kevin's conversion of QED to coroutines)
>
> - block jobs, before-write notifiers, the write threshold mechanism,
> snapshots, replication, key management (part 3, 16 patches)
>
> - devices (virtio-blk/virtio-scsi, part 4, 5 patches)
>
> Once these four parts are done the AioContext lock can be removed
> together with all temporary workarounds that have accumulated.
Thanks! I had a few nits and questions, but it's pretty good in general:
Reviewed-by: Fam Zheng <famz@redhat.com>
Fam
^ permalink raw reply [flat|nested] 65+ messages in thread* Re: [Qemu-devel] [Qemu-block] [PATCH for 2.10 00/17] Block layer thread safety, part 1
2017-04-20 12:00 [Qemu-devel] [PATCH for 2.10 00/17] Block layer thread safety, part 1 Paolo Bonzini
` (20 preceding siblings ...)
2017-05-04 8:09 ` Fam Zheng
@ 2017-05-05 13:01 ` Stefan Hajnoczi
21 siblings, 0 replies; 65+ messages in thread
From: Stefan Hajnoczi @ 2017-05-05 13:01 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, qemu-block
[-- Attachment #1: Type: text/plain, Size: 946 bytes --]
On Thu, Apr 20, 2017 at 02:00:41PM +0200, Paolo Bonzini wrote:
> This series uses mutexes or atomic operations around core block layer
> operations. The remaining parts include:
>
> - drivers, though most of them are already thread safe (part 2, 8 patches,
> depends on Kevin's conversion of QED to coroutines)
>
> - block jobs, before-write notifiers, the write threshold mechanism,
> snapshots, replication, key management (part 3, 16 patches)
>
> - devices (virtio-blk/virtio-scsi, part 4, 5 patches)
>
> Once these four parts are done the AioContext lock can be removed
> together with all temporary workarounds that have accumulated.
I like the direction these changes are heading in.
Avoiding low-level tricks and custom primitives would be nice until
there is performance data justifying them because they are
time-consuming to review, increase complexity, and are likely to be
misused by others.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 65+ messages in thread