* [PULL 0/2] Block patches
@ 2021-03-24 14:52 Stefan Hajnoczi
2021-03-24 14:52 ` [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy Stefan Hajnoczi
` (3 more replies)
0 siblings, 4 replies; 12+ messages in thread
From: Stefan Hajnoczi @ 2021-03-24 14:52 UTC (permalink / raw)
To: Peter Maydell, qemu-devel
Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy,
Eduardo Habkost, qemu-block, Juan Quintela, John Snow,
Michael S. Tsirkin, Dr. David Alan Gilbert, Stefan Hajnoczi,
Max Reitz
The following changes since commit 67c1115edd98f388ca89dd38322ea3fadf034523:
Merge remote-tracking branch 'remotes/kraxel/tags/ui-20210323-pull-request' into staging (2021-03-23 23:47:30 +0000)
are available in the Git repository at:
https://gitlab.com/stefanha/qemu.git tags/block-pull-request
for you to fetch changes up to 3460fd7f3959d1fa7bcc255796844aa261c805a4:
migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps (2021-03-24 13:41:19 +0000)
----------------------------------------------------------------
Pull request
This dirty bitmap fix solves a crash that can be triggered in the destination
QEMU process during live migration.
----------------------------------------------------------------
Vladimir Sementsov-Ogievskiy (2):
migration/block-dirty-bitmap: make incoming disabled bitmaps busy
migrate-bitmaps-postcopy-test: check that we can't remove in-flight
bitmaps
migration/block-dirty-bitmap.c | 6 ++++++
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 10 ++++++++++
2 files changed, 16 insertions(+)
--
2.30.2
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy
2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi
@ 2021-03-24 14:52 ` Stefan Hajnoczi
2021-03-24 14:52 ` [PULL 2/2] migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps Stefan Hajnoczi
` (2 subsequent siblings)
3 siblings, 0 replies; 12+ messages in thread
From: Stefan Hajnoczi @ 2021-03-24 14:52 UTC (permalink / raw)
To: Peter Maydell, qemu-devel
Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy,
Eduardo Habkost, qemu-block, Juan Quintela, John Snow,
Michael S. Tsirkin, Dr. David Alan Gilbert, Stefan Hajnoczi,
Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Incoming enabled bitmaps are busy, because we do
bdrv_dirty_bitmap_create_successor() for them. But disabled bitmaps
being migrated are not marked busy, and user can remove them during the
incoming migration. Then we may crash in cancel_incoming_locked() when
try to remove the bitmap that was already removed by user, like this:
#0 qemu_mutex_lock_impl (mutex=0x5593d88c50d1, file=0x559680554b20
"../block/dirty-bitmap.c", line=64) at ../util/qemu-thread-posix.c:77
#1 bdrv_dirty_bitmaps_lock (bs=0x5593d88c0ee9)
at ../block/dirty-bitmap.c:64
#2 bdrv_release_dirty_bitmap (bitmap=0x5596810e9570)
at ../block/dirty-bitmap.c:362
#3 cancel_incoming_locked (s=0x559680be8208 <dbm_state+40>)
at ../migration/block-dirty-bitmap.c:918
#4 dirty_bitmap_load (f=0x559681d02b10, opaque=0x559680be81e0
<dbm_state>, version_id=1) at ../migration/block-dirty-bitmap.c:1194
#5 vmstate_load (f=0x559681d02b10, se=0x559680fb5810)
at ../migration/savevm.c:908
#6 qemu_loadvm_section_part_end (f=0x559681d02b10,
mis=0x559680fb4a30) at ../migration/savevm.c:2473
#7 qemu_loadvm_state_main (f=0x559681d02b10, mis=0x559680fb4a30)
at ../migration/savevm.c:2626
#8 postcopy_ram_listen_thread (opaque=0x0)
at ../migration/savevm.c:1871
#9 qemu_thread_start (args=0x5596817ccd10)
at ../util/qemu-thread-posix.c:521
#10 start_thread () at /lib64/libpthread.so.0
#11 clone () at /lib64/libc.so.6
Note bs pointer taken from bitmap: it's definitely bad aligned. That's
because we are in use after free, bitmap is already freed.
So, let's make disabled bitmaps (being migrated) busy during incoming
migration.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20210322094906.5079-2-vsementsov@virtuozzo.com>
---
migration/block-dirty-bitmap.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
index 975093610a..35f5ef688d 100644
--- a/migration/block-dirty-bitmap.c
+++ b/migration/block-dirty-bitmap.c
@@ -839,6 +839,8 @@ static int dirty_bitmap_load_start(QEMUFile *f, DBMLoadState *s)
error_report_err(local_err);
return -EINVAL;
}
+ } else {
+ bdrv_dirty_bitmap_set_busy(s->bitmap, true);
}
b = g_new(LoadBitmapState, 1);
@@ -914,6 +916,8 @@ static void cancel_incoming_locked(DBMLoadState *s)
assert(!s->before_vm_start_handled || !b->migrated);
if (bdrv_dirty_bitmap_has_successor(b->bitmap)) {
bdrv_reclaim_dirty_bitmap(b->bitmap, &error_abort);
+ } else {
+ bdrv_dirty_bitmap_set_busy(b->bitmap, false);
}
bdrv_release_dirty_bitmap(b->bitmap);
}
@@ -951,6 +955,8 @@ static void dirty_bitmap_load_complete(QEMUFile *f, DBMLoadState *s)
if (bdrv_dirty_bitmap_has_successor(s->bitmap)) {
bdrv_reclaim_dirty_bitmap(s->bitmap, &error_abort);
+ } else {
+ bdrv_dirty_bitmap_set_busy(s->bitmap, false);
}
for (item = s->bitmaps; item; item = g_slist_next(item)) {
--
2.30.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PULL 2/2] migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps
2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi
2021-03-24 14:52 ` [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy Stefan Hajnoczi
@ 2021-03-24 14:52 ` Stefan Hajnoczi
2021-03-24 18:05 ` [PULL 0/2] Block patches Peter Maydell
2021-03-26 10:22 ` Peter Maydell
3 siblings, 0 replies; 12+ messages in thread
From: Stefan Hajnoczi @ 2021-03-24 14:52 UTC (permalink / raw)
To: Peter Maydell, qemu-devel
Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy,
Eduardo Habkost, qemu-block, Juan Quintela, John Snow,
Michael S. Tsirkin, Dr. David Alan Gilbert, Stefan Hajnoczi,
Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Check that we can't remove bitmaps being migrated on destination vm.
The new check proves that previous commit helps.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20210322094906.5079-3-vsementsov@virtuozzo.com>
---
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
index d046ebeb94..584062b412 100755
--- a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
+++ b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
@@ -224,6 +224,16 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
self.start_postcopy()
self.vm_b_events += self.vm_b.get_qmp_events()
+
+ # While being here, let's check that we can't remove in-flight bitmaps.
+ for vm in (self.vm_a, self.vm_b):
+ for i in range(0, nb_bitmaps):
+ result = vm.qmp('block-dirty-bitmap-remove', node='drive0',
+ name=f'bitmap{i}')
+ self.assert_qmp(result, 'error/desc',
+ f"Bitmap 'bitmap{i}' is currently in use by "
+ "another operation and cannot be used")
+
self.vm_b.shutdown()
# recreate vm_b, so there is no incoming option, which prevents
# loading bitmaps from disk
--
2.30.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches
2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi
2021-03-24 14:52 ` [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy Stefan Hajnoczi
2021-03-24 14:52 ` [PULL 2/2] migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps Stefan Hajnoczi
@ 2021-03-24 18:05 ` Peter Maydell
2021-03-24 20:18 ` Vladimir Sementsov-Ogievskiy
2021-03-26 10:22 ` Peter Maydell
3 siblings, 1 reply; 12+ messages in thread
From: Peter Maydell @ 2021-03-24 18:05 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy,
Eduardo Habkost, Qemu-block, Juan Quintela, John Snow,
Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert,
Max Reitz
On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> The following changes since commit 67c1115edd98f388ca89dd38322ea3fadf034523:
>
> Merge remote-tracking branch 'remotes/kraxel/tags/ui-20210323-pull-request' into staging (2021-03-23 23:47:30 +0000)
>
> are available in the Git repository at:
>
> https://gitlab.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 3460fd7f3959d1fa7bcc255796844aa261c805a4:
>
> migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps (2021-03-24 13:41:19 +0000)
>
> ----------------------------------------------------------------
> Pull request
>
> This dirty bitmap fix solves a crash that can be triggered in the destination
> QEMU process during live migration.
>
> ----------------------------------------------------------------
>
> Vladimir Sementsov-Ogievskiy (2):
> migration/block-dirty-bitmap: make incoming disabled bitmaps busy
> migrate-bitmaps-postcopy-test: check that we can't remove in-flight
> bitmaps
This failed the 'qsd-jobs' iotest on s390x:
TEST iotest-qcow2: 309
TEST iotest-qcow2: 313
TEST iotest-qcow2: qsd-jobs [fail]
QEMU --
"/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-system-s390x"
-nodefaults -display none -accel qtest
QEMU_IMG -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-img"
QEMU_IO --
"/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-io" --cache
writeback --aio threads -f qcow2
QEMU_NBD -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
IMGFMT -- qcow2
IMGPROTO -- file
PLATFORM -- Linux/s390x qemu01 4.15.0-132-generic
TEST_DIR -- /home/ubuntu/qemu/build/all/tests/qemu-iotests/scratch
SOCK_DIR -- /tmp/tmp807j_qyh
SOCKET_SCM_HELPER --
/home/ubuntu/qemu/build/all/tests/qemu-iotests/socket_scm_helper
--- /home/ubuntu/qemu/tests/qemu-iotests/tests/qsd-jobs.out
+++ qsd-jobs.out.bad
@@ -9,11 +9,11 @@
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id":
"job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id":
"job0"}}
{"return": {}}
+{"return": {}}
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id":
"job0"}}
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id":
"job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id":
"job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0,
"offset": 0, "speed": 0, "type": "commit"}}
-{"return": {}}
-{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id":
"job0"}}
-{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id":
"job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id":
"job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id":
"job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
"event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0,
"offset": 0, "speed": 0, "type": "commit"}}
Not run: 172 186 192 220 287
Failures: qsd-jobs
Failed 1 of 118 iotests
thanks
-- PMM
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches
2021-03-24 18:05 ` [PULL 0/2] Block patches Peter Maydell
@ 2021-03-24 20:18 ` Vladimir Sementsov-Ogievskiy
2021-03-24 20:42 ` Peter Maydell
0 siblings, 1 reply; 12+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-03-24 20:18 UTC (permalink / raw)
To: Peter Maydell, Stefan Hajnoczi
Cc: QEMU Developers, John Snow, Juan Quintela,
Dr. David Alan Gilbert, Eric Blake, Fam Zheng,
Michael S. Tsirkin, Eduardo Habkost, Kevin Wolf,
Marcel Apfelbaum, Qemu-block, Max Reitz
24.03.2021 21:05, Peter Maydell wrote:
> On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>
>> The following changes since commit 67c1115edd98f388ca89dd38322ea3fadf034523:
>>
>> Merge remote-tracking branch 'remotes/kraxel/tags/ui-20210323-pull-request' into staging (2021-03-23 23:47:30 +0000)
>>
>> are available in the Git repository at:
>>
>> https://gitlab.com/stefanha/qemu.git tags/block-pull-request
>>
>> for you to fetch changes up to 3460fd7f3959d1fa7bcc255796844aa261c805a4:
>>
>> migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps (2021-03-24 13:41:19 +0000)
>>
>> ----------------------------------------------------------------
>> Pull request
>>
>> This dirty bitmap fix solves a crash that can be triggered in the destination
>> QEMU process during live migration.
>>
>> ----------------------------------------------------------------
>>
>> Vladimir Sementsov-Ogievskiy (2):
>> migration/block-dirty-bitmap: make incoming disabled bitmaps busy
>> migrate-bitmaps-postcopy-test: check that we can't remove in-flight
>> bitmaps
>
> This failed the 'qsd-jobs' iotest on s390x:
>
>
> TEST iotest-qcow2: 309
> TEST iotest-qcow2: 313
> TEST iotest-qcow2: qsd-jobs [fail]
> QEMU --
> "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-system-s390x"
> -nodefaults -display none -accel qtest
> QEMU_IMG -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-img"
> QEMU_IO --
> "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-io" --cache
> writeback --aio threads -f qcow2
> QEMU_NBD -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT -- qcow2
> IMGPROTO -- file
> PLATFORM -- Linux/s390x qemu01 4.15.0-132-generic
> TEST_DIR -- /home/ubuntu/qemu/build/all/tests/qemu-iotests/scratch
> SOCK_DIR -- /tmp/tmp807j_qyh
> SOCKET_SCM_HELPER --
> /home/ubuntu/qemu/build/all/tests/qemu-iotests/socket_scm_helper
> --- /home/ubuntu/qemu/tests/qemu-iotests/tests/qsd-jobs.out
> +++ qsd-jobs.out.bad
> @@ -9,11 +9,11 @@
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id":
> "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id":
> "job0"}}
> {"return": {}}
> +{"return": {}}
> +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id":
> "job0"}}
> +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id":
> "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id":
> "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0,
> "offset": 0, "speed": 0, "type": "commit"}}
> -{"return": {}}
> -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id":
> "job0"}}
> -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id":
> "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id":
> "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id":
> "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP},
> "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0,
> "offset": 0, "speed": 0, "type": "commit"}}
> Not run: 172 186 192 220 287
> Failures: qsd-jobs
> Failed 1 of 118 iotests
>
> thanks
> -- PMM
>
I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches
2021-03-24 20:18 ` Vladimir Sementsov-Ogievskiy
@ 2021-03-24 20:42 ` Peter Maydell
2021-03-25 9:56 ` Stefan Hajnoczi
0 siblings, 1 reply; 12+ messages in thread
From: Peter Maydell @ 2021-03-24 20:42 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: Fam Zheng, Kevin Wolf, Eduardo Habkost, Qemu-block,
Juan Quintela, John Snow, Michael S. Tsirkin,
Dr. David Alan Gilbert, QEMU Developers, Stefan Hajnoczi,
Max Reitz
On Wed, 24 Mar 2021 at 20:18, Vladimir Sementsov-Ogievskiy
<vsementsov@virtuozzo.com> wrote:
>
> 24.03.2021 21:05, Peter Maydell wrote:
> > On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >>
> >> Vladimir Sementsov-Ogievskiy (2):
> >> migration/block-dirty-bitmap: make incoming disabled bitmaps busy
> >> migrate-bitmaps-postcopy-test: check that we can't remove in-flight
> >> bitmaps
> >
> > This failed the 'qsd-jobs' iotest on s390x:
> I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test.
It's possible it's an intermittent, but it's not one I've seen
before. We still have lots of time this release cycle to figure
out the issue and get this fix in.
-- PMM
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches
2021-03-24 20:42 ` Peter Maydell
@ 2021-03-25 9:56 ` Stefan Hajnoczi
2021-03-25 10:17 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 12+ messages in thread
From: Stefan Hajnoczi @ 2021-03-25 9:56 UTC (permalink / raw)
To: Peter Maydell
Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy,
Eduardo Habkost, Qemu-block, Juan Quintela, John Snow,
Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert,
Max Reitz
[-- Attachment #1: Type: text/plain, Size: 1084 bytes --]
On Wed, Mar 24, 2021 at 08:42:27PM +0000, Peter Maydell wrote:
> On Wed, 24 Mar 2021 at 20:18, Vladimir Sementsov-Ogievskiy
> <vsementsov@virtuozzo.com> wrote:
> >
> > 24.03.2021 21:05, Peter Maydell wrote:
> > > On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > >>
> > >> Vladimir Sementsov-Ogievskiy (2):
> > >> migration/block-dirty-bitmap: make incoming disabled bitmaps busy
> > >> migrate-bitmaps-postcopy-test: check that we can't remove in-flight
> > >> bitmaps
> > >
> > > This failed the 'qsd-jobs' iotest on s390x:
>
> > I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test.
>
> It's possible it's an intermittent, but it's not one I've seen
> before. We still have lots of time this release cycle to figure
> out the issue and get this fix in.
Vladimir: I'll get hold of an s390 machine and try to reproduce the
failure. I should have some news by Monday.
Let's put the pull request on hold for now.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches
2021-03-25 9:56 ` Stefan Hajnoczi
@ 2021-03-25 10:17 ` Vladimir Sementsov-Ogievskiy
2021-03-25 16:28 ` Stefan Hajnoczi
0 siblings, 1 reply; 12+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-03-25 10:17 UTC (permalink / raw)
To: Stefan Hajnoczi, Peter Maydell
Cc: QEMU Developers, John Snow, Juan Quintela,
Dr. David Alan Gilbert, Eric Blake, Fam Zheng,
Michael S. Tsirkin, Eduardo Habkost, Kevin Wolf,
Marcel Apfelbaum, Qemu-block, Max Reitz
25.03.2021 12:56, Stefan Hajnoczi wrote:
> On Wed, Mar 24, 2021 at 08:42:27PM +0000, Peter Maydell wrote:
>> On Wed, 24 Mar 2021 at 20:18, Vladimir Sementsov-Ogievskiy
>> <vsementsov@virtuozzo.com> wrote:
>>>
>>> 24.03.2021 21:05, Peter Maydell wrote:
>>>> On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>>>
>>>>> Vladimir Sementsov-Ogievskiy (2):
>>>>> migration/block-dirty-bitmap: make incoming disabled bitmaps busy
>>>>> migrate-bitmaps-postcopy-test: check that we can't remove in-flight
>>>>> bitmaps
>>>>
>>>> This failed the 'qsd-jobs' iotest on s390x:
>>
>>> I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test.
>>
>> It's possible it's an intermittent, but it's not one I've seen
>> before. We still have lots of time this release cycle to figure
>> out the issue and get this fix in.
>
> Vladimir: I'll get hold of an s390 machine and try to reproduce the
> failure. I should have some news by Monday.
Thanks! My path modifies migration/block-dirty-bitmap.c. qsd-jobs runs block-commit and block-stream jobs and don't start any kind of migration or snapshot or savevm, so it seems impossible that qsd-jobs runs the code touched by my patch..
>
> Let's put the pull request on hold for now.
>
> Stefan
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches
2021-03-25 10:17 ` Vladimir Sementsov-Ogievskiy
@ 2021-03-25 16:28 ` Stefan Hajnoczi
2021-03-25 16:36 ` Peter Maydell
0 siblings, 1 reply; 12+ messages in thread
From: Stefan Hajnoczi @ 2021-03-25 16:28 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: Fam Zheng, Peter Maydell, Eduardo Habkost, Qemu-block,
Juan Quintela, John Snow, Michael S. Tsirkin, QEMU Developers,
Dr. David Alan Gilbert, Kevin Wolf, Max Reitz
[-- Attachment #1: Type: text/plain, Size: 3741 bytes --]
On Thu, Mar 25, 2021 at 01:17:50PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> 25.03.2021 12:56, Stefan Hajnoczi wrote:
> > On Wed, Mar 24, 2021 at 08:42:27PM +0000, Peter Maydell wrote:
> > > On Wed, 24 Mar 2021 at 20:18, Vladimir Sementsov-Ogievskiy
> > > <vsementsov@virtuozzo.com> wrote:
> > > >
> > > > 24.03.2021 21:05, Peter Maydell wrote:
> > > > > On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > > > >
> > > > > > Vladimir Sementsov-Ogievskiy (2):
> > > > > > migration/block-dirty-bitmap: make incoming disabled bitmaps busy
> > > > > > migrate-bitmaps-postcopy-test: check that we can't remove in-flight
> > > > > > bitmaps
> > > > >
> > > > > This failed the 'qsd-jobs' iotest on s390x:
> > >
> > > > I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test.
> > >
> > > It's possible it's an intermittent, but it's not one I've seen
> > > before. We still have lots of time this release cycle to figure
> > > out the issue and get this fix in.
> >
> > Vladimir: I'll get hold of an s390 machine and try to reproduce the
> > failure. I should have some news by Monday.
>
> Thanks! My path modifies migration/block-dirty-bitmap.c. qsd-jobs runs block-commit and block-stream jobs and don't start any kind of migration or snapshot or savevm, so it seems impossible that qsd-jobs runs the code touched by my patch..
Confirmed. The failure is not related to this pull request.
I reproduced the same s390x host failure on commit
9e2e9fe3df9f539f8b6941ceb96d25355fdae47e (HEAD -> master, tag:
v6.0.0-rc0, origin/master, origin/HEAD):
qsd-jobs fail [11:04:58] [11:04:58] 0.1s output mismatch (see qsd-jobs.out.bad)
--- /root/qemu/tests/qemu-iotests/tests/qsd-jobs.out
+++ qsd-jobs.out.bad
@@ -9,11 +9,11 @@
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}}
{"return": {}}
+{"return": {}}
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id": "job0"}}
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}
-{"return": {}}
-{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "job0"}}
-{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "job0"}}
{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}
Peter: Please merge this pull request since the failure is pre-existing.
Thanks,
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches
2021-03-25 16:28 ` Stefan Hajnoczi
@ 2021-03-25 16:36 ` Peter Maydell
2021-03-25 17:40 ` Stefan Hajnoczi
0 siblings, 1 reply; 12+ messages in thread
From: Peter Maydell @ 2021-03-25 16:36 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy,
Eduardo Habkost, Qemu-block, Juan Quintela, John Snow,
Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert,
Max Reitz
On Thu, 25 Mar 2021 at 16:28, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Thu, Mar 25, 2021 at 01:17:50PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > Thanks! My path modifies migration/block-dirty-bitmap.c. qsd-jobs runs block-commit and block-stream jobs and don't start any kind of migration or snapshot or savevm, so it seems impossible that qsd-jobs runs the code touched by my patch..
>
> Confirmed. The failure is not related to this pull request.
>
> I reproduced the same s390x host failure on commit
> 9e2e9fe3df9f539f8b6941ceb96d25355fdae47e (HEAD -> master, tag:
> v6.0.0-rc0, origin/master, origin/HEAD):
>
> qsd-jobs fail [11:04:58] [11:04:58] 0.1s output mismatch (see qsd-jobs.out.bad)
> --- /root/qemu/tests/qemu-iotests/tests/qsd-jobs.out
> +++ qsd-jobs.out.bad
> @@ -9,11 +9,11 @@
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}}
> {"return": {}}
> +{"return": {}}
> +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id": "job0"}}
> +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}
> -{"return": {}}
> -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "job0"}}
> -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "job0"}}
> {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}
>
> Peter: Please merge this pull request since the failure is pre-existing.
OK, I'll run the pullreq through merge testing again.
Could somebody on the block team look at this s390 host intermittent ?
thanks
-- PMM
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches
2021-03-25 16:36 ` Peter Maydell
@ 2021-03-25 17:40 ` Stefan Hajnoczi
0 siblings, 0 replies; 12+ messages in thread
From: Stefan Hajnoczi @ 2021-03-25 17:40 UTC (permalink / raw)
To: Peter Maydell
Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy,
Eduardo Habkost, Qemu-block, Juan Quintela, John Snow,
Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert,
Max Reitz
[-- Attachment #1: Type: text/plain, Size: 3043 bytes --]
On Thu, Mar 25, 2021 at 04:36:43PM +0000, Peter Maydell wrote:
> On Thu, 25 Mar 2021 at 16:28, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > On Thu, Mar 25, 2021 at 01:17:50PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > Thanks! My path modifies migration/block-dirty-bitmap.c. qsd-jobs runs block-commit and block-stream jobs and don't start any kind of migration or snapshot or savevm, so it seems impossible that qsd-jobs runs the code touched by my patch..
> >
> > Confirmed. The failure is not related to this pull request.
> >
> > I reproduced the same s390x host failure on commit
> > 9e2e9fe3df9f539f8b6941ceb96d25355fdae47e (HEAD -> master, tag:
> > v6.0.0-rc0, origin/master, origin/HEAD):
> >
> > qsd-jobs fail [11:04:58] [11:04:58] 0.1s output mismatch (see qsd-jobs.out.bad)
> > --- /root/qemu/tests/qemu-iotests/tests/qsd-jobs.out
> > +++ qsd-jobs.out.bad
> > @@ -9,11 +9,11 @@
> > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}}
> > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}}
> > {"return": {}}
> > +{"return": {}}
> > +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id": "job0"}}
> > +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}}
> > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}}
> > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}
> > -{"return": {}}
> > -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "job0"}}
> > -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}}
> > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "job0"}}
> > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "job0"}}
> > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}}
> >
> > Peter: Please merge this pull request since the failure is pre-existing.
>
> OK, I'll run the pullreq through merge testing again.
>
> Could somebody on the block team look at this s390 host intermittent ?
I need to look into more s390x stuff on Monday so I'd be happy to dig
deeper then.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches
2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi
` (2 preceding siblings ...)
2021-03-24 18:05 ` [PULL 0/2] Block patches Peter Maydell
@ 2021-03-26 10:22 ` Peter Maydell
3 siblings, 0 replies; 12+ messages in thread
From: Peter Maydell @ 2021-03-26 10:22 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy,
Eduardo Habkost, Qemu-block, Juan Quintela, John Snow,
Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert,
Max Reitz
On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> The following changes since commit 67c1115edd98f388ca89dd38322ea3fadf034523:
>
> Merge remote-tracking branch 'remotes/kraxel/tags/ui-20210323-pull-request' into staging (2021-03-23 23:47:30 +0000)
>
> are available in the Git repository at:
>
> https://gitlab.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 3460fd7f3959d1fa7bcc255796844aa261c805a4:
>
> migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps (2021-03-24 13:41:19 +0000)
>
> ----------------------------------------------------------------
> Pull request
>
> This dirty bitmap fix solves a crash that can be triggered in the destination
> QEMU process during live migration.
>
Applied, thanks.
Please update the changelog at https://wiki.qemu.org/ChangeLog/6.0
for any user-visible changes.
-- PMM
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-03-26 10:24 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi
2021-03-24 14:52 ` [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy Stefan Hajnoczi
2021-03-24 14:52 ` [PULL 2/2] migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps Stefan Hajnoczi
2021-03-24 18:05 ` [PULL 0/2] Block patches Peter Maydell
2021-03-24 20:18 ` Vladimir Sementsov-Ogievskiy
2021-03-24 20:42 ` Peter Maydell
2021-03-25 9:56 ` Stefan Hajnoczi
2021-03-25 10:17 ` Vladimir Sementsov-Ogievskiy
2021-03-25 16:28 ` Stefan Hajnoczi
2021-03-25 16:36 ` Peter Maydell
2021-03-25 17:40 ` Stefan Hajnoczi
2021-03-26 10:22 ` Peter Maydell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).