* [PULL 0/2] Block patches
@ 2021-03-24 14:52 Stefan Hajnoczi
2021-03-24 14:52 ` [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy Stefan Hajnoczi
` (3 more replies)
0 siblings, 4 replies; 12+ messages in thread
From: Stefan Hajnoczi @ 2021-03-24 14:52 UTC (permalink / raw)
To: Peter Maydell, qemu-devel
Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy,
Eduardo Habkost, qemu-block, Juan Quintela, John Snow,
Michael S. Tsirkin, Dr. David Alan Gilbert, Stefan Hajnoczi,
Max Reitz
The following changes since commit 67c1115edd98f388ca89dd38322ea3fadf034523:
Merge remote-tracking branch 'remotes/kraxel/tags/ui-20210323-pull-request' into staging (2021-03-23 23:47:30 +0000)
are available in the Git repository at:
https://gitlab.com/stefanha/qemu.git tags/block-pull-request
for you to fetch changes up to 3460fd7f3959d1fa7bcc255796844aa261c805a4:
migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps (2021-03-24 13:41:19 +0000)
----------------------------------------------------------------
Pull request
This dirty bitmap fix solves a crash that can be triggered in the destination
QEMU process during live migration.
----------------------------------------------------------------
Vladimir Sementsov-Ogievskiy (2):
migration/block-dirty-bitmap: make incoming disabled bitmaps busy
migrate-bitmaps-postcopy-test: check that we can't remove in-flight
bitmaps
migration/block-dirty-bitmap.c | 6 ++++++
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 10 ++++++++++
2 files changed, 16 insertions(+)
--
2.30.2
^ permalink raw reply [flat|nested] 12+ messages in thread* [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy 2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi @ 2021-03-24 14:52 ` Stefan Hajnoczi 2021-03-24 14:52 ` [PULL 2/2] migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps Stefan Hajnoczi ` (2 subsequent siblings) 3 siblings, 0 replies; 12+ messages in thread From: Stefan Hajnoczi @ 2021-03-24 14:52 UTC (permalink / raw) To: Peter Maydell, qemu-devel Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy, Eduardo Habkost, qemu-block, Juan Quintela, John Snow, Michael S. Tsirkin, Dr. David Alan Gilbert, Stefan Hajnoczi, Max Reitz From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Incoming enabled bitmaps are busy, because we do bdrv_dirty_bitmap_create_successor() for them. But disabled bitmaps being migrated are not marked busy, and user can remove them during the incoming migration. Then we may crash in cancel_incoming_locked() when try to remove the bitmap that was already removed by user, like this: #0 qemu_mutex_lock_impl (mutex=0x5593d88c50d1, file=0x559680554b20 "../block/dirty-bitmap.c", line=64) at ../util/qemu-thread-posix.c:77 #1 bdrv_dirty_bitmaps_lock (bs=0x5593d88c0ee9) at ../block/dirty-bitmap.c:64 #2 bdrv_release_dirty_bitmap (bitmap=0x5596810e9570) at ../block/dirty-bitmap.c:362 #3 cancel_incoming_locked (s=0x559680be8208 <dbm_state+40>) at ../migration/block-dirty-bitmap.c:918 #4 dirty_bitmap_load (f=0x559681d02b10, opaque=0x559680be81e0 <dbm_state>, version_id=1) at ../migration/block-dirty-bitmap.c:1194 #5 vmstate_load (f=0x559681d02b10, se=0x559680fb5810) at ../migration/savevm.c:908 #6 qemu_loadvm_section_part_end (f=0x559681d02b10, mis=0x559680fb4a30) at ../migration/savevm.c:2473 #7 qemu_loadvm_state_main (f=0x559681d02b10, mis=0x559680fb4a30) at ../migration/savevm.c:2626 #8 postcopy_ram_listen_thread (opaque=0x0) at ../migration/savevm.c:1871 #9 qemu_thread_start (args=0x5596817ccd10) at ../util/qemu-thread-posix.c:521 #10 start_thread () at /lib64/libpthread.so.0 #11 clone () at /lib64/libc.so.6 Note bs pointer taken from bitmap: it's definitely bad aligned. That's because we are in use after free, bitmap is already freed. So, let's make disabled bitmaps (being migrated) busy during incoming migration. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Message-Id: <20210322094906.5079-2-vsementsov@virtuozzo.com> --- migration/block-dirty-bitmap.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c index 975093610a..35f5ef688d 100644 --- a/migration/block-dirty-bitmap.c +++ b/migration/block-dirty-bitmap.c @@ -839,6 +839,8 @@ static int dirty_bitmap_load_start(QEMUFile *f, DBMLoadState *s) error_report_err(local_err); return -EINVAL; } + } else { + bdrv_dirty_bitmap_set_busy(s->bitmap, true); } b = g_new(LoadBitmapState, 1); @@ -914,6 +916,8 @@ static void cancel_incoming_locked(DBMLoadState *s) assert(!s->before_vm_start_handled || !b->migrated); if (bdrv_dirty_bitmap_has_successor(b->bitmap)) { bdrv_reclaim_dirty_bitmap(b->bitmap, &error_abort); + } else { + bdrv_dirty_bitmap_set_busy(b->bitmap, false); } bdrv_release_dirty_bitmap(b->bitmap); } @@ -951,6 +955,8 @@ static void dirty_bitmap_load_complete(QEMUFile *f, DBMLoadState *s) if (bdrv_dirty_bitmap_has_successor(s->bitmap)) { bdrv_reclaim_dirty_bitmap(s->bitmap, &error_abort); + } else { + bdrv_dirty_bitmap_set_busy(s->bitmap, false); } for (item = s->bitmaps; item; item = g_slist_next(item)) { -- 2.30.2 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PULL 2/2] migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps 2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi 2021-03-24 14:52 ` [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy Stefan Hajnoczi @ 2021-03-24 14:52 ` Stefan Hajnoczi 2021-03-24 18:05 ` [PULL 0/2] Block patches Peter Maydell 2021-03-26 10:22 ` Peter Maydell 3 siblings, 0 replies; 12+ messages in thread From: Stefan Hajnoczi @ 2021-03-24 14:52 UTC (permalink / raw) To: Peter Maydell, qemu-devel Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy, Eduardo Habkost, qemu-block, Juan Quintela, John Snow, Michael S. Tsirkin, Dr. David Alan Gilbert, Stefan Hajnoczi, Max Reitz From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Check that we can't remove bitmaps being migrated on destination vm. The new check proves that previous commit helps. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Message-Id: <20210322094906.5079-3-vsementsov@virtuozzo.com> --- tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test index d046ebeb94..584062b412 100755 --- a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test +++ b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test @@ -224,6 +224,16 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase): self.start_postcopy() self.vm_b_events += self.vm_b.get_qmp_events() + + # While being here, let's check that we can't remove in-flight bitmaps. + for vm in (self.vm_a, self.vm_b): + for i in range(0, nb_bitmaps): + result = vm.qmp('block-dirty-bitmap-remove', node='drive0', + name=f'bitmap{i}') + self.assert_qmp(result, 'error/desc', + f"Bitmap 'bitmap{i}' is currently in use by " + "another operation and cannot be used") + self.vm_b.shutdown() # recreate vm_b, so there is no incoming option, which prevents # loading bitmaps from disk -- 2.30.2 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches 2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi 2021-03-24 14:52 ` [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy Stefan Hajnoczi 2021-03-24 14:52 ` [PULL 2/2] migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps Stefan Hajnoczi @ 2021-03-24 18:05 ` Peter Maydell 2021-03-24 20:18 ` Vladimir Sementsov-Ogievskiy 2021-03-26 10:22 ` Peter Maydell 3 siblings, 1 reply; 12+ messages in thread From: Peter Maydell @ 2021-03-24 18:05 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy, Eduardo Habkost, Qemu-block, Juan Quintela, John Snow, Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert, Max Reitz On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > The following changes since commit 67c1115edd98f388ca89dd38322ea3fadf034523: > > Merge remote-tracking branch 'remotes/kraxel/tags/ui-20210323-pull-request' into staging (2021-03-23 23:47:30 +0000) > > are available in the Git repository at: > > https://gitlab.com/stefanha/qemu.git tags/block-pull-request > > for you to fetch changes up to 3460fd7f3959d1fa7bcc255796844aa261c805a4: > > migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps (2021-03-24 13:41:19 +0000) > > ---------------------------------------------------------------- > Pull request > > This dirty bitmap fix solves a crash that can be triggered in the destination > QEMU process during live migration. > > ---------------------------------------------------------------- > > Vladimir Sementsov-Ogievskiy (2): > migration/block-dirty-bitmap: make incoming disabled bitmaps busy > migrate-bitmaps-postcopy-test: check that we can't remove in-flight > bitmaps This failed the 'qsd-jobs' iotest on s390x: TEST iotest-qcow2: 309 TEST iotest-qcow2: 313 TEST iotest-qcow2: qsd-jobs [fail] QEMU -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-system-s390x" -nodefaults -display none -accel qtest QEMU_IMG -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-img" QEMU_IO -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-io" --cache writeback --aio threads -f qcow2 QEMU_NBD -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-nbd" IMGFMT -- qcow2 IMGPROTO -- file PLATFORM -- Linux/s390x qemu01 4.15.0-132-generic TEST_DIR -- /home/ubuntu/qemu/build/all/tests/qemu-iotests/scratch SOCK_DIR -- /tmp/tmp807j_qyh SOCKET_SCM_HELPER -- /home/ubuntu/qemu/build/all/tests/qemu-iotests/socket_scm_helper --- /home/ubuntu/qemu/tests/qemu-iotests/tests/qsd-jobs.out +++ qsd-jobs.out.bad @@ -9,11 +9,11 @@ {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} {"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id": "job0"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} -{"return": {}} -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "job0"}} -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} Not run: 172 186 192 220 287 Failures: qsd-jobs Failed 1 of 118 iotests thanks -- PMM ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches 2021-03-24 18:05 ` [PULL 0/2] Block patches Peter Maydell @ 2021-03-24 20:18 ` Vladimir Sementsov-Ogievskiy 2021-03-24 20:42 ` Peter Maydell 0 siblings, 1 reply; 12+ messages in thread From: Vladimir Sementsov-Ogievskiy @ 2021-03-24 20:18 UTC (permalink / raw) To: Peter Maydell, Stefan Hajnoczi Cc: QEMU Developers, John Snow, Juan Quintela, Dr. David Alan Gilbert, Eric Blake, Fam Zheng, Michael S. Tsirkin, Eduardo Habkost, Kevin Wolf, Marcel Apfelbaum, Qemu-block, Max Reitz 24.03.2021 21:05, Peter Maydell wrote: > On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote: >> >> The following changes since commit 67c1115edd98f388ca89dd38322ea3fadf034523: >> >> Merge remote-tracking branch 'remotes/kraxel/tags/ui-20210323-pull-request' into staging (2021-03-23 23:47:30 +0000) >> >> are available in the Git repository at: >> >> https://gitlab.com/stefanha/qemu.git tags/block-pull-request >> >> for you to fetch changes up to 3460fd7f3959d1fa7bcc255796844aa261c805a4: >> >> migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps (2021-03-24 13:41:19 +0000) >> >> ---------------------------------------------------------------- >> Pull request >> >> This dirty bitmap fix solves a crash that can be triggered in the destination >> QEMU process during live migration. >> >> ---------------------------------------------------------------- >> >> Vladimir Sementsov-Ogievskiy (2): >> migration/block-dirty-bitmap: make incoming disabled bitmaps busy >> migrate-bitmaps-postcopy-test: check that we can't remove in-flight >> bitmaps > > This failed the 'qsd-jobs' iotest on s390x: > > > TEST iotest-qcow2: 309 > TEST iotest-qcow2: 313 > TEST iotest-qcow2: qsd-jobs [fail] > QEMU -- > "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-system-s390x" > -nodefaults -display none -accel qtest > QEMU_IMG -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-img" > QEMU_IO -- > "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-io" --cache > writeback --aio threads -f qcow2 > QEMU_NBD -- "/home/ubuntu/qemu/build/all/tests/qemu-iotests/../../qemu-nbd" > IMGFMT -- qcow2 > IMGPROTO -- file > PLATFORM -- Linux/s390x qemu01 4.15.0-132-generic > TEST_DIR -- /home/ubuntu/qemu/build/all/tests/qemu-iotests/scratch > SOCK_DIR -- /tmp/tmp807j_qyh > SOCKET_SCM_HELPER -- > /home/ubuntu/qemu/build/all/tests/qemu-iotests/socket_scm_helper > --- /home/ubuntu/qemu/tests/qemu-iotests/tests/qsd-jobs.out > +++ qsd-jobs.out.bad > @@ -9,11 +9,11 @@ > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": > "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": > "job0"}} > {"return": {}} > +{"return": {}} > +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id": > "job0"}} > +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": > "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": > "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0, > "offset": 0, "speed": 0, "type": "commit"}} > -{"return": {}} > -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": > "job0"}} > -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": > "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": > "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": > "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, > "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0, > "offset": 0, "speed": 0, "type": "commit"}} > Not run: 172 186 192 220 287 > Failures: qsd-jobs > Failed 1 of 118 iotests > > thanks > -- PMM > I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test. -- Best regards, Vladimir ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches 2021-03-24 20:18 ` Vladimir Sementsov-Ogievskiy @ 2021-03-24 20:42 ` Peter Maydell 2021-03-25 9:56 ` Stefan Hajnoczi 0 siblings, 1 reply; 12+ messages in thread From: Peter Maydell @ 2021-03-24 20:42 UTC (permalink / raw) To: Vladimir Sementsov-Ogievskiy Cc: Fam Zheng, Kevin Wolf, Eduardo Habkost, Qemu-block, Juan Quintela, John Snow, Michael S. Tsirkin, Dr. David Alan Gilbert, QEMU Developers, Stefan Hajnoczi, Max Reitz On Wed, 24 Mar 2021 at 20:18, Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> wrote: > > 24.03.2021 21:05, Peter Maydell wrote: > > On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote: > >> > >> Vladimir Sementsov-Ogievskiy (2): > >> migration/block-dirty-bitmap: make incoming disabled bitmaps busy > >> migrate-bitmaps-postcopy-test: check that we can't remove in-flight > >> bitmaps > > > > This failed the 'qsd-jobs' iotest on s390x: > I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test. It's possible it's an intermittent, but it's not one I've seen before. We still have lots of time this release cycle to figure out the issue and get this fix in. -- PMM ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches 2021-03-24 20:42 ` Peter Maydell @ 2021-03-25 9:56 ` Stefan Hajnoczi 2021-03-25 10:17 ` Vladimir Sementsov-Ogievskiy 0 siblings, 1 reply; 12+ messages in thread From: Stefan Hajnoczi @ 2021-03-25 9:56 UTC (permalink / raw) To: Peter Maydell Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy, Eduardo Habkost, Qemu-block, Juan Quintela, John Snow, Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert, Max Reitz [-- Attachment #1: Type: text/plain, Size: 1084 bytes --] On Wed, Mar 24, 2021 at 08:42:27PM +0000, Peter Maydell wrote: > On Wed, 24 Mar 2021 at 20:18, Vladimir Sementsov-Ogievskiy > <vsementsov@virtuozzo.com> wrote: > > > > 24.03.2021 21:05, Peter Maydell wrote: > > > On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > >> > > >> Vladimir Sementsov-Ogievskiy (2): > > >> migration/block-dirty-bitmap: make incoming disabled bitmaps busy > > >> migrate-bitmaps-postcopy-test: check that we can't remove in-flight > > >> bitmaps > > > > > > This failed the 'qsd-jobs' iotest on s390x: > > > I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test. > > It's possible it's an intermittent, but it's not one I've seen > before. We still have lots of time this release cycle to figure > out the issue and get this fix in. Vladimir: I'll get hold of an s390 machine and try to reproduce the failure. I should have some news by Monday. Let's put the pull request on hold for now. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches 2021-03-25 9:56 ` Stefan Hajnoczi @ 2021-03-25 10:17 ` Vladimir Sementsov-Ogievskiy 2021-03-25 16:28 ` Stefan Hajnoczi 0 siblings, 1 reply; 12+ messages in thread From: Vladimir Sementsov-Ogievskiy @ 2021-03-25 10:17 UTC (permalink / raw) To: Stefan Hajnoczi, Peter Maydell Cc: QEMU Developers, John Snow, Juan Quintela, Dr. David Alan Gilbert, Eric Blake, Fam Zheng, Michael S. Tsirkin, Eduardo Habkost, Kevin Wolf, Marcel Apfelbaum, Qemu-block, Max Reitz 25.03.2021 12:56, Stefan Hajnoczi wrote: > On Wed, Mar 24, 2021 at 08:42:27PM +0000, Peter Maydell wrote: >> On Wed, 24 Mar 2021 at 20:18, Vladimir Sementsov-Ogievskiy >> <vsementsov@virtuozzo.com> wrote: >>> >>> 24.03.2021 21:05, Peter Maydell wrote: >>>> On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote: >>>>> >>>>> Vladimir Sementsov-Ogievskiy (2): >>>>> migration/block-dirty-bitmap: make incoming disabled bitmaps busy >>>>> migrate-bitmaps-postcopy-test: check that we can't remove in-flight >>>>> bitmaps >>>> >>>> This failed the 'qsd-jobs' iotest on s390x: >> >>> I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test. >> >> It's possible it's an intermittent, but it's not one I've seen >> before. We still have lots of time this release cycle to figure >> out the issue and get this fix in. > > Vladimir: I'll get hold of an s390 machine and try to reproduce the > failure. I should have some news by Monday. Thanks! My path modifies migration/block-dirty-bitmap.c. qsd-jobs runs block-commit and block-stream jobs and don't start any kind of migration or snapshot or savevm, so it seems impossible that qsd-jobs runs the code touched by my patch.. > > Let's put the pull request on hold for now. > > Stefan > -- Best regards, Vladimir ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches 2021-03-25 10:17 ` Vladimir Sementsov-Ogievskiy @ 2021-03-25 16:28 ` Stefan Hajnoczi 2021-03-25 16:36 ` Peter Maydell 0 siblings, 1 reply; 12+ messages in thread From: Stefan Hajnoczi @ 2021-03-25 16:28 UTC (permalink / raw) To: Vladimir Sementsov-Ogievskiy Cc: Fam Zheng, Peter Maydell, Eduardo Habkost, Qemu-block, Juan Quintela, John Snow, Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert, Kevin Wolf, Max Reitz [-- Attachment #1: Type: text/plain, Size: 3741 bytes --] On Thu, Mar 25, 2021 at 01:17:50PM +0300, Vladimir Sementsov-Ogievskiy wrote: > 25.03.2021 12:56, Stefan Hajnoczi wrote: > > On Wed, Mar 24, 2021 at 08:42:27PM +0000, Peter Maydell wrote: > > > On Wed, 24 Mar 2021 at 20:18, Vladimir Sementsov-Ogievskiy > > > <vsementsov@virtuozzo.com> wrote: > > > > > > > > 24.03.2021 21:05, Peter Maydell wrote: > > > > > On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > > > > > > > > > > > Vladimir Sementsov-Ogievskiy (2): > > > > > > migration/block-dirty-bitmap: make incoming disabled bitmaps busy > > > > > > migrate-bitmaps-postcopy-test: check that we can't remove in-flight > > > > > > bitmaps > > > > > > > > > > This failed the 'qsd-jobs' iotest on s390x: > > > > > > > I can't believe it related. My commit modifies bitmap status during bitmaps migration on target vm. There is no kind of migration in qsd-jobs test. > > > > > > It's possible it's an intermittent, but it's not one I've seen > > > before. We still have lots of time this release cycle to figure > > > out the issue and get this fix in. > > > > Vladimir: I'll get hold of an s390 machine and try to reproduce the > > failure. I should have some news by Monday. > > Thanks! My path modifies migration/block-dirty-bitmap.c. qsd-jobs runs block-commit and block-stream jobs and don't start any kind of migration or snapshot or savevm, so it seems impossible that qsd-jobs runs the code touched by my patch.. Confirmed. The failure is not related to this pull request. I reproduced the same s390x host failure on commit 9e2e9fe3df9f539f8b6941ceb96d25355fdae47e (HEAD -> master, tag: v6.0.0-rc0, origin/master, origin/HEAD): qsd-jobs fail [11:04:58] [11:04:58] 0.1s output mismatch (see qsd-jobs.out.bad) --- /root/qemu/tests/qemu-iotests/tests/qsd-jobs.out +++ qsd-jobs.out.bad @@ -9,11 +9,11 @@ {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} {"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id": "job0"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} -{"return": {}} -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "job0"}} -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} Peter: Please merge this pull request since the failure is pre-existing. Thanks, Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches 2021-03-25 16:28 ` Stefan Hajnoczi @ 2021-03-25 16:36 ` Peter Maydell 2021-03-25 17:40 ` Stefan Hajnoczi 0 siblings, 1 reply; 12+ messages in thread From: Peter Maydell @ 2021-03-25 16:36 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy, Eduardo Habkost, Qemu-block, Juan Quintela, John Snow, Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert, Max Reitz On Thu, 25 Mar 2021 at 16:28, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > On Thu, Mar 25, 2021 at 01:17:50PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > Thanks! My path modifies migration/block-dirty-bitmap.c. qsd-jobs runs block-commit and block-stream jobs and don't start any kind of migration or snapshot or savevm, so it seems impossible that qsd-jobs runs the code touched by my patch.. > > Confirmed. The failure is not related to this pull request. > > I reproduced the same s390x host failure on commit > 9e2e9fe3df9f539f8b6941ceb96d25355fdae47e (HEAD -> master, tag: > v6.0.0-rc0, origin/master, origin/HEAD): > > qsd-jobs fail [11:04:58] [11:04:58] 0.1s output mismatch (see qsd-jobs.out.bad) > --- /root/qemu/tests/qemu-iotests/tests/qsd-jobs.out > +++ qsd-jobs.out.bad > @@ -9,11 +9,11 @@ > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} > {"return": {}} > +{"return": {}} > +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id": "job0"}} > +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} > -{"return": {}} > -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "job0"}} > -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "job0"}} > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} > > Peter: Please merge this pull request since the failure is pre-existing. OK, I'll run the pullreq through merge testing again. Could somebody on the block team look at this s390 host intermittent ? thanks -- PMM ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches 2021-03-25 16:36 ` Peter Maydell @ 2021-03-25 17:40 ` Stefan Hajnoczi 0 siblings, 0 replies; 12+ messages in thread From: Stefan Hajnoczi @ 2021-03-25 17:40 UTC (permalink / raw) To: Peter Maydell Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy, Eduardo Habkost, Qemu-block, Juan Quintela, John Snow, Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert, Max Reitz [-- Attachment #1: Type: text/plain, Size: 3043 bytes --] On Thu, Mar 25, 2021 at 04:36:43PM +0000, Peter Maydell wrote: > On Thu, 25 Mar 2021 at 16:28, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > > > On Thu, Mar 25, 2021 at 01:17:50PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > Thanks! My path modifies migration/block-dirty-bitmap.c. qsd-jobs runs block-commit and block-stream jobs and don't start any kind of migration or snapshot or savevm, so it seems impossible that qsd-jobs runs the code touched by my patch.. > > > > Confirmed. The failure is not related to this pull request. > > > > I reproduced the same s390x host failure on commit > > 9e2e9fe3df9f539f8b6941ceb96d25355fdae47e (HEAD -> master, tag: > > v6.0.0-rc0, origin/master, origin/HEAD): > > > > qsd-jobs fail [11:04:58] [11:04:58] 0.1s output mismatch (see qsd-jobs.out.bad) > > --- /root/qemu/tests/qemu-iotests/tests/qsd-jobs.out > > +++ qsd-jobs.out.bad > > @@ -9,11 +9,11 @@ > > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}} > > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} > > {"return": {}} > > +{"return": {}} > > +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "paused", "id": "job0"}} > > +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} > > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}} > > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} > > -{"return": {}} > > -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "job0"}} > > -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "job0"}} > > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "job0"}} > > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "job0"}} > > {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} > > > > Peter: Please merge this pull request since the failure is pre-existing. > > OK, I'll run the pullreq through merge testing again. > > Could somebody on the block team look at this s390 host intermittent ? I need to look into more s390x stuff on Monday so I'd be happy to dig deeper then. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/2] Block patches 2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi ` (2 preceding siblings ...) 2021-03-24 18:05 ` [PULL 0/2] Block patches Peter Maydell @ 2021-03-26 10:22 ` Peter Maydell 3 siblings, 0 replies; 12+ messages in thread From: Peter Maydell @ 2021-03-26 10:22 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Fam Zheng, Kevin Wolf, Vladimir Sementsov-Ogievskiy, Eduardo Habkost, Qemu-block, Juan Quintela, John Snow, Michael S. Tsirkin, QEMU Developers, Dr. David Alan Gilbert, Max Reitz On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > The following changes since commit 67c1115edd98f388ca89dd38322ea3fadf034523: > > Merge remote-tracking branch 'remotes/kraxel/tags/ui-20210323-pull-request' into staging (2021-03-23 23:47:30 +0000) > > are available in the Git repository at: > > https://gitlab.com/stefanha/qemu.git tags/block-pull-request > > for you to fetch changes up to 3460fd7f3959d1fa7bcc255796844aa261c805a4: > > migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps (2021-03-24 13:41:19 +0000) > > ---------------------------------------------------------------- > Pull request > > This dirty bitmap fix solves a crash that can be triggered in the destination > QEMU process during live migration. > Applied, thanks. Please update the changelog at https://wiki.qemu.org/ChangeLog/6.0 for any user-visible changes. -- PMM ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-03-26 10:24 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-03-24 14:52 [PULL 0/2] Block patches Stefan Hajnoczi 2021-03-24 14:52 ` [PULL 1/2] migration/block-dirty-bitmap: make incoming disabled bitmaps busy Stefan Hajnoczi 2021-03-24 14:52 ` [PULL 2/2] migrate-bitmaps-postcopy-test: check that we can't remove in-flight bitmaps Stefan Hajnoczi 2021-03-24 18:05 ` [PULL 0/2] Block patches Peter Maydell 2021-03-24 20:18 ` Vladimir Sementsov-Ogievskiy 2021-03-24 20:42 ` Peter Maydell 2021-03-25 9:56 ` Stefan Hajnoczi 2021-03-25 10:17 ` Vladimir Sementsov-Ogievskiy 2021-03-25 16:28 ` Stefan Hajnoczi 2021-03-25 16:36 ` Peter Maydell 2021-03-25 17:40 ` Stefan Hajnoczi 2021-03-26 10:22 ` Peter Maydell
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).