[Qemu-devel] [PATCH v2] qemu-iotests: Extend non-shared storage migration test (194)

[Qemu-devel] [PATCH v2] qemu-iotests: Extend non-shared storage migration test (194)

[Kashyap Chamarthy]

This is the follow-up patch that was discussed[*] as part of feedback to
qemu-iotest 194.

Changes in this patch:

  - Supply 'job-id' parameter to `drive-mirror` invocation.

  - Issue QMP `block-job-cancel` command on the source QEMU to
    gracefully complete `drive-mirror` operation.

  - Stop the NBD server on the destination QEMU.

  - Finally, check for both the events: MIGRATION and
    BLOCK_JOB_COMPLETED

With the above, the test will also be (almost) in sync with the
procedure outlined in the document 'live-block-operations.rst'[+]
(section: "QMP invocation for live storage migration with
``drive-mirror`` + NBD").

[*] https://lists.nongnu.org/archive/html/qemu-devel/2017-08/msg04820.html
    -- qemu-iotests: add 194 non-shared storage migration test
[+] https://git.qemu.org/gitweb.cgi?p=qemu.git;a=blob;f=docs/interop/live-block-operations.rst

Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
---
Changes in v2:
 - Check for both the events: MIGRATION and BLOCK_JOB_COMPLETED (Eric)
---
 tests/qemu-iotests/194     | 25 +++++++++++++++++++------
 tests/qemu-iotests/194.out | 11 ++++++++---
 2 files changed, 27 insertions(+), 9 deletions(-)

diff --git a/tests/qemu-iotests/194 b/tests/qemu-iotests/194
index 8028111e21bed5cf4a2e8e32dc04aa5a9ea9caca..9d81189fe3e790d060da5a0fd41836f4b57f95c7 100755
--- a/tests/qemu-iotests/194
+++ b/tests/qemu-iotests/194
@@ -46,16 +46,17 @@ iotests.log('Launching NBD server on destination...')
 iotests.log(dest_vm.qmp('nbd-server-start', addr={'type': 'unix', 'data': {'path': nbd_sock_path}}))
 iotests.log(dest_vm.qmp('nbd-server-add', device='drive0', writable=True))
 
-iotests.log('Starting drive-mirror on source...')
+iotests.log('Starting `drive-mirror` on source...')
 iotests.log(source_vm.qmp(
               'drive-mirror',
               device='drive0',
               target='nbd+unix:///drive0?socket={0}'.format(nbd_sock_path),
               sync='full',
               format='raw', # always raw, the server handles the format
-              mode='existing'))
+              mode='existing',
+              job_id='mirror-job0'))
 
-iotests.log('Waiting for drive-mirror to complete...')
+iotests.log('Waiting for `drive-mirror` to complete...')
 iotests.log(source_vm.event_wait('BLOCK_JOB_READY'),
             filters=[iotests.filter_qmp_event])
 
@@ -66,8 +67,20 @@ dest_vm.qmp('migrate-set-capabilities',
             capabilities=[{'capability': 'events', 'state': True}])
 iotests.log(source_vm.qmp('migrate', uri='unix:{0}'.format(migration_sock_path)))
 
+iotests.log('Gracefully ending the `drive-mirror` job on source...')
+iotests.log(source_vm.qmp('block-job-cancel', device='mirror-job0'))
+
+iotests.log('Stopping the NBD server on destination...')
+iotests.log(dest_vm.qmp('nbd-server-stop'))
+
 while True:
-    event = source_vm.event_wait('MIGRATION')
-    iotests.log(event, filters=[iotests.filter_qmp_event])
-    if event['data']['status'] in ('completed', 'failed'):
+    event1 = source_vm.event_wait('MIGRATION')
+    iotests.log(event1, filters=[iotests.filter_qmp_event])
+    if event1['data']['status'] in ('completed', 'failed'):
+        break
+
+while True:
+    event2 = source_vm.event_wait('BLOCK_JOB_COMPLETED')
+    iotests.log(event2, filters=[iotests.filter_qmp_event])
+    if event2['event'] == 'BLOCK_JOB_COMPLETED':
         break
diff --git a/tests/qemu-iotests/194.out b/tests/qemu-iotests/194.out
index ae501fecacb706b1851cb9063ce9c9d5a28bb7ea..a461d7e21abb2dee54d72d93d644100f9fbcb17d 100644
--- a/tests/qemu-iotests/194.out
+++ b/tests/qemu-iotests/194.out
@@ -2,12 +2,17 @@ Launching VMs...
 Launching NBD server on destination...
 {u'return': {}}
 {u'return': {}}
-Starting drive-mirror on source...
+Starting `drive-mirror` on source...
 {u'return': {}}
-Waiting for drive-mirror to complete...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'drive0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'}
+Waiting for `drive-mirror` to complete...
+{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'}
 Starting migration...
 {u'return': {}}
+Gracefully ending the `drive-mirror` job on source...
+{u'return': {}}
+Stopping the NBD server on destination...
+{u'return': {}}
 {u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'}
 {u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'}
 {u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'}
+{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_COMPLETED'}
-- 
2.9.5

Re: [Qemu-devel] [Qemu-block] [PATCH v2] qemu-iotests: Extend non-shared storage migration test (194)

[Stefan Hajnoczi]

On Tue, Aug 29, 2017 at 11:06:22AM +0200, Kashyap Chamarthy wrote:
> This is the follow-up patch that was discussed[*] as part of feedback to
> qemu-iotest 194.
> 
> Changes in this patch:
> 
>   - Supply 'job-id' parameter to `drive-mirror` invocation.
> 
>   - Issue QMP `block-job-cancel` command on the source QEMU to
>     gracefully complete `drive-mirror` operation.
> 
>   - Stop the NBD server on the destination QEMU.
> 
>   - Finally, check for both the events: MIGRATION and
>     BLOCK_JOB_COMPLETED
> 
> With the above, the test will also be (almost) in sync with the
> procedure outlined in the document 'live-block-operations.rst'[+]
> (section: "QMP invocation for live storage migration with
> ``drive-mirror`` + NBD").
> 
> [*] https://lists.nongnu.org/archive/html/qemu-devel/2017-08/msg04820.html
>     -- qemu-iotests: add 194 non-shared storage migration test
> [+] https://git.qemu.org/gitweb.cgi?p=qemu.git;a=blob;f=docs/interop/live-block-operations.rst
> 
> Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
> ---
> Changes in v2:
>  - Check for both the events: MIGRATION and BLOCK_JOB_COMPLETED (Eric)
> ---
>  tests/qemu-iotests/194     | 25 +++++++++++++++++++------
>  tests/qemu-iotests/194.out | 11 ++++++++---
>  2 files changed, 27 insertions(+), 9 deletions(-)
> 
> diff --git a/tests/qemu-iotests/194 b/tests/qemu-iotests/194
> index 8028111e21bed5cf4a2e8e32dc04aa5a9ea9caca..9d81189fe3e790d060da5a0fd41836f4b57f95c7 100755
> --- a/tests/qemu-iotests/194
> +++ b/tests/qemu-iotests/194
> @@ -46,16 +46,17 @@ iotests.log('Launching NBD server on destination...')
>  iotests.log(dest_vm.qmp('nbd-server-start', addr={'type': 'unix', 'data': {'path': nbd_sock_path}}))
>  iotests.log(dest_vm.qmp('nbd-server-add', device='drive0', writable=True))
>  
> -iotests.log('Starting drive-mirror on source...')
> +iotests.log('Starting `drive-mirror` on source...')
>  iotests.log(source_vm.qmp(
>                'drive-mirror',
>                device='drive0',
>                target='nbd+unix:///drive0?socket={0}'.format(nbd_sock_path),
>                sync='full',
>                format='raw', # always raw, the server handles the format
> -              mode='existing'))
> +              mode='existing',
> +              job_id='mirror-job0'))
>  
> -iotests.log('Waiting for drive-mirror to complete...')
> +iotests.log('Waiting for `drive-mirror` to complete...')
>  iotests.log(source_vm.event_wait('BLOCK_JOB_READY'),
>              filters=[iotests.filter_qmp_event])
>  
> @@ -66,8 +67,20 @@ dest_vm.qmp('migrate-set-capabilities',
>              capabilities=[{'capability': 'events', 'state': True}])
>  iotests.log(source_vm.qmp('migrate', uri='unix:{0}'.format(migration_sock_path)))
>  
> +iotests.log('Gracefully ending the `drive-mirror` job on source...')
> +iotests.log(source_vm.qmp('block-job-cancel', device='mirror-job0'))

Two issues:

1. Migration may not have completed yet so drive-mirror cannot be
   stopped here.  We need to wait for migration to complete on the
   source before cancelling the block job.

2. block-job-cancel is asynchronous.  The block job may still be running
   after this returns.  Therefore it may still be using the NBD drive...

> +
> +iotests.log('Stopping the NBD server on destination...')
> +iotests.log(dest_vm.qmp('nbd-server-stop'))

...and that races with this command on the destination.

We need to wait for the BLOCK_JOB_CANCELLED/COMPLETED event on the
source QEMU before stopping the NBD server on the destination.

> +
>  while True:
> -    event = source_vm.event_wait('MIGRATION')
> -    iotests.log(event, filters=[iotests.filter_qmp_event])
> -    if event['data']['status'] in ('completed', 'failed'):
> +    event1 = source_vm.event_wait('MIGRATION')
> +    iotests.log(event1, filters=[iotests.filter_qmp_event])
> +    if event1['data']['status'] in ('completed', 'failed'):
> +        break
> +
> +while True:
> +    event2 = source_vm.event_wait('BLOCK_JOB_COMPLETED')
> +    iotests.log(event2, filters=[iotests.filter_qmp_event])
> +    if event2['event'] == 'BLOCK_JOB_COMPLETED':
>          break
> diff --git a/tests/qemu-iotests/194.out b/tests/qemu-iotests/194.out
> index ae501fecacb706b1851cb9063ce9c9d5a28bb7ea..a461d7e21abb2dee54d72d93d644100f9fbcb17d 100644
> --- a/tests/qemu-iotests/194.out
> +++ b/tests/qemu-iotests/194.out
> @@ -2,12 +2,17 @@ Launching VMs...
>  Launching NBD server on destination...
>  {u'return': {}}
>  {u'return': {}}
> -Starting drive-mirror on source...
> +Starting `drive-mirror` on source...
>  {u'return': {}}
> -Waiting for drive-mirror to complete...
> -{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'drive0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'}
> +Waiting for `drive-mirror` to complete...
> +{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'}
>  Starting migration...
>  {u'return': {}}
> +Gracefully ending the `drive-mirror` job on source...
> +{u'return': {}}
> +Stopping the NBD server on destination...
> +{u'return': {}}
>  {u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'}
>  {u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'}
>  {u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'}
> +{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_COMPLETED'}
> -- 
> 2.9.5
> 
> 

Re: [Qemu-devel] [Qemu-block] [PATCH v2] qemu-iotests: Extend non-shared storage migration test (194)

[Kashyap Chamarthy]

On Tue, Aug 29, 2017 at 03:19:54PM +0100, Stefan Hajnoczi wrote:
> On Tue, Aug 29, 2017 at 11:06:22AM +0200, Kashyap Chamarthy wrote:

[...]

> > +iotests.log('Gracefully ending the `drive-mirror` job on source...')
> > +iotests.log(source_vm.qmp('block-job-cancel', device='mirror-job0'))
> 
> Two issues:
> 
> 1. Migration may not have completed yet so drive-mirror cannot be
>    stopped here.  We need to wait for migration to complete on the
>    source before cancelling the block job.
> 
> 2. block-job-cancel is asynchronous.  The block job may still be running
>    after this returns.  Therefore it may still be using the NBD drive...
> 
> > +
> > +iotests.log('Stopping the NBD server on destination...')
> > +iotests.log(dest_vm.qmp('nbd-server-stop'))
> 
> ...and that races with this command on the destination.
> 
> We need to wait for the BLOCK_JOB_CANCELLED/COMPLETED event on the
> source QEMU before stopping the NBD server on the destination.

Yes, both good points.  How about the following `diff` to address both
the issues you raised?

#----------------------------------------------------------------------
diff --git a/tests/qemu-iotests/194 b/tests/qemu-iotests/194
index 9d81189..a3e3bad 100755
--- a/tests/qemu-iotests/194
+++ b/tests/qemu-iotests/194
@@ -67,20 +67,18 @@ dest_vm.qmp('migrate-set-capabilities',
             capabilities=[{'capability': 'events', 'state': True}])
 iotests.log(source_vm.qmp('migrate', uri='unix:{0}'.format(migration_sock_path)))
 
-iotests.log('Gracefully ending the `drive-mirror` job on source...')
-iotests.log(source_vm.qmp('block-job-cancel', device='mirror-job0'))
-
-iotests.log('Stopping the NBD server on destination...')
-iotests.log(dest_vm.qmp('nbd-server-stop'))
-
 while True:
     event1 = source_vm.event_wait('MIGRATION')
     iotests.log(event1, filters=[iotests.filter_qmp_event])
     if event1['data']['status'] in ('completed', 'failed'):
+        iotests.log('Gracefully ending the `drive-mirror` job on source...')
+        iotests.log(source_vm.qmp('block-job-cancel', device='mirror-job0'))
         break
 
 while True:
     event2 = source_vm.event_wait('BLOCK_JOB_COMPLETED')
     iotests.log(event2, filters=[iotests.filter_qmp_event])
     if event2['event'] == 'BLOCK_JOB_COMPLETED':
+        iotests.log('Stopping the NBD server on destination...')
+        iotests.log(dest_vm.qmp('nbd-server-stop'))
         break
#----------------------------------------------------------------------

And the output would be:

#----------------------------------------------------------------------
Launching VMs...
Launching NBD server on destination...
{u'return': {}}
{u'return': {}}
Starting `drive-mirror` on source...
{u'return': {}}
Waiting for `drive-mirror` to complete...
{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'}
Starting migration...
{u'return': {}}
{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'}
{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'}
{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'}
Gracefully ending the `drive-mirror` job on source...
{u'return': {}}
{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_COMPLETED'}
Stopping the NBD server on destination...
{u'return': {}}
#----------------------------------------------------------------------

-- 
/kashyap