[Qemu-devel] [PULL 00/71] Block layer patches

[Qemu-devel] [PULL 00/71] Block layer patches

[Kevin Wolf]

The following changes since commit 59a568b57848b10e8a44518a889323f12ccdd8f4:

  Merge remote-tracking branch 'remotes/kraxel/tags/vga-20190222-pull-request' into staging (2019-02-25 12:49:07 +0000)

are available in the Git repository at:

  git://repo.or.cz/qemu/kevin.git tags/for-upstream

for you to fetch changes up to 1b967e9f348d48788a2ab481d45398b80ce71fa6:

  Merge remote-tracking branch 'mreitz/tags/pull-block-2019-02-25' into queue-block (2019-02-25 15:16:57 +0100)

----------------------------------------------------------------
Block layer patches:

- Block graph change fixes (avoid loops, cope with non-tree graphs)
- bdrv_set_aio_context() related fixes
- HMP snapshot commands: Use only tag, not the ID to identify snapshots
- qmeu-img, commit: Error path fixes
- block/nvme: Build fix for gcc 9
- MAINTAINERS updates
- Fix various issues with bdrv_refresh_filename()
- Fix various iotests
- Include LUKS overhead in qemu-img measure for qcow2
- A fix for vmdk's image creation interface

----------------------------------------------------------------
Alberto Garcia (2):
      commit: Replace commit_top_bs on failure after deleting the block job
      qcow2: Assert that L2 table offsets fit in the L1 table

Daniel Henrique Barboza (3):
      block/snapshot.c: eliminate use of ID input in snapshot operations
      block/snapshot: remove bdrv_snapshot_delete_by_id_or_name
      qcow2-snapshot: remove redundant find_snapshot_by_id_and_name call

Daniel P. Berrangé (1):
      qemu-img: fix error reporting for -object

Denis Plotnikov (1):
      block: don't set the same context

Jeff Cody (2):
      MAINTAINERS: Replace myself with John Snow for block jobs
      MAINTAINERS: Remove myself as block maintainer

Kevin Wolf (14):
      block-backend: Make blk_inc/dec_in_flight public
      virtio-blk: Increase in_flight for request restart BH
      nbd: Restrict connection_co reentrance
      io: Make qio_channel_yield() interruptible
      io: Remove redundant read/write_coroutine assignments
      nbd: Move nbd_read_eof() to nbd/client.c
      nbd: Use low-level QIOChannel API in nbd_read_eof()
      nbd: Increase bs->in_flight during AioContext switch
      block: Don't poll in bdrv_set_aio_context()
      block: Fix AioContext switch for drained node
      test-bdrv-drain: AioContext switch in drained section
      block: Use normal drain for bdrv_set_aio_context()
      aio-posix: Assert that aio_poll() is always called in home thread
      Merge remote-tracking branch 'mreitz/tags/pull-block-2019-02-25' into queue-block

Max Reitz (42):
      block: Use bdrv_refresh_filename() to pull
      block: Use children list in bdrv_refresh_filename
      block: Skip implicit nodes for filename info
      block: Add BDS.auto_backing_file
      block: Respect backing bs in bdrv_refresh_filename
      iotests.py: Add filter_imgfmt()
      iotests.py: Add node_info()
      iotests: Add test for backing file overrides
      block: Make path_combine() return the path
      block: bdrv_get_full_backing_filename_from_...'s ret. val.
      block: bdrv_get_full_backing_filename's ret. val.
      block: Add bdrv_make_absolute_filename()
      block: Fix bdrv_find_backing_image()
      block: Add bdrv_dirname()
      blkverify: Make bdrv_dirname() return NULL
      quorum: Make bdrv_dirname() return NULL
      block/nbd: Make bdrv_dirname() return NULL
      block/nfs: Implement bdrv_dirname()
      block: Use bdrv_dirname() for relative filenames
      iotests: Add quorum case to test 110
      block: Add strong_runtime_opts to BlockDriver
      block: Add BlockDriver.bdrv_gather_child_options
      block: Generically refresh runtime options
      block: Purify .bdrv_refresh_filename()
      block: Do not copy exact_filename from format file
      block/nvme: Fix bdrv_refresh_filename()
      block/curl: Harmonize option defaults
      block/curl: Implement bdrv_refresh_filename()
      block/null: Generate filename even with latency-ns
      block: BDS options may lack the "driver" option
      iotests: Test json:{} filenames of internal BDSs
      iotests: Re-add filename filters
      iotests: Fix 237 for Python 2.x
      iotests: Remove superfluous rm from 232
      iotests: Fix 232 for LUKS
      iotests: Fix 207 to use QMP filters for qmp_log
      iotests.py: Add is_str()
      iotests.py: Filter filename in any string value
      iotests: Filter SSH paths
      iotests: Let 045 be run concurrently
      iotests.py: s/_/-/g on keys in qmp_log()
      iotests: Skip 211 on insufficient memory

Stefan Hajnoczi (2):
      qcow2: include LUKS payload overhead in qemu-img measure
      iotests: add LUKS payload overhead to 178 qemu-img measure test

Thomas Huth (1):
      block/nvme: Remove QEMU_PACKED from naturally aligned NVMeRegs struct

Vladimir Sementsov-Ogievskiy (3):
      block: improve should_update_child
      block: fix bdrv_check_perm for non-tree subgraph
      tests: add test-bdrv-graph-mod

yuchenlin (1):
      vmdk: false positive of compat6 with hwversion not set

 block/nbd-client.h               |   1 +
 include/block/block.h            |  16 +-
 include/block/block_int.h        |  53 +++-
 include/block/nbd.h              |   3 +-
 include/block/snapshot.h         |   3 -
 include/io/channel.h             |   9 +-
 include/sysemu/block-backend.h   |   2 +
 nbd/nbd-internal.h               |  19 --
 block.c                          | 597 +++++++++++++++++++++++++++------------
 block/blkdebug.c                 |  70 ++---
 block/blklogwrites.c             |  33 +--
 block/blkverify.c                |  29 +-
 block/block-backend.c            |   4 +-
 block/commit.c                   |   7 +-
 block/crypto.c                   |   8 +
 block/curl.c                     |  55 +++-
 block/gluster.c                  |  19 ++
 block/iscsi.c                    |  18 ++
 block/mirror.c                   |   3 +-
 block/nbd-client.c               |  36 ++-
 block/nbd.c                      |  46 +--
 block/nfs.c                      |  54 ++--
 block/null.c                     |  32 ++-
 block/nvme.c                     |  29 +-
 block/qapi.c                     |  16 +-
 block/qcow.c                     |  14 +-
 block/qcow2-cluster.c            |   3 +
 block/qcow2-snapshot.c           |   5 -
 block/qcow2.c                    |  89 +++++-
 block/qed.c                      |   7 +-
 block/quorum.c                   |  71 +++--
 block/raw-format.c               |  11 +-
 block/rbd.c                      |  14 +
 block/replication.c              |  10 +-
 block/sheepdog.c                 |  12 +
 block/snapshot.c                 |  25 +-
 block/ssh.c                      |  12 +
 block/throttle.c                 |   7 +
 block/vhdx-log.c                 |   1 +
 block/vmdk.c                     |  46 ++-
 block/vpc.c                      |   7 +
 block/vvfat.c                    |  12 +
 block/vxhs.c                     |  11 +
 blockdev.c                       |   8 +
 hw/block/virtio-blk.c            |   4 +
 io/channel.c                     |  22 +-
 nbd/client.c                     |  52 +++-
 qemu-img.c                       |  64 +++--
 tests/test-bdrv-drain.c          |  32 +++
 tests/test-bdrv-graph-mod.c      | 198 +++++++++++++
 util/aio-posix.c                 |   3 +-
 MAINTAINERS                      |  22 +-
 hmp-commands.hx                  |  32 ++-
 scripts/qemu.py                  |   5 +-
 tests/Makefile.include           |   2 +
 tests/qemu-iotests/045           |   2 +-
 tests/qemu-iotests/051.out       |   8 +-
 tests/qemu-iotests/051.pc.out    |   8 +-
 tests/qemu-iotests/110           |  29 +-
 tests/qemu-iotests/110.out       |   9 +-
 tests/qemu-iotests/178           |   8 +
 tests/qemu-iotests/178.out.qcow2 |  24 ++
 tests/qemu-iotests/206.out       |  56 ++--
 tests/qemu-iotests/207           |  10 +-
 tests/qemu-iotests/207.out       |  18 +-
 tests/qemu-iotests/210           |   5 +-
 tests/qemu-iotests/210.out       |  28 +-
 tests/qemu-iotests/211           |   9 +-
 tests/qemu-iotests/211.out       |  26 +-
 tests/qemu-iotests/212           |   5 +-
 tests/qemu-iotests/212.out       |  44 +--
 tests/qemu-iotests/213           |   5 +-
 tests/qemu-iotests/213.out       |  46 +--
 tests/qemu-iotests/224           | 139 +++++++++
 tests/qemu-iotests/224.out       |  18 ++
 tests/qemu-iotests/228           | 239 ++++++++++++++++
 tests/qemu-iotests/228.out       |  84 ++++++
 tests/qemu-iotests/232           |   5 +-
 tests/qemu-iotests/237           |   7 +-
 tests/qemu-iotests/237.out       |  54 ++--
 tests/qemu-iotests/common.rc     |   1 +
 tests/qemu-iotests/group         |   2 +
 tests/qemu-iotests/iotests.py    |  36 ++-
 83 files changed, 2191 insertions(+), 697 deletions(-)
 create mode 100644 tests/test-bdrv-graph-mod.c
 create mode 100755 tests/qemu-iotests/224
 create mode 100644 tests/qemu-iotests/224.out
 create mode 100755 tests/qemu-iotests/228
 create mode 100644 tests/qemu-iotests/228.out

[Qemu-devel] [PULL 01/71] MAINTAINERS: Replace myself with John Snow for block jobs

[Kevin Wolf]

From: Jeff Cody <jcody@redhat.com>

I'll not be involved with day-to-day qemu development, and John
Snow is a block jobs wizard.  Have him take over block job
maintainership duties.

Signed-off-by: Jeff Cody <jcody@redhat.com>
Acked-by: John Snow <jsnow@redhat.com>
Message-Id: <d56d7c6592e7d68aa72764e9616878394bffbc14.1537984851.git.jcody@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 MAINTAINERS | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 6ae55ff732..bb4d9829a4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1698,7 +1698,7 @@ F: include/scsi/*
 F: scsi/*
 
 Block Jobs
-M: Jeff Cody <jcody@redhat.com>
+M: John Snow <jsnow@redhat.com>
 L: qemu-block@nongnu.org
 S: Supported
 F: blockjob.c
@@ -1711,7 +1711,7 @@ F: block/commit.c
 F: block/stream.c
 F: block/mirror.c
 F: qapi/job.json
-T: git https://github.com/codyprime/qemu-kvm-jtc.git block
+T: git https://github.com/jnsnow/qemu.git jobs
 
 Block QAPI, monitor, command line
 M: Markus Armbruster <armbru@redhat.com>
-- 
2.20.1

[Qemu-devel] [PULL 02/71] MAINTAINERS: Remove myself as block maintainer

[Kevin Wolf]

From: Jeff Cody <jcody@redhat.com>

I'll not be involved in day-to-day qemu development.  Remove myself as
maintainer from the remainder of the network block drivers, and revert
them to the general block layer maintainership.

Move 'sheepdog' to the 'Odd Fixes' support level.

For VHDX, added my personal email address as a maintainer, as I can
answer questions or send the occassional bug fix.  Leaving it as
'Supported', instead of 'Odd Fixes', because I think the rest of the
block layer maintainers and developers will upkeep it as well, if
needed.

Signed-off-by: Jeff Cody <jcody@redhat.com>
Acked-by: Max Reitz <mreitz@redhat.com>
Message-Id: <63e205cb84c8f0a10c1bc6d5d6856d72ceb56e41.1537984851.git.jcody@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 MAINTAINERS | 18 +++---------------
 1 file changed, 3 insertions(+), 15 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index bb4d9829a4..e7f69f31ed 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2246,26 +2246,22 @@ F: block/vmdk.c
 
 RBD
 M: Josh Durgin <jdurgin@redhat.com>
-M: Jeff Cody <jcody@redhat.com>
 L: qemu-block@nongnu.org
 S: Supported
 F: block/rbd.c
-T: git https://github.com/codyprime/qemu-kvm-jtc.git block
 
 Sheepdog
 M: Liu Yuan <namei.unix@gmail.com>
-M: Jeff Cody <jcody@redhat.com>
 L: qemu-block@nongnu.org
-S: Supported
+L: sheepdog@lists.wpkg.org
+S: Odd Fixes
 F: block/sheepdog.c
-T: git https://github.com/codyprime/qemu-kvm-jtc.git block
 
 VHDX
-M: Jeff Cody <jcody@redhat.com>
+M: Jeff Cody <codyprime@gmail.com>
 L: qemu-block@nongnu.org
 S: Supported
 F: block/vhdx*
-T: git https://github.com/codyprime/qemu-kvm-jtc.git block
 
 VDI
 M: Stefan Weil <sw@weilnetz.de>
@@ -2295,34 +2291,26 @@ F: docs/interop/nbd.txt
 T: git https://repo.or.cz/qemu/ericb.git nbd
 
 NFS
-M: Jeff Cody <jcody@redhat.com>
 M: Peter Lieven <pl@kamp.de>
 L: qemu-block@nongnu.org
 S: Maintained
 F: block/nfs.c
-T: git https://github.com/codyprime/qemu-kvm-jtc.git block
 
 SSH
 M: Richard W.M. Jones <rjones@redhat.com>
-M: Jeff Cody <jcody@redhat.com>
 L: qemu-block@nongnu.org
 S: Supported
 F: block/ssh.c
-T: git https://github.com/codyprime/qemu-kvm-jtc.git block
 
 CURL
-M: Jeff Cody <jcody@redhat.com>
 L: qemu-block@nongnu.org
 S: Supported
 F: block/curl.c
-T: git https://github.com/codyprime/qemu-kvm-jtc.git block
 
 GLUSTER
-M: Jeff Cody <jcody@redhat.com>
 L: qemu-block@nongnu.org
 S: Supported
 F: block/gluster.c
-T: git https://github.com/codyprime/qemu-kvm-jtc.git block
 
 Null Block Driver
 M: Fam Zheng <fam@euphon.net>
-- 
2.20.1

[Qemu-devel] [PULL 03/71] block/snapshot.c: eliminate use of ID input in snapshot operations

[Kevin Wolf]

From: Daniel Henrique Barboza <danielhb413@gmail.com>

At this moment, QEMU attempts to create/load/delete snapshots
by using either an ID (id_str) or a name. The problem is that the code
isn't consistent of whether the entered argument is an ID or a name,
causing unexpected behaviors.

For example, when creating snapshots via savevm <arg>, what happens is that
"arg" is treated as both name and id_str. In a guest without snapshots, create
a single snapshot via savevm:

(qemu) savevm 0
(qemu) info snapshots
List of snapshots present on all disks:
ID        TAG                 VM SIZE                DATE       VM CLOCK
--        0                      741M 2018-07-31 13:39:56   00:41:25.313

A snapshot with name "0" is created. ID is hidden from the user, but the
ID is a non-zero integer that starts at "1". Thus, this snapshot has
id_str=1, TAG="0". Creating a second snapshot with arg = 1, the first one
is deleted:

(qemu) savevm 1
(qemu) info snapshots
List of snapshots present on all disks:
ID        TAG                 VM SIZE                DATE       VM CLOCK
--        1                      741M 2018-07-31 13:42:14   00:41:55.252

What happened?

- when creating the second snapshot, a verification is done inside
bdrv_all_delete_snapshot to delete any existing snapshots that matches an
string argument. Here, the code calls bdrv_all_delete_snapshot("1", ...);

- bdrv_all_delete_snapshot calls bdrv_snapshot_find(..., "1") for each
BlockDriverState of the guest. And this is where things goes tilting:
bdrv_snapshot_find does a search by both id_str and name. It finds
out that there is a snapshot that has id_str = 1, stores a reference
to the snapshot in the sn_info pointer and then returns match found;

- since a match was found, a call to bdrv_snapshot_delete_by_id_or_name() is
made. This function ignores the pointer written by bdrv_snapshot_find. Instead,
it deletes the snapshot using bdrv_snapshot_delete() calling it first with
id_str = 1. If it fails to delete, then it calls it again with name = 1.

- after all that, QEMU creates the new snapshot, that has id_str = 1 and
name = 1. The user is left wondering that happened with the first snapshot
created. Similar bugs can be triggered when using loadvm and delvm.

Before contemplating discarding the use of ID input in these operations,
I've searched the code of what would be the implications. My findings
are:

- the RBD and Sheepdog drivers don't care. Both uses the 'name' field as
key in their logic, making id_str = name when appropriate.
replay-snapshot.c does not make any special use of id_str;

- qcow2 uses id_str as an unique identifier but it is automatically
calculated, not being influenced by user input. Other than that, there are
no distinguish operations made only with id_str;

- in blockdev.c, the delete operation uses a match of both id_str AND
name. Given that id_str is either a copy of 'name' or auto-generated,
we're fine here.

This gives motivation to not consider ID as a valid user input in HMP
commands - sticking with 'name' input only is more consistent. To
accomplish that, the following changes were made in this patch:

- bdrv_snapshot_find() does not match for id_str anymore, only 'name'. The
function is called in save_snapshot(), load_snapshot(), bdrv_all_delete_snapshot()
and bdrv_all_find_snapshot(). This change makes the search function more
predictable and does not change the behavior of any underlying code that uses
these affected functions, which are related to HMP (which is fine) and the
main loop inside vl.c (which doesn't care about it anyways);

- bdrv_all_delete_snapshot() does not call bdrv_snapshot_delete_by_id_or_name
anymore. Instead, it uses the pointer returned by bdrv_snapshot_find to
erase the snapshot with the exact match of id_str an name. This function
is called in save_snapshot and hmp_delvm, thus this change  produces the
intended effect;

- documentation changes to reflect the new behavior. I consider this to
be an API fix instead of an API change - the user was already creating
snapshots using 'name', but now he/she will also enjoy a consistent
behavior.

Ideally we would get rid of the id_str field entirely, but this would have
repercussions on existing snapshots. Another day perhaps.

Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Acked-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/snapshot.c |  5 +++--
 hmp-commands.hx  | 32 ++++++++++++++++++++------------
 2 files changed, 23 insertions(+), 14 deletions(-)

diff --git a/block/snapshot.c b/block/snapshot.c
index 3218a542df..e371d2243d 100644
--- a/block/snapshot.c
+++ b/block/snapshot.c
@@ -63,7 +63,7 @@ int bdrv_snapshot_find(BlockDriverState *bs, QEMUSnapshotInfo *sn_info,
     }
     for (i = 0; i < nb_sns; i++) {
         sn = &sn_tab[i];
-        if (!strcmp(sn->id_str, name) || !strcmp(sn->name, name)) {
+        if (!strcmp(sn->name, name)) {
             *sn_info = *sn;
             ret = 0;
             break;
@@ -448,7 +448,8 @@ int bdrv_all_delete_snapshot(const char *name, BlockDriverState **first_bad_bs,
         aio_context_acquire(ctx);
         if (bdrv_can_snapshot(bs) &&
                 bdrv_snapshot_find(bs, snapshot, name) >= 0) {
-            ret = bdrv_snapshot_delete_by_id_or_name(bs, name, err);
+            ret = bdrv_snapshot_delete(bs, snapshot->id_str,
+                                       snapshot->name, err);
         }
         aio_context_release(ctx);
         if (ret < 0) {
diff --git a/hmp-commands.hx b/hmp-commands.hx
index ba71558c25..e5fbc2ca59 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -350,49 +350,57 @@ ETEXI
     {
         .name       = "savevm",
         .args_type  = "name:s?",
-        .params     = "[tag|id]",
-        .help       = "save a VM snapshot. If no tag or id are provided, a new snapshot is created",
+        .params     = "tag",
+        .help       = "save a VM snapshot. If no tag is provided, a new snapshot is created",
         .cmd        = hmp_savevm,
     },
 
 STEXI
-@item savevm [@var{tag}|@var{id}]
+@item savevm @var{tag}
 @findex savevm
 Create a snapshot of the whole virtual machine. If @var{tag} is
 provided, it is used as human readable identifier. If there is already
-a snapshot with the same tag or ID, it is replaced. More info at
+a snapshot with the same tag, it is replaced. More info at
 @ref{vm_snapshots}.
+
+Since 4.0, savevm stopped allowing the snapshot id to be set, accepting
+only @var{tag} as parameter.
 ETEXI
 
     {
         .name       = "loadvm",
         .args_type  = "name:s",
-        .params     = "tag|id",
-        .help       = "restore a VM snapshot from its tag or id",
+        .params     = "tag",
+        .help       = "restore a VM snapshot from its tag",
         .cmd        = hmp_loadvm,
         .command_completion = loadvm_completion,
     },
 
 STEXI
-@item loadvm @var{tag}|@var{id}
+@item loadvm @var{tag}
 @findex loadvm
 Set the whole virtual machine to the snapshot identified by the tag
-@var{tag} or the unique snapshot ID @var{id}.
+@var{tag}.
+
+Since 4.0, loadvm stopped accepting snapshot id as parameter.
 ETEXI
 
     {
         .name       = "delvm",
         .args_type  = "name:s",
-        .params     = "tag|id",
-        .help       = "delete a VM snapshot from its tag or id",
+        .params     = "tag",
+        .help       = "delete a VM snapshot from its tag",
         .cmd        = hmp_delvm,
         .command_completion = delvm_completion,
     },
 
 STEXI
-@item delvm @var{tag}|@var{id}
+@item delvm @var{tag}
 @findex delvm
-Delete the snapshot identified by @var{tag} or @var{id}.
+Delete the snapshot identified by @var{tag}.
+
+Since 4.0, delvm stopped deleting snapshots by snapshot id, accepting
+only @var{tag} as parameter.
 ETEXI
 
     {
-- 
2.20.1

[Qemu-devel] [PULL 04/71] block/snapshot: remove bdrv_snapshot_delete_by_id_or_name

[Kevin Wolf]

From: Daniel Henrique Barboza <danielhb413@gmail.com>

After the previous patch, the only instance of this function left
is inside qemu-img.c.

qemu-img is using it inside the 'img_snapshot' function to delete
snapshots in the SNAPSHOT_DELETE case, based on a "snapshot_name"
string that refers to the tag, not ID, of the QEMUSnapshotInfo struct.
This can be verified by checking the SNAPSHOT_CREATE case that
comes shortly before SNAPSHOT_DELETE. In that case, the same
"snapshot_name" variable is being strcpy to the 'name' field
of the QEMUSnapshotInfo struct sn:

pstrcpy(sn.name, sizeof(sn.name), snapshot_name);

Based on that, it is unlikely that "snapshot_name" might contain
an "id" in SNAPSHOT_DELETE.

This patch changes SNAPSHOT_DELETE to use snapshot_find() and
snapshot_delete() instead of bdrv_snapshot_delete_by_id_or_name.
After that, there is no instances left of bdrv_snapshot_delete_by_id_or_name
in the code, so it is safe to remove it entirely.

Suggested-by: Murilo Opsfelder Araujo <muriloo@linux.ibm.com>
Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 include/block/snapshot.h |  3 ---
 block/snapshot.c         | 20 --------------------
 qemu-img.c               | 15 +++++++++++----
 3 files changed, 11 insertions(+), 27 deletions(-)

diff --git a/include/block/snapshot.h b/include/block/snapshot.h
index f73d1094af..b5d5084a12 100644
--- a/include/block/snapshot.h
+++ b/include/block/snapshot.h
@@ -61,9 +61,6 @@ int bdrv_snapshot_delete(BlockDriverState *bs,
                          const char *snapshot_id,
                          const char *name,
                          Error **errp);
-int bdrv_snapshot_delete_by_id_or_name(BlockDriverState *bs,
-                                       const char *id_or_name,
-                                       Error **errp);
 int bdrv_snapshot_list(BlockDriverState *bs,
                        QEMUSnapshotInfo **psn_info);
 int bdrv_snapshot_load_tmp(BlockDriverState *bs,
diff --git a/block/snapshot.c b/block/snapshot.c
index e371d2243d..f2f48f926a 100644
--- a/block/snapshot.c
+++ b/block/snapshot.c
@@ -301,26 +301,6 @@ int bdrv_snapshot_delete(BlockDriverState *bs,
     return ret;
 }
 
-int bdrv_snapshot_delete_by_id_or_name(BlockDriverState *bs,
-                                       const char *id_or_name,
-                                       Error **errp)
-{
-    int ret;
-    Error *local_err = NULL;
-
-    ret = bdrv_snapshot_delete(bs, id_or_name, NULL, &local_err);
-    if (ret == -ENOENT || ret == -EINVAL) {
-        error_free(local_err);
-        local_err = NULL;
-        ret = bdrv_snapshot_delete(bs, NULL, id_or_name, &local_err);
-    }
-
-    if (ret < 0) {
-        error_propagate(errp, local_err);
-    }
-    return ret;
-}
-
 int bdrv_snapshot_list(BlockDriverState *bs,
                        QEMUSnapshotInfo **psn_info)
 {
diff --git a/qemu-img.c b/qemu-img.c
index 25288c4d18..9737b815e4 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -3123,11 +3123,18 @@ static int img_snapshot(int argc, char **argv)
         break;
 
     case SNAPSHOT_DELETE:
-        bdrv_snapshot_delete_by_id_or_name(bs, snapshot_name, &err);
-        if (err) {
-            error_reportf_err(err, "Could not delete snapshot '%s': ",
-                              snapshot_name);
+        ret = bdrv_snapshot_find(bs, &sn, snapshot_name);
+        if (ret < 0) {
+            error_report("Could not delete snapshot '%s': snapshot not "
+                         "found", snapshot_name);
             ret = 1;
+        } else {
+            ret = bdrv_snapshot_delete(bs, sn.id_str, sn.name, &err);
+            if (ret < 0) {
+                error_reportf_err(err, "Could not delete snapshot '%s': ",
+                                  snapshot_name);
+                ret = 1;
+            }
         }
         break;
     }
-- 
2.20.1

[Qemu-devel] [PULL 05/71] qcow2-snapshot: remove redundant find_snapshot_by_id_and_name call

[Kevin Wolf]

From: Daniel Henrique Barboza <danielhb413@gmail.com>

In qcow2_snapshot_create there is the following code block:

    /* Generate an ID */
    find_new_snapshot_id(bs, sn_info->id_str, sizeof(sn_info->id_str));

    /* Check that the ID is unique */
    if (find_snapshot_by_id_and_name(bs, sn_info->id_str, NULL) >= 0) {
        return -EEXIST;
    }

find_new_snapshot_id cycles through all snapshots, getting the id_str
as an unsigned long int, calculating the max id_max value of all the
existing id_strs and writing in the id_str pointer id_max + 1:

    for(i = 0; i < s->nb_snapshots; i++) {
        sn = s->snapshots + i;
        id = strtoul(sn->id_str, NULL, 10);
        if (id > id_max)
            id_max = id;
    }
    snprintf(id_str, id_str_size, "%lu", id_max + 1);

Here, sn_info->id_str will have the unique value id_max + 1. Right
after that, find_snapshot_by_id_and_name is called with
id = sn_info->id_str and name = NULL. This will cause the function
to execute the following:

    } else if (id) {
        for (i = 0; i < s->nb_snapshots; i++) {
            if (!strcmp(s->snapshots[i].id_str, id)) {
                return i;
            }
        }
    }

In short, we're searching the existing snapshots to see if sn_info->id_str
matches any existing id, right after we set in the previous line a
sn_info->id_str value that is already unique.

The first code block goes way back to commit 585f8587ad, a 2006 commit from
Fabrice Bellard that simply says "new qcow2 disk image format". No more
info is provided about this logic in any subsequent commits that moved
this code block around.

I can't say about the original design, but the current logic is redundant.
bdrv_snapshot_create is called in aio_context lock, forbidding any
concurrent call to accidentally create a new snapshot between
the find_new_snapshot_id and find_snapshot_by_id_and_name calls. What
we're ending up doing is to cycle through the snapshots two times
for no viable reason.

This patch eliminates the redundancy by removing the 'id is unique'
check that calls find_snapshot_by_id_and_name.

Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/qcow2-snapshot.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index bb6a5b7516..20e8472191 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -358,11 +358,6 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
     /* Generate an ID */
     find_new_snapshot_id(bs, sn_info->id_str, sizeof(sn_info->id_str));
 
-    /* Check that the ID is unique */
-    if (find_snapshot_by_id_and_name(bs, sn_info->id_str, NULL) >= 0) {
-        return -EEXIST;
-    }
-
     /* Populate sn with passed data */
     sn->id_str = g_strdup(sn_info->id_str);
     sn->name = g_strdup(sn_info->name);
-- 
2.20.1

[Qemu-devel] [PULL 06/71] block: don't set the same context

[Kevin Wolf]

From: Denis Plotnikov <dplotnikov@virtuozzo.com>

Adds a fast path on aio context setting preventing
unnecessary context setting routine.
Also, it prevents issues with cyclic walk of child
bds-es appeared because of registering aio walking
notifiers:

Call stack:

0  __GI_raise
1  __GI_abort
2  __assert_fail_base
3  __GI___assert_fail
4  bdrv_detach_aio_context (bs=0x55f54d65c000)      <<<
5  bdrv_detach_aio_context (bs=0x55f54fc8a800)
6  bdrv_set_aio_context (bs=0x55f54fc8a800, ...)
7  block_job_attached_aio_context
8  bdrv_attach_aio_context (bs=0x55f54d65c000, ...) <<<
9  bdrv_set_aio_context (bs=0x55f54d65c000)
10 blk_set_aio_context
11 virtio_blk_data_plane_stop
12 virtio_bus_stop_ioeventfd
13 virtio_vmstate_change
14 vm_state_notify (running=0, state=RUN_STATE_SHUTDOWN)
15 do_vm_stop (state=RUN_STATE_SHUTDOWN, send_stop=true)
16 vm_stop (state=RUN_STATE_SHUTDOWN)
17 main_loop_should_exit
18 main_loop
19 main

This can happen because of "new" context attachment to VM disk bds.
When attaching a new context the corresponding aio context handler is
called for each of aio_notifiers registered on the VM disk bds context.
Among those handlers, there is the block_job_attached_aio_context handler
which sets a new aio context for the block job bds. When doing so,
the old context is detached from all the block job bds children and one of
them is the VM disk bds, serving as backing store for the blockjob bds,
although the VM disk bds is actually the initializer of that process.
Since the VM disk bds is protected with walking_aio_notifiers flag
from double processing in recursive calls, the assert fires.

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/block.c b/block.c
index 4ad0e90d7e..0c12632661 100644
--- a/block.c
+++ b/block.c
@@ -5265,6 +5265,10 @@ void bdrv_set_aio_context(BlockDriverState *bs, AioContext *new_context)
 {
     AioContext *ctx = bdrv_get_aio_context(bs);
 
+    if (ctx == new_context) {
+        return;
+    }
+
     aio_disable_external(ctx);
     bdrv_parent_drained_begin(bs, NULL, false);
     bdrv_drain(bs); /* ensure there are no in-flight requests */
-- 
2.20.1

[Qemu-devel] [PULL 07/71] commit: Replace commit_top_bs on failure after deleting the block job

[Kevin Wolf]

From: Alberto Garcia <berto@igalia.com>

If there's an error in commit_start() then the block job must be
deleted before replacing commit_top_bs, otherwise it will fail because
of lack of permissions. This happens since the permission system was
introduced in 8dfba2797761d8a43744e4e6571c8175e448a478.

Fortunately this bug doesn't seem to be possible to reproduce at the
moment without changing the code.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/commit.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/block/commit.c b/block/commit.c
index 53148e610b..5deb05925b 100644
--- a/block/commit.c
+++ b/block/commit.c
@@ -374,10 +374,12 @@ fail:
     if (s->top) {
         blk_unref(s->top);
     }
+    job_early_fail(&s->common.job);
+    /* commit_top_bs has to be replaced after deleting the block job,
+     * otherwise this would fail because of lack of permissions. */
     if (commit_top_bs) {
         bdrv_replace_node(commit_top_bs, top, &error_abort);
     }
-    job_early_fail(&s->common.job);
 }
 
 
-- 
2.20.1

[Qemu-devel] [PULL 08/71] qemu-img: fix error reporting for -object

[Kevin Wolf]

From: Daniel P. Berrangé <berrange@redhat.com>

Error reporting for user_creatable_add_opts_foreach was changed so that
it no longer called 'error_report_err' in:

  commit 7e1e0c11127bde81cff260fc6859690435c509d6
  Author: Markus Armbruster <armbru@redhat.com>
  Date:   Wed Oct 17 10:26:43 2018 +0200

    qom: Clean up error reporting in user_creatable_add_opts_foreach()

Some callers were updated to pass in "&error_fatal" but all the ones in
qemu-img were left passing NULL. As a result all errors went to
/dev/null instead of being reported to the user.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-img.c | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index 9737b815e4..eb5045c742 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -503,7 +503,7 @@ static int img_create(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         goto fail;
     }
 
@@ -753,7 +753,7 @@ static int img_check(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         return 1;
     }
 
@@ -966,7 +966,7 @@ static int img_commit(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         return 1;
     }
 
@@ -1349,7 +1349,7 @@ static int img_compare(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         ret = 2;
         goto out4;
     }
@@ -2159,7 +2159,7 @@ static int img_convert(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         goto fail_getopt;
     }
 
@@ -2713,7 +2713,7 @@ static int img_info(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         return 1;
     }
 
@@ -2932,7 +2932,7 @@ static int img_map(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         return 1;
     }
 
@@ -3081,7 +3081,7 @@ static int img_snapshot(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         return 1;
     }
 
@@ -3248,7 +3248,7 @@ static int img_rebase(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         return 1;
     }
 
@@ -3628,7 +3628,7 @@ static int img_resize(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         return 1;
     }
 
@@ -3872,7 +3872,7 @@ static int img_amend(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         ret = -1;
         goto out_no_progress;
     }
@@ -4516,7 +4516,7 @@ static int img_dd(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         ret = -1;
         goto out;
     }
@@ -4793,7 +4793,7 @@ static int img_measure(int argc, char **argv)
 
     if (qemu_opts_foreach(&qemu_object_opts,
                           user_creatable_add_opts_foreach,
-                          NULL, NULL)) {
+                          NULL, &error_fatal)) {
         goto out;
     }
 
-- 
2.20.1

[Qemu-devel] [PULL 09/71] block-backend: Make blk_inc/dec_in_flight public

[Kevin Wolf]

For some users of BlockBackends, just increasing the in_flight counter
is easier than implementing separate handlers in BlockDevOps. Make the
helper functions for this public.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 include/sysemu/block-backend.h | 2 ++
 block/block-backend.c          | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index 832a4bf168..e2066eb06b 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -156,6 +156,8 @@ int blk_co_pdiscard(BlockBackend *blk, int64_t offset, int bytes);
 int blk_co_flush(BlockBackend *blk);
 int blk_flush(BlockBackend *blk);
 int blk_commit_all(void);
+void blk_inc_in_flight(BlockBackend *blk);
+void blk_dec_in_flight(BlockBackend *blk);
 void blk_drain(BlockBackend *blk);
 void blk_drain_all(void);
 void blk_set_on_error(BlockBackend *blk, BlockdevOnError on_read_error,
diff --git a/block/block-backend.c b/block/block-backend.c
index f6ea824308..0219555f89 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -1262,12 +1262,12 @@ int blk_make_zero(BlockBackend *blk, BdrvRequestFlags flags)
     return bdrv_make_zero(blk->root, flags);
 }
 
-static void blk_inc_in_flight(BlockBackend *blk)
+void blk_inc_in_flight(BlockBackend *blk)
 {
     atomic_inc(&blk->in_flight);
 }
 
-static void blk_dec_in_flight(BlockBackend *blk)
+void blk_dec_in_flight(BlockBackend *blk)
 {
     atomic_dec(&blk->in_flight);
     aio_wait_kick();
-- 
2.20.1

[Qemu-devel] [PULL 10/71] virtio-blk: Increase in_flight for request restart BH

[Kevin Wolf]

virtio_blk_dma_restart_bh() submits new requests, so in order to make
sure that these requests are not started inside a drained section of the
attached BlockBackend, we need to make sure that draining the
BlockBackend waits for the BH to be executed.

This BH is still questionable because its scheduled in the main thread
instead of the configured iothread. Leave a FIXME comment for this.

But with this fix, enabling the data plane at least waits for these
requests (in bdrv_set_aio_context()) instead of changing the AioContext
under their feet and making them run in the wrong thread, causing
crashes and failures (e.g. due to missing locking).

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 hw/block/virtio-blk.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index cf7f47eaba..e11e6e45d0 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -675,6 +675,7 @@ static void virtio_blk_dma_restart_bh(void *opaque)
     if (mrb.num_reqs) {
         virtio_blk_submit_multireq(s->blk, &mrb);
     }
+    blk_dec_in_flight(s->conf.conf.blk);
     aio_context_release(blk_get_aio_context(s->conf.conf.blk));
 }
 
@@ -688,8 +689,11 @@ static void virtio_blk_dma_restart_cb(void *opaque, int running,
     }
 
     if (!s->bh) {
+        /* FIXME The data plane is not started yet, so these requests are
+         * processed in the main thread. */
         s->bh = aio_bh_new(blk_get_aio_context(s->conf.conf.blk),
                            virtio_blk_dma_restart_bh, s);
+        blk_inc_in_flight(s->conf.conf.blk);
         qemu_bh_schedule(s->bh);
     }
 }
-- 
2.20.1

[Qemu-devel] [PULL 11/71] nbd: Restrict connection_co reentrance

[Kevin Wolf]

nbd_client_attach_aio_context() schedules connection_co in the new
AioContext and this way reenters it in any arbitrary place that has
yielded. We can restrict this a bit to the function call where the
coroutine actually sits waiting when it's idle.

This doesn't solve any bug yet, but it shows where in the code we need
to support this random reentrance and where we don't have to care.

Add FIXME comments for the existing bugs that the rest of this series
will fix.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 block/nbd-client.h |  1 +
 block/nbd-client.c | 22 ++++++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/block/nbd-client.h b/block/nbd-client.h
index d990207a5c..09e03013d2 100644
--- a/block/nbd-client.h
+++ b/block/nbd-client.h
@@ -35,6 +35,7 @@ typedef struct NBDClientSession {
 
     NBDClientRequest requests[MAX_NBD_REQUESTS];
     NBDReply reply;
+    BlockDriverState *bs;
     bool quit;
 } NBDClientSession;
 
diff --git a/block/nbd-client.c b/block/nbd-client.c
index f0ad54ce21..5ce4aa9520 100644
--- a/block/nbd-client.c
+++ b/block/nbd-client.c
@@ -76,8 +76,24 @@ static coroutine_fn void nbd_connection_entry(void *opaque)
     Error *local_err = NULL;
 
     while (!s->quit) {
+        /*
+         * The NBD client can only really be considered idle when it has
+         * yielded from qio_channel_readv_all_eof(), waiting for data. This is
+         * the point where the additional scheduled coroutine entry happens
+         * after nbd_client_attach_aio_context().
+         *
+         * Therefore we keep an additional in_flight reference all the time and
+         * only drop it temporarily here.
+         *
+         * FIXME This is not safe because the QIOChannel could wake up the
+         * coroutine for a second time; it is not prepared for coroutine
+         * resumption from external code.
+         */
+        bdrv_dec_in_flight(s->bs);
         assert(s->reply.handle == 0);
         ret = nbd_receive_reply(s->ioc, &s->reply, &local_err);
+        bdrv_inc_in_flight(s->bs);
+
         if (local_err) {
             trace_nbd_read_reply_entry_fail(ret, error_get_pretty(local_err));
             error_free(local_err);
@@ -116,6 +132,8 @@ static coroutine_fn void nbd_connection_entry(void *opaque)
 
     s->quit = true;
     nbd_recv_coroutines_wake_all(s);
+    bdrv_dec_in_flight(s->bs);
+
     s->connection_co = NULL;
     aio_wait_kick();
 }
@@ -970,6 +988,8 @@ void nbd_client_attach_aio_context(BlockDriverState *bs,
 {
     NBDClientSession *client = nbd_get_client_session(bs);
     qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
+
+    /* FIXME Really need a bdrv_inc_in_flight() here */
     aio_co_schedule(new_context, client->connection_co);
 }
 
@@ -1076,6 +1096,7 @@ static int nbd_client_connect(BlockDriverState *bs,
      * kick the reply mechanism.  */
     qio_channel_set_blocking(QIO_CHANNEL(sioc), false, NULL);
     client->connection_co = qemu_coroutine_create(nbd_connection_entry, client);
+    bdrv_inc_in_flight(bs);
     nbd_client_attach_aio_context(bs, bdrv_get_aio_context(bs));
 
     logout("Established connection with NBD server\n");
@@ -1108,6 +1129,7 @@ int nbd_client_init(BlockDriverState *bs,
 {
     NBDClientSession *client = nbd_get_client_session(bs);
 
+    client->bs = bs;
     qemu_co_mutex_init(&client->send_mutex);
     qemu_co_queue_init(&client->free_sema);
 
-- 
2.20.1

[Qemu-devel] [PULL 12/71] io: Make qio_channel_yield() interruptible

[Kevin Wolf]

Similar to how qemu_co_sleep_ns() allows preemption from an external
coroutine entry, allow reentering qio_channel_yield() early.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 include/io/channel.h |  9 ++++++---
 io/channel.c         | 10 ++++++++++
 2 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/include/io/channel.h b/include/io/channel.h
index da2f138200..59460cb1ec 100644
--- a/include/io/channel.h
+++ b/include/io/channel.h
@@ -739,10 +739,13 @@ void qio_channel_detach_aio_context(QIOChannel *ioc);
  * addition, no two coroutine can be waiting on the same condition
  * and channel at the same time.
  *
- * This must only be called from coroutine context
+ * This must only be called from coroutine context. It is safe to
+ * reenter the coroutine externally while it is waiting; in this
+ * case the function will return even if @condition is not yet
+ * available.
  */
-void qio_channel_yield(QIOChannel *ioc,
-                       GIOCondition condition);
+void coroutine_fn qio_channel_yield(QIOChannel *ioc,
+                                    GIOCondition condition);
 
 /**
  * qio_channel_wait:
diff --git a/io/channel.c b/io/channel.c
index 8dd0684f5d..303376e08d 100644
--- a/io/channel.c
+++ b/io/channel.c
@@ -469,6 +469,16 @@ void coroutine_fn qio_channel_yield(QIOChannel *ioc,
     }
     qio_channel_set_aio_fd_handlers(ioc);
     qemu_coroutine_yield();
+
+    /* Allow interrupting the operation by reentering the coroutine other than
+     * through the aio_fd_handlers. */
+    if (condition == G_IO_IN && ioc->read_coroutine) {
+        ioc->read_coroutine = NULL;
+        qio_channel_set_aio_fd_handlers(ioc);
+    } else if (condition == G_IO_OUT && ioc->write_coroutine) {
+        ioc->write_coroutine = NULL;
+        qio_channel_set_aio_fd_handlers(ioc);
+    }
 }
 
 
-- 
2.20.1

[Qemu-devel] [PULL 13/71] io: Remove redundant read/write_coroutine assignments

[Kevin Wolf]

qio_channel_yield() now updates ioc->read_write/coroutine and calls
qio_channel_set_aio_fd_handlers(), so the code in the handlers has
become redundant and can be removed.

This does not make a difference in intermediate states because
aio_co_wake() really enters the coroutine immediately here: These
handlers are never run in coroutine context, and we're in the right
AioContext because qio_channel_attach_aio_context() asserts that the
handlers are inactive.

To make these conditions more obvious, assert the right AioContext.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 io/channel.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/io/channel.c b/io/channel.c
index 303376e08d..2a26c2a2c0 100644
--- a/io/channel.c
+++ b/io/channel.c
@@ -400,15 +400,14 @@ off_t qio_channel_io_seek(QIOChannel *ioc,
 }
 
 
-static void qio_channel_set_aio_fd_handlers(QIOChannel *ioc);
-
 static void qio_channel_restart_read(void *opaque)
 {
     QIOChannel *ioc = opaque;
     Coroutine *co = ioc->read_coroutine;
 
-    ioc->read_coroutine = NULL;
-    qio_channel_set_aio_fd_handlers(ioc);
+    /* Assert that aio_co_wake() reenters the coroutine directly */
+    assert(qemu_get_current_aio_context() ==
+           qemu_coroutine_get_aio_context(co));
     aio_co_wake(co);
 }
 
@@ -417,8 +416,9 @@ static void qio_channel_restart_write(void *opaque)
     QIOChannel *ioc = opaque;
     Coroutine *co = ioc->write_coroutine;
 
-    ioc->write_coroutine = NULL;
-    qio_channel_set_aio_fd_handlers(ioc);
+    /* Assert that aio_co_wake() reenters the coroutine directly */
+    assert(qemu_get_current_aio_context() ==
+           qemu_coroutine_get_aio_context(co));
     aio_co_wake(co);
 }
 
-- 
2.20.1

[Qemu-devel] [PULL 14/71] nbd: Move nbd_read_eof() to nbd/client.c

[Kevin Wolf]

The only caller of nbd_read_eof() is nbd_receive_reply(), so it doesn't
have to live in the header file, but can move next to its caller.

Also add the missing coroutine_fn to the function and its caller.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 include/block/nbd.h |  3 ++-
 nbd/nbd-internal.h  | 19 -------------------
 nbd/client.c        | 22 +++++++++++++++++++++-
 3 files changed, 23 insertions(+), 21 deletions(-)

diff --git a/include/block/nbd.h b/include/block/nbd.h
index 96cfb1d7d5..cad975e00c 100644
--- a/include/block/nbd.h
+++ b/include/block/nbd.h
@@ -300,7 +300,8 @@ int nbd_receive_export_list(QIOChannel *ioc, QCryptoTLSCreds *tlscreds,
 int nbd_init(int fd, QIOChannelSocket *sioc, NBDExportInfo *info,
              Error **errp);
 int nbd_send_request(QIOChannel *ioc, NBDRequest *request);
-int nbd_receive_reply(QIOChannel *ioc, NBDReply *reply, Error **errp);
+int coroutine_fn nbd_receive_reply(QIOChannel *ioc, NBDReply *reply,
+                                   Error **errp);
 int nbd_client(int fd);
 int nbd_disconnect(int fd);
 int nbd_errno_to_system_errno(int err);
diff --git a/nbd/nbd-internal.h b/nbd/nbd-internal.h
index 82aa221227..049f83df77 100644
--- a/nbd/nbd-internal.h
+++ b/nbd/nbd-internal.h
@@ -64,25 +64,6 @@
 #define NBD_SET_TIMEOUT             _IO(0xab, 9)
 #define NBD_SET_FLAGS               _IO(0xab, 10)
 
-/* nbd_read_eof
- * Tries to read @size bytes from @ioc.
- * Returns 1 on success
- *         0 on eof, when no data was read (errp is not set)
- *         negative errno on failure (errp is set)
- */
-static inline int nbd_read_eof(QIOChannel *ioc, void *buffer, size_t size,
-                               Error **errp)
-{
-    int ret;
-
-    assert(size);
-    ret = qio_channel_read_all_eof(ioc, buffer, size, errp);
-    if (ret < 0) {
-        ret = -EIO;
-    }
-    return ret;
-}
-
 /* nbd_write
  * Writes @size bytes to @ioc. Returns 0 on success.
  */
diff --git a/nbd/client.c b/nbd/client.c
index 10a52ad7d0..28d174c0f3 100644
--- a/nbd/client.c
+++ b/nbd/client.c
@@ -1387,12 +1387,32 @@ static int nbd_receive_structured_reply_chunk(QIOChannel *ioc,
     return 0;
 }
 
+/* nbd_read_eof
+ * Tries to read @size bytes from @ioc.
+ * Returns 1 on success
+ *         0 on eof, when no data was read (errp is not set)
+ *         negative errno on failure (errp is set)
+ */
+static inline int coroutine_fn
+nbd_read_eof(QIOChannel *ioc, void *buffer, size_t size, Error **errp)
+{
+    int ret;
+
+    assert(size);
+    ret = qio_channel_read_all_eof(ioc, buffer, size, errp);
+    if (ret < 0) {
+        ret = -EIO;
+    }
+    return ret;
+}
+
 /* nbd_receive_reply
  * Returns 1 on success
  *         0 on eof, when no data was read (errp is not set)
  *         negative errno on failure (errp is set)
  */
-int nbd_receive_reply(QIOChannel *ioc, NBDReply *reply, Error **errp)
+int coroutine_fn nbd_receive_reply(QIOChannel *ioc, NBDReply *reply,
+                                   Error **errp)
 {
     int ret;
     const char *type;
-- 
2.20.1

[Qemu-devel] [PULL 15/71] nbd: Use low-level QIOChannel API in nbd_read_eof()

[Kevin Wolf]

Instead of using the convenience wrapper qio_channel_read_all_eof(), use
the lower level QIOChannel API. This means duplicating some code, but
we'll need this because this coroutine yield is special: We want it to
be interruptible so that nbd_client_attach_aio_context() can correctly
reenter the coroutine.

This moves the bdrv_dec/inc_in_flight() pair into nbd_read_eof(), so
that connection_co will always sit in this exact qio_channel_yield()
call when bdrv_drain() returns.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 include/block/nbd.h |  4 ++--
 block/nbd-client.c  |  8 +-------
 nbd/client.c        | 46 ++++++++++++++++++++++++++++++++++++---------
 3 files changed, 40 insertions(+), 18 deletions(-)

diff --git a/include/block/nbd.h b/include/block/nbd.h
index cad975e00c..c6ef1ef42e 100644
--- a/include/block/nbd.h
+++ b/include/block/nbd.h
@@ -300,8 +300,8 @@ int nbd_receive_export_list(QIOChannel *ioc, QCryptoTLSCreds *tlscreds,
 int nbd_init(int fd, QIOChannelSocket *sioc, NBDExportInfo *info,
              Error **errp);
 int nbd_send_request(QIOChannel *ioc, NBDRequest *request);
-int coroutine_fn nbd_receive_reply(QIOChannel *ioc, NBDReply *reply,
-                                   Error **errp);
+int coroutine_fn nbd_receive_reply(BlockDriverState *bs, QIOChannel *ioc,
+                                   NBDReply *reply, Error **errp);
 int nbd_client(int fd);
 int nbd_disconnect(int fd);
 int nbd_errno_to_system_errno(int err);
diff --git a/block/nbd-client.c b/block/nbd-client.c
index 5ce4aa9520..60f38f0320 100644
--- a/block/nbd-client.c
+++ b/block/nbd-client.c
@@ -84,15 +84,9 @@ static coroutine_fn void nbd_connection_entry(void *opaque)
          *
          * Therefore we keep an additional in_flight reference all the time and
          * only drop it temporarily here.
-         *
-         * FIXME This is not safe because the QIOChannel could wake up the
-         * coroutine for a second time; it is not prepared for coroutine
-         * resumption from external code.
          */
-        bdrv_dec_in_flight(s->bs);
         assert(s->reply.handle == 0);
-        ret = nbd_receive_reply(s->ioc, &s->reply, &local_err);
-        bdrv_inc_in_flight(s->bs);
+        ret = nbd_receive_reply(s->bs, s->ioc, &s->reply, &local_err);
 
         if (local_err) {
             trace_nbd_read_reply_entry_fail(ret, error_get_pretty(local_err));
diff --git a/nbd/client.c b/nbd/client.c
index 28d174c0f3..de7da48246 100644
--- a/nbd/client.c
+++ b/nbd/client.c
@@ -1394,30 +1394,58 @@ static int nbd_receive_structured_reply_chunk(QIOChannel *ioc,
  *         negative errno on failure (errp is set)
  */
 static inline int coroutine_fn
-nbd_read_eof(QIOChannel *ioc, void *buffer, size_t size, Error **errp)
+nbd_read_eof(BlockDriverState *bs, QIOChannel *ioc, void *buffer, size_t size,
+             Error **errp)
 {
-    int ret;
+    bool partial = false;
 
     assert(size);
-    ret = qio_channel_read_all_eof(ioc, buffer, size, errp);
-    if (ret < 0) {
-        ret = -EIO;
+    while (size > 0) {
+        struct iovec iov = { .iov_base = buffer, .iov_len = size };
+        ssize_t len;
+
+        len = qio_channel_readv(ioc, &iov, 1, errp);
+        if (len == QIO_CHANNEL_ERR_BLOCK) {
+            bdrv_dec_in_flight(bs);
+            qio_channel_yield(ioc, G_IO_IN);
+            bdrv_inc_in_flight(bs);
+            continue;
+        } else if (len < 0) {
+            return -EIO;
+        } else if (len == 0) {
+            if (partial) {
+                error_setg(errp,
+                           "Unexpected end-of-file before all bytes were read");
+                return -EIO;
+            } else {
+                return 0;
+            }
+        }
+
+        partial = true;
+        size -= len;
+        buffer = (uint8_t*) buffer + len;
     }
-    return ret;
+    return 1;
 }
 
 /* nbd_receive_reply
+ *
+ * Decreases bs->in_flight while waiting for a new reply. This yield is where
+ * we wait indefinitely and the coroutine must be able to be safely reentered
+ * for nbd_client_attach_aio_context().
+ *
  * Returns 1 on success
  *         0 on eof, when no data was read (errp is not set)
  *         negative errno on failure (errp is set)
  */
-int coroutine_fn nbd_receive_reply(QIOChannel *ioc, NBDReply *reply,
-                                   Error **errp)
+int coroutine_fn nbd_receive_reply(BlockDriverState *bs, QIOChannel *ioc,
+                                   NBDReply *reply, Error **errp)
 {
     int ret;
     const char *type;
 
-    ret = nbd_read_eof(ioc, &reply->magic, sizeof(reply->magic), errp);
+    ret = nbd_read_eof(bs, ioc, &reply->magic, sizeof(reply->magic), errp);
     if (ret <= 0) {
         return ret;
     }
-- 
2.20.1

[Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Kevin Wolf]

bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
needs to be increased while the coroutine is waiting to be scheduled
in the new AioContext after nbd_client_attach_aio_context().

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/nbd-client.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/block/nbd-client.c b/block/nbd-client.c
index 60f38f0320..bfbaf7ebe9 100644
--- a/block/nbd-client.c
+++ b/block/nbd-client.c
@@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
     qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
 }
 
+static void nbd_client_attach_aio_context_bh(void *opaque)
+{
+    BlockDriverState *bs = opaque;
+    NBDClientSession *client = nbd_get_client_session(bs);
+
+    /* The node is still drained, so we know the coroutine has yielded in
+     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
+     * entered for the first time. Both places are safe for entering the
+     * coroutine.*/
+    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
+    bdrv_dec_in_flight(bs);
+}
+
 void nbd_client_attach_aio_context(BlockDriverState *bs,
                                    AioContext *new_context)
 {
     NBDClientSession *client = nbd_get_client_session(bs);
     qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
 
-    /* FIXME Really need a bdrv_inc_in_flight() here */
-    aio_co_schedule(new_context, client->connection_co);
+    bdrv_inc_in_flight(bs);
+
+    /* Need to wait here for the BH to run because the BH must run while the
+     * node is still drained. */
+    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
 }
 
 void nbd_client_close(BlockDriverState *bs)
-- 
2.20.1

[Qemu-devel] [PULL 17/71] block: Don't poll in bdrv_set_aio_context()

[Kevin Wolf]

The explicit aio_poll() call in bdrv_set_aio_context() was added in
commit c2b6428d388 as a workaround for bdrv_drain() failing to achieve
to actually quiesce everything (specifically the NBD client code to
switch AioContext).

Now that the NBD client has been fixed to complete this operation during
bdrv_drain(), we don't need the workaround any more.

It was wrong anyway: aio_poll() must always be run in the home thread of
the AioContext.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 block.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/block.c b/block.c
index 0c12632661..17bc1d3dca 100644
--- a/block.c
+++ b/block.c
@@ -5273,10 +5273,6 @@ void bdrv_set_aio_context(BlockDriverState *bs, AioContext *new_context)
     bdrv_parent_drained_begin(bs, NULL, false);
     bdrv_drain(bs); /* ensure there are no in-flight requests */
 
-    while (aio_poll(ctx, false)) {
-        /* wait for all bottom halves to execute */
-    }
-
     bdrv_detach_aio_context(bs);
 
     /* This function executes in the old AioContext so acquire the new one in
-- 
2.20.1

[Qemu-devel] [PULL 18/71] block: Fix AioContext switch for drained node

[Kevin Wolf]

When a drained node changes its AioContext, we need to move its
aio_disable_external() to the new context, too.

Without this fix, drain_end will try to reenable the new context, which
has never been disabled, so an assertion failure is triggered.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 block.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/block.c b/block.c
index 17bc1d3dca..aefb5701f5 100644
--- a/block.c
+++ b/block.c
@@ -5227,6 +5227,9 @@ void bdrv_detach_aio_context(BlockDriverState *bs)
         bdrv_detach_aio_context(child->bs);
     }
 
+    if (bs->quiesce_counter) {
+        aio_enable_external(bs->aio_context);
+    }
     bs->aio_context = NULL;
 }
 
@@ -5240,6 +5243,10 @@ void bdrv_attach_aio_context(BlockDriverState *bs,
         return;
     }
 
+    if (bs->quiesce_counter) {
+        aio_disable_external(new_context);
+    }
+
     bs->aio_context = new_context;
 
     QLIST_FOREACH(child, &bs->children, next) {
-- 
2.20.1

[Qemu-devel] [PULL 19/71] test-bdrv-drain: AioContext switch in drained section

[Kevin Wolf]

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 tests/test-bdrv-drain.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/tests/test-bdrv-drain.c b/tests/test-bdrv-drain.c
index ee1740ff06..1b1f6c17a5 100644
--- a/tests/test-bdrv-drain.c
+++ b/tests/test-bdrv-drain.c
@@ -1522,6 +1522,36 @@ static void test_append_to_drained(void)
     blk_unref(blk);
 }
 
+static void test_set_aio_context(void)
+{
+    BlockDriverState *bs;
+    IOThread *a = iothread_new();
+    IOThread *b = iothread_new();
+    AioContext *ctx_a = iothread_get_aio_context(a);
+    AioContext *ctx_b = iothread_get_aio_context(b);
+
+    bs = bdrv_new_open_driver(&bdrv_test, "test-node", BDRV_O_RDWR,
+                              &error_abort);
+
+    bdrv_drained_begin(bs);
+    bdrv_set_aio_context(bs, ctx_a);
+
+    aio_context_acquire(ctx_a);
+    bdrv_drained_end(bs);
+
+    bdrv_drained_begin(bs);
+    bdrv_set_aio_context(bs, ctx_b);
+    aio_context_release(ctx_a);
+    aio_context_acquire(ctx_b);
+    bdrv_set_aio_context(bs, qemu_get_aio_context());
+    aio_context_release(ctx_b);
+    bdrv_drained_end(bs);
+
+    bdrv_unref(bs);
+    iothread_join(a);
+    iothread_join(b);
+}
+
 int main(int argc, char **argv)
 {
     int ret;
@@ -1603,6 +1633,8 @@ int main(int argc, char **argv)
 
     g_test_add_func("/bdrv-drain/attach/drain", test_append_to_drained);
 
+    g_test_add_func("/bdrv-drain/set_aio_context", test_set_aio_context);
+
     ret = g_test_run();
     qemu_event_destroy(&done_event);
     return ret;
-- 
2.20.1

[Qemu-devel] [PULL 20/71] block: Use normal drain for bdrv_set_aio_context()

[Kevin Wolf]

Now that bdrv_set_aio_context() works inside drained sections, it can
also use the real drain function instead of open coding something
similar.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/block.c b/block.c
index aefb5701f5..375a216f76 100644
--- a/block.c
+++ b/block.c
@@ -5268,18 +5268,16 @@ void bdrv_attach_aio_context(BlockDriverState *bs,
     bs->walking_aio_notifiers = false;
 }
 
+/* The caller must own the AioContext lock for the old AioContext of bs, but it
+ * must not own the AioContext lock for new_context (unless new_context is
+ * the same as the current context of bs). */
 void bdrv_set_aio_context(BlockDriverState *bs, AioContext *new_context)
 {
-    AioContext *ctx = bdrv_get_aio_context(bs);
-
-    if (ctx == new_context) {
+    if (bdrv_get_aio_context(bs) == new_context) {
         return;
     }
 
-    aio_disable_external(ctx);
-    bdrv_parent_drained_begin(bs, NULL, false);
-    bdrv_drain(bs); /* ensure there are no in-flight requests */
-
+    bdrv_drained_begin(bs);
     bdrv_detach_aio_context(bs);
 
     /* This function executes in the old AioContext so acquire the new one in
@@ -5287,8 +5285,7 @@ void bdrv_set_aio_context(BlockDriverState *bs, AioContext *new_context)
      */
     aio_context_acquire(new_context);
     bdrv_attach_aio_context(bs, new_context);
-    bdrv_parent_drained_end(bs, NULL, false);
-    aio_enable_external(ctx);
+    bdrv_drained_end(bs);
     aio_context_release(new_context);
 }
 
-- 
2.20.1

[Qemu-devel] [PULL 21/71] aio-posix: Assert that aio_poll() is always called in home thread

[Kevin Wolf]

aio_poll() has an existing assertion that the function is only called
from the AioContext's home thread if blocking is allowed.

This is not enough, some handlers make assumptions about the thread they
run in. Extend the assertion to non-blocking calls, too.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 util/aio-posix.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/util/aio-posix.c b/util/aio-posix.c
index 8640dfde9f..6fbfa7924f 100644
--- a/util/aio-posix.c
+++ b/util/aio-posix.c
@@ -613,6 +613,8 @@ bool aio_poll(AioContext *ctx, bool blocking)
     int64_t timeout;
     int64_t start = 0;
 
+    assert(in_aio_context_home_thread(ctx));
+
     /* aio_notify can avoid the expensive event_notifier_set if
      * everything (file descriptors, bottom halves, timers) will
      * be re-evaluated before the next blocking poll().  This is
@@ -621,7 +623,6 @@ bool aio_poll(AioContext *ctx, bool blocking)
      * so disable the optimization now.
      */
     if (blocking) {
-        assert(in_aio_context_home_thread(ctx));
         atomic_add(&ctx->notify_me, 2);
     }
 
-- 
2.20.1

[Qemu-devel] [PULL 22/71] block: improve should_update_child

[Kevin Wolf]

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

As it already said in the comment, we don't want to create loops in
parent->child relations. So, when we try to append @to to @c, we should
check that @c is not in @to children subtree, and we should check it
recursively, not only the first level. The patch provides BFS-based
search, to check the relations.

This is needed for further fleecing-hook filter usage: we need to
append it to source, when the hook is already a parent of target, and
source may be in a backing chain of target (fleecing-scheme). So, on
appending, the hook should not became a child (direct or through
children subtree) of the target.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block.c | 43 +++++++++++++++++++++++++++++++++++++------
 1 file changed, 37 insertions(+), 6 deletions(-)

diff --git a/block.c b/block.c
index 375a216f76..bb4bf1237c 100644
--- a/block.c
+++ b/block.c
@@ -3542,7 +3542,9 @@ void bdrv_close_all(void)
 
 static bool should_update_child(BdrvChild *c, BlockDriverState *to)
 {
-    BdrvChild *to_c;
+    GQueue *queue;
+    GHashTable *found;
+    bool ret;
 
     if (c->role->stay_at_node) {
         return false;
@@ -3578,14 +3580,43 @@ static bool should_update_child(BdrvChild *c, BlockDriverState *to)
      * if A is a child of B, that means we cannot replace A by B there
      * because that would create a loop.  Silently detaching A from B
      * is also not really an option.  So overall just leaving A in
-     * place there is the most sensible choice. */
-    QLIST_FOREACH(to_c, &to->children, next) {
-        if (to_c == c) {
-            return false;
+     * place there is the most sensible choice.
+     *
+     * We would also create a loop in any cases where @c is only
+     * indirectly referenced by @to. Prevent this by returning false
+     * if @c is found (by breadth-first search) anywhere in the whole
+     * subtree of @to.
+     */
+
+    ret = true;
+    found = g_hash_table_new(NULL, NULL);
+    g_hash_table_add(found, to);
+    queue = g_queue_new();
+    g_queue_push_tail(queue, to);
+
+    while (!g_queue_is_empty(queue)) {
+        BlockDriverState *v = g_queue_pop_head(queue);
+        BdrvChild *c2;
+
+        QLIST_FOREACH(c2, &v->children, next) {
+            if (c2 == c) {
+                ret = false;
+                break;
+            }
+
+            if (g_hash_table_contains(found, c2->bs)) {
+                continue;
+            }
+
+            g_queue_push_tail(queue, c2->bs);
+            g_hash_table_add(found, c2->bs);
         }
     }
 
-    return true;
+    g_queue_free(queue);
+    g_hash_table_destroy(found);
+
+    return ret;
 }
 
 void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
-- 
2.20.1

[Qemu-devel] [PULL 23/71] block: fix bdrv_check_perm for non-tree subgraph

[Kevin Wolf]

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

bdrv_check_perm in it's recursion checks each node in context of new
permissions for one parent, because of nature of DFS. It works well,
while children subgraph of top-most updated node is a tree, i.e. it
doesn't have any kind of loops. But if we have a loop (not oriented,
of course), i.e. we have two different ways from top-node to some
child-node, then bdrv_check_perm will do wrong thing:

  top
  | \
  |  |
  v  v
  A  B
  |  |
  v  v
  node

It will once check new permissions of node in context of new A
permissions and old B permissions and once visa-versa. It's a wrong way
and may lead to corruption of permission system. We may start with
no-permissions and all-shared for both A->node and B->node relations
and finish up with non shared write permission for both ways.

The following commit will add a test, which shows this bug.

To fix this situation, let's really set BdrvChild permissions during
bdrv_check_perm procedure. And we are happy here, as check-perm is
already written in transaction manner, so we just need to restore
backed-up permissions in _abort.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 include/block/block_int.h |  5 +++++
 block.c                   | 27 ++++++++++++++++++++++++++-
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index 0075bafd10..8437df85a2 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -662,6 +662,11 @@ struct BdrvChild {
      */
     uint64_t shared_perm;
 
+    /* backup of permissions during permission update procedure */
+    bool has_backup_perm;
+    uint64_t backup_perm;
+    uint64_t backup_shared_perm;
+
     QLIST_ENTRY(BdrvChild) next;
     QLIST_ENTRY(BdrvChild) next_parent;
 };
diff --git a/block.c b/block.c
index bb4bf1237c..16d59e0b32 100644
--- a/block.c
+++ b/block.c
@@ -1954,13 +1954,32 @@ static int bdrv_child_check_perm(BdrvChild *c, BlockReopenQueue *q,
     ret = bdrv_check_update_perm(c->bs, q, perm, shared, ignore_children, errp);
     g_slist_free(ignore_children);
 
-    return ret;
+    if (ret < 0) {
+        return ret;
+    }
+
+    if (!c->has_backup_perm) {
+        c->has_backup_perm = true;
+        c->backup_perm = c->perm;
+        c->backup_shared_perm = c->shared_perm;
+    }
+    /*
+     * Note: it's OK if c->has_backup_perm was already set, as we can find the
+     * same child twice during check_perm procedure
+     */
+
+    c->perm = perm;
+    c->shared_perm = shared;
+
+    return 0;
 }
 
 static void bdrv_child_set_perm(BdrvChild *c, uint64_t perm, uint64_t shared)
 {
     uint64_t cumulative_perms, cumulative_shared_perms;
 
+    c->has_backup_perm = false;
+
     c->perm = perm;
     c->shared_perm = shared;
 
@@ -1971,6 +1990,12 @@ static void bdrv_child_set_perm(BdrvChild *c, uint64_t perm, uint64_t shared)
 
 static void bdrv_child_abort_perm_update(BdrvChild *c)
 {
+    if (c->has_backup_perm) {
+        c->perm = c->backup_perm;
+        c->shared_perm = c->backup_shared_perm;
+        c->has_backup_perm = false;
+    }
+
     bdrv_abort_perm_update(c->bs);
 }
 
-- 
2.20.1

[Qemu-devel] [PULL 24/71] tests: add test-bdrv-graph-mod

[Kevin Wolf]

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Add two tests of node graph modification.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/test-bdrv-graph-mod.c | 198 ++++++++++++++++++++++++++++++++++++
 tests/Makefile.include      |   2 +
 2 files changed, 200 insertions(+)
 create mode 100644 tests/test-bdrv-graph-mod.c

diff --git a/tests/test-bdrv-graph-mod.c b/tests/test-bdrv-graph-mod.c
new file mode 100644
index 0000000000..458dfa6661
--- /dev/null
+++ b/tests/test-bdrv-graph-mod.c
@@ -0,0 +1,198 @@
+/*
+ * Block node graph modifications tests
+ *
+ * Copyright (c) 2019 Virtuozzo International GmbH. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "block/block_int.h"
+#include "sysemu/block-backend.h"
+
+static BlockDriver bdrv_pass_through = {
+    .format_name = "pass-through",
+    .bdrv_child_perm = bdrv_filter_default_perms,
+};
+
+static void no_perm_default_perms(BlockDriverState *bs, BdrvChild *c,
+                                         const BdrvChildRole *role,
+                                         BlockReopenQueue *reopen_queue,
+                                         uint64_t perm, uint64_t shared,
+                                         uint64_t *nperm, uint64_t *nshared)
+{
+    *nperm = 0;
+    *nshared = BLK_PERM_ALL;
+}
+
+static BlockDriver bdrv_no_perm = {
+    .format_name = "no-perm",
+    .bdrv_child_perm = no_perm_default_perms,
+};
+
+static BlockDriverState *no_perm_node(const char *name)
+{
+    return bdrv_new_open_driver(&bdrv_no_perm, name, BDRV_O_RDWR, &error_abort);
+}
+
+static BlockDriverState *pass_through_node(const char *name)
+{
+    return bdrv_new_open_driver(&bdrv_pass_through, name,
+                                BDRV_O_RDWR, &error_abort);
+}
+
+/*
+ * test_update_perm_tree
+ *
+ * When checking node for a possibility to update permissions, it's subtree
+ * should be correctly checked too. New permissions for each node should be
+ * calculated and checked in context of permissions of other nodes. If we
+ * check new permissions of the node only in context of old permissions of
+ * its neighbors, we can finish up with wrong permission graph.
+ *
+ * This test firstly create the following graph:
+ *                                +--------+
+ *                                |  root  |
+ *                                +--------+
+ *                                    |
+ *                                    | perm: write, read
+ *                                    | shared: except write
+ *                                    v
+ *  +-------------------+           +----------------+
+ *  | passtrough filter |---------->|  null-co node  |
+ *  +-------------------+           +----------------+
+ *
+ *
+ * and then, tries to append filter under node. Expected behavior: fail.
+ * Otherwise we'll get the following picture, with two BdrvChild'ren, having
+ * write permission to one node, without actually sharing it.
+ *
+ *                     +--------+
+ *                     |  root  |
+ *                     +--------+
+ *                         |
+ *                         | perm: write, read
+ *                         | shared: except write
+ *                         v
+ *                +-------------------+
+ *                | passtrough filter |
+ *                +-------------------+
+ *                       |   |
+ *     perm: write, read |   | perm: write, read
+ *  shared: except write |   | shared: except write
+ *                       v   v
+ *                +----------------+
+ *                |  null co node  |
+ *                +----------------+
+ */
+static void test_update_perm_tree(void)
+{
+    Error *local_err = NULL;
+
+    BlockBackend *root = blk_new(BLK_PERM_WRITE | BLK_PERM_CONSISTENT_READ,
+                                 BLK_PERM_ALL & ~BLK_PERM_WRITE);
+    BlockDriverState *bs = no_perm_node("node");
+    BlockDriverState *filter = pass_through_node("filter");
+
+    blk_insert_bs(root, bs, &error_abort);
+
+    bdrv_attach_child(filter, bs, "child", &child_file, &error_abort);
+
+    bdrv_append(filter, bs, &local_err);
+
+    g_assert_nonnull(local_err);
+
+    bdrv_unref(bs);
+    blk_unref(root);
+}
+
+/*
+ * test_should_update_child
+ *
+ * Test that bdrv_replace_node, and concretely should_update_child
+ * do the right thing, i.e. not creating loops on the graph.
+ *
+ * The test does the following:
+ * 1. initial graph:
+ *
+ *   +------+          +--------+
+ *   | root |          | filter |
+ *   +------+          +--------+
+ *      |                  |
+ *  root|            target|
+ *      v                  v
+ *   +------+          +--------+
+ *   | node |<---------| target |
+ *   +------+  backing +--------+
+ *
+ * 2. Append @filter above @node. If should_update_child works correctly,
+ * it understands, that backing child of @target should not be updated,
+ * as it will create a loop on node graph. Resulting picture should
+ * be the left one, not the right:
+ *
+ *     +------+                            +------+
+ *     | root |                            | root |
+ *     +------+                            +------+
+ *        |                                   |
+ *    root|                               root|
+ *        v                                   v
+ *    +--------+   target                 +--------+   target
+ *    | filter |--------------+           | filter |--------------+
+ *    +--------+              |           +--------+              |
+ *        |                   |               |  ^                v
+ * backing|                   |        backing|  |           +--------+
+ *        v                   v               |  +-----------| target |
+ *     +------+          +--------+           v      backing +--------+
+ *     | node |<---------| target |        +------+
+ *     +------+  backing +--------+        | node |
+ *                                         +------+
+ *
+ *    (good picture)                       (bad picture)
+ *
+ */
+static void test_should_update_child(void)
+{
+    BlockBackend *root = blk_new(0, BLK_PERM_ALL);
+    BlockDriverState *bs = no_perm_node("node");
+    BlockDriverState *filter = no_perm_node("filter");
+    BlockDriverState *target = no_perm_node("target");
+
+    blk_insert_bs(root, bs, &error_abort);
+
+    bdrv_set_backing_hd(target, bs, &error_abort);
+
+    g_assert(target->backing->bs == bs);
+    bdrv_attach_child(filter, target, "target", &child_file, &error_abort);
+    bdrv_append(filter, bs, &error_abort);
+    g_assert(target->backing->bs == bs);
+
+    bdrv_unref(bs);
+    blk_unref(root);
+}
+
+int main(int argc, char *argv[])
+{
+    bdrv_init();
+    qemu_init_main_loop(&error_abort);
+
+    g_test_init(&argc, &argv, NULL);
+
+    g_test_add_func("/bdrv-graph-mod/update-perm-tree", test_update_perm_tree);
+    g_test_add_func("/bdrv-graph-mod/should-update-child",
+                    test_should_update_child);
+
+    return g_test_run();
+}
diff --git a/tests/Makefile.include b/tests/Makefile.include
index b39e989f72..992378e031 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -70,6 +70,7 @@ check-unit-y += tests/test-throttle$(EXESUF)
 check-unit-y += tests/test-thread-pool$(EXESUF)
 check-unit-y += tests/test-hbitmap$(EXESUF)
 check-unit-y += tests/test-bdrv-drain$(EXESUF)
+check-unit-y += tests/test-bdrv-graph-mod$(EXESUF)
 check-unit-y += tests/test-blockjob$(EXESUF)
 check-unit-y += tests/test-blockjob-txn$(EXESUF)
 check-unit-y += tests/test-block-backend$(EXESUF)
@@ -555,6 +556,7 @@ tests/test-aio$(EXESUF): tests/test-aio.o $(test-block-obj-y)
 tests/test-aio-multithread$(EXESUF): tests/test-aio-multithread.o $(test-block-obj-y)
 tests/test-throttle$(EXESUF): tests/test-throttle.o $(test-block-obj-y)
 tests/test-bdrv-drain$(EXESUF): tests/test-bdrv-drain.o $(test-block-obj-y) $(test-util-obj-y)
+tests/test-bdrv-graph-mod$(EXESUF): tests/test-bdrv-graph-mod.o $(test-block-obj-y) $(test-util-obj-y)
 tests/test-blockjob$(EXESUF): tests/test-blockjob.o $(test-block-obj-y) $(test-util-obj-y)
 tests/test-blockjob-txn$(EXESUF): tests/test-blockjob-txn.o $(test-block-obj-y) $(test-util-obj-y)
 tests/test-block-backend$(EXESUF): tests/test-block-backend.o $(test-block-obj-y) $(test-util-obj-y)
-- 
2.20.1

[Qemu-devel] [PULL 25/71] qcow2: Assert that L2 table offsets fit in the L1 table

[Kevin Wolf]

From: Alberto Garcia <berto@igalia.com>

L1 table entries have a field to store the offset of an L2 table.
The rest of the bits of the entry are currently reserved except from
bit 63, which stores the COPIED flag.

The offset is always taken from the entry using L1E_OFFSET_MASK to
ensure that we only use the bits that belong to that field.

While that mask is used every time we read from the L1 table, it is
never used when we write to it. Due to the limits set elsewhere in the
code QEMU can never produce L2 table offsets that don't fit in that
field so any such offset when allocating an L2 table would indicate a
bug in QEMU.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/qcow2-cluster.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 30eca26c47..179aa2c728 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -285,6 +285,9 @@ static int l2_allocate(BlockDriverState *bs, int l1_index)
         goto fail;
     }
 
+    /* The offset must fit in the offset field of the L1 table entry */
+    assert((l2_offset & L1E_OFFSET_MASK) == l2_offset);
+
     /* If we're allocating the table at offset 0 then something is wrong */
     if (l2_offset == 0) {
         qcow2_signal_corruption(bs, true, -1, -1, "Preventing invalid "
-- 
2.20.1

[Qemu-devel] [PULL 26/71] block/nvme: Remove QEMU_PACKED from naturally aligned NVMeRegs struct

[Kevin Wolf]

From: Thomas Huth <thuth@redhat.com>

The QEMU_PACKED is causing a compiler warning/error with GCC 9:

  CC      block/nvme.o
block/nvme.c: In function ‘nvme_create_queue_pair’:
block/nvme.c:209:22: error: taking address of packed member of
 ‘struct <anonymous>’ may result in an unaligned pointer value
 [-Werror=address-of-packed-member]
  209 |     q->sq.doorbell = &s->regs->doorbells[idx * 2 * s->doorbell_scale];

All members of the struct are naturally aligned, so there should
not be the need for QEMU_PACKED here, and the following QEMU_BUILD_BUG_ON
also ensures that there is no padding. Thus simply remove the QEMU_PACKED
here.

Buglink: https://bugs.launchpad.net/qemu/+bug/1817525
Reported-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/nvme.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/nvme.c b/block/nvme.c
index b5952c9b08..6c2ce7dfa5 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -82,7 +82,7 @@ typedef volatile struct {
     uint8_t  reserved1[0xec0];
     uint8_t  cmd_set_specfic[0x100];
     uint32_t doorbells[];
-} QEMU_PACKED NVMeRegs;
+} NVMeRegs;
 
 QEMU_BUILD_BUG_ON(offsetof(NVMeRegs, doorbells) != 0x1000);
 
-- 
2.20.1

[Qemu-devel] [PULL 27/71] block: Use bdrv_refresh_filename() to pull

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Before this patch, bdrv_refresh_filename() is used in a pushing manner:
Whenever the BDS graph is modified, the parents of the modified edges
are supposed to be updated (recursively upwards).  However, that is
nonviable, considering that we want child changes not to concern
parents.

Also, in the long run we want a pull model anyway: Here, we would have a
bdrv_filename() function which returns a BDS's filename, freshly
constructed.

This patch is an intermediate step.  It adds bdrv_refresh_filename()
calls before every place a BDS.filename value is used.  The only
exceptions are protocol drivers that use their own filename, which
clearly would not profit from refreshing that filename before.

Also, bdrv_get_encrypted_filename() is removed along the way (as a user
of BDS.filename), since it is completely unused.

In turn, all of the calls to bdrv_refresh_filename() before this patch
are removed, because we no longer have to call this function on graph
changes.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-2-mreitz@redhat.com
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block.h |  1 -
 block.c               | 31 +++++++++++++++----------------
 block/qapi.c          |  4 ++++
 block/raw-format.c    |  1 +
 block/replication.c   |  2 --
 block/vhdx-log.c      |  1 +
 block/vmdk.c          |  6 ++++++
 blockdev.c            |  8 ++++++++
 qemu-img.c            | 11 +++++++++--
 9 files changed, 44 insertions(+), 21 deletions(-)

diff --git a/include/block/block.h b/include/block/block.h
index 73357c6c25..aaae900925 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -485,7 +485,6 @@ void bdrv_round_to_clusters(BlockDriverState *bs,
                             int64_t *cluster_offset,
                             int64_t *cluster_bytes);
 
-const char *bdrv_get_encrypted_filename(BlockDriverState *bs);
 void bdrv_get_backing_filename(BlockDriverState *bs,
                                char *filename, int filename_size);
 void bdrv_get_full_backing_filename(BlockDriverState *bs,
diff --git a/block.c b/block.c
index 16d59e0b32..88aa687a68 100644
--- a/block.c
+++ b/block.c
@@ -323,8 +323,11 @@ void bdrv_get_full_backing_filename_from_filename(const char *backed,
 void bdrv_get_full_backing_filename(BlockDriverState *bs, char *dest, size_t sz,
                                     Error **errp)
 {
-    char *backed = bs->exact_filename[0] ? bs->exact_filename : bs->filename;
+    char *backed;
 
+    bdrv_refresh_filename(bs);
+
+    backed = bs->exact_filename[0] ? bs->exact_filename : bs->filename;
     bdrv_get_full_backing_filename_from_filename(backed, bs->backing_file,
                                                  dest, sz, errp);
 }
@@ -1004,6 +1007,8 @@ static void bdrv_backing_attach(BdrvChild *c)
                "node is used as backing hd of '%s'",
                bdrv_get_device_or_node_name(parent));
 
+    bdrv_refresh_filename(backing_hd);
+
     parent->open_flags &= ~BDRV_O_NO_BACKING;
     pstrcpy(parent->backing_file, sizeof(parent->backing_file),
             backing_hd->filename);
@@ -1413,6 +1418,7 @@ static int bdrv_open_common(BlockDriverState *bs, BlockBackend *file,
     }
 
     if (file != NULL) {
+        bdrv_refresh_filename(blk_bs(file));
         filename = blk_bs(file)->filename;
     } else {
         /*
@@ -2334,8 +2340,6 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
         bdrv_unref(backing_hd);
     }
 
-    bdrv_refresh_filename(bs);
-
 out:
     bdrv_refresh_limits(bs, NULL);
 }
@@ -2864,8 +2868,6 @@ static BlockDriverState *bdrv_open_inherit(const char *filename,
         g_free(child_key_dot);
     }
 
-    bdrv_refresh_filename(bs);
-
     /* Check if any unknown options were used */
     if (qdict_size(options) != 0) {
         const QDictEntry *entry = qdict_first(options);
@@ -3310,6 +3312,7 @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,
             if (local_err != NULL) {
                 error_propagate(errp, local_err);
             } else {
+                bdrv_refresh_filename(reopen_state->bs);
                 error_setg(errp, "failed while preparing to reopen image '%s'",
                            reopen_state->bs->filename);
             }
@@ -3937,7 +3940,10 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,
     /* success - we can delete the intermediate states, and link top->base */
     /* TODO Check graph modification op blockers (BLK_PERM_GRAPH_MOD) once
      * we've figured out how they should work. */
-    backing_file_str = backing_file_str ? backing_file_str : base->filename;
+    if (!backing_file_str) {
+        bdrv_refresh_filename(base);
+        backing_file_str = base->filename;
+    }
 
     QLIST_FOREACH_SAFE(c, &top->parents, next_parent, next) {
         /* Check whether we are allowed to switch c from top to base */
@@ -4485,16 +4491,6 @@ bool bdrv_can_write_zeroes_with_unmap(BlockDriverState *bs)
     return bs->supported_zero_flags & BDRV_REQ_MAY_UNMAP;
 }
 
-const char *bdrv_get_encrypted_filename(BlockDriverState *bs)
-{
-    if (bs->backing && bs->backing->bs->encrypted)
-        return bs->backing_file;
-    else if (bs->encrypted)
-        return bs->filename;
-    else
-        return NULL;
-}
-
 void bdrv_get_backing_filename(BlockDriverState *bs,
                                char *filename, int filename_size)
 {
@@ -4615,6 +4611,9 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
 
     is_protocol = path_has_protocol(backing_file);
 
+    /* This will recursively refresh everything in the backing chain */
+    bdrv_refresh_filename(bs);
+
     for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) {
 
         /* If either of the filename paths is actually a protocol, then
diff --git a/block/qapi.c b/block/qapi.c
index 00291f9105..4623de1d7b 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -51,6 +51,8 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
         return NULL;
     }
 
+    bdrv_refresh_filename(bs);
+
     info = g_malloc0(sizeof(*info));
     info->file                   = g_strdup(bs->filename);
     info->ro                     = bs->read_only;
@@ -264,6 +266,8 @@ void bdrv_query_image_info(BlockDriverState *bs,
         goto out;
     }
 
+    bdrv_refresh_filename(bs);
+
     info = g_new0(ImageInfo, 1);
     info->filename        = g_strdup(bs->filename);
     info->format          = g_strdup(bdrv_get_format_name(bs));
diff --git a/block/raw-format.c b/block/raw-format.c
index 6f6dc99b2c..d07bcdae62 100644
--- a/block/raw-format.c
+++ b/block/raw-format.c
@@ -436,6 +436,7 @@ static int raw_open(BlockDriverState *bs, QDict *options, int flags,
             bs->file->bs->supported_zero_flags);
 
     if (bs->probed && !bdrv_is_read_only(bs)) {
+        bdrv_refresh_filename(bs->file->bs);
         fprintf(stderr,
                 "WARNING: Image format was not specified for '%s' and probing "
                 "guessed raw.\n"
diff --git a/block/replication.c b/block/replication.c
index e70dd95001..9b332002ee 100644
--- a/block/replication.c
+++ b/block/replication.c
@@ -616,8 +616,6 @@ static void replication_done(void *opaque, int ret)
     if (ret == 0) {
         s->stage = BLOCK_REPLICATION_DONE;
 
-        /* refresh top bs's filename */
-        bdrv_refresh_filename(bs);
         s->active_disk = NULL;
         s->secondary_disk = NULL;
         s->hidden_disk = NULL;
diff --git a/block/vhdx-log.c b/block/vhdx-log.c
index ecd64266c5..3149ff08d8 100644
--- a/block/vhdx-log.c
+++ b/block/vhdx-log.c
@@ -803,6 +803,7 @@ int vhdx_parse_log(BlockDriverState *bs, BDRVVHDXState *s, bool *flushed,
 
     if (logs.valid) {
         if (bs->read_only) {
+            bdrv_refresh_filename(bs);
             ret = -EPERM;
             error_setg(errp,
                        "VHDX image file '%s' opened read-only, but "
diff --git a/block/vmdk.c b/block/vmdk.c
index 096e8eb662..117dc6adfe 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -479,6 +479,7 @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
                      extent->l1_table,
                      l1_size);
     if (ret < 0) {
+        bdrv_refresh_filename(extent->file->bs);
         error_setg_errno(errp, -ret,
                          "Could not read l1 table from extent '%s'",
                          extent->file->bs->filename);
@@ -499,6 +500,7 @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
                          extent->l1_backup_table,
                          l1_size);
         if (ret < 0) {
+            bdrv_refresh_filename(extent->file->bs);
             error_setg_errno(errp, -ret,
                              "Could not read l1 backup table from extent '%s'",
                              extent->file->bs->filename);
@@ -530,6 +532,7 @@ static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
 
     ret = bdrv_pread(file, sizeof(magic), &header, sizeof(header));
     if (ret < 0) {
+        bdrv_refresh_filename(file->bs);
         error_setg_errno(errp, -ret,
                          "Could not read header from file '%s'",
                          file->bs->filename);
@@ -607,6 +610,7 @@ static int vmdk_open_vmdk4(BlockDriverState *bs,
 
     ret = bdrv_pread(file, sizeof(magic), &header, sizeof(header));
     if (ret < 0) {
+        bdrv_refresh_filename(file->bs);
         error_setg_errno(errp, -ret,
                          "Could not read header from file '%s'",
                          file->bs->filename);
@@ -861,6 +865,7 @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
         if (!path_is_absolute(fname) && !path_has_protocol(fname) &&
             !desc_file_path[0])
         {
+            bdrv_refresh_filename(bs->file->bs);
             error_setg(errp, "Cannot use relative extent paths with VMDK "
                        "descriptor file '%s'", bs->file->bs->filename);
             return -EINVAL;
@@ -2470,6 +2475,7 @@ static ImageInfo *vmdk_get_extent_info(VmdkExtent *extent)
 {
     ImageInfo *info = g_new0(ImageInfo, 1);
 
+    bdrv_refresh_filename(extent->file->bs);
     *info = (ImageInfo){
         .filename         = g_strdup(extent->file->bs->filename),
         .format           = g_strdup(extent->type),
diff --git a/blockdev.c b/blockdev.c
index 8714ad2702..7e6bf9955c 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1627,6 +1627,7 @@ static void external_snapshot_prepare(BlkActionState *common,
                 error_setg_errno(errp, -size, "bdrv_getlength failed");
                 goto out;
             }
+            bdrv_refresh_filename(state->old_bs);
             bdrv_img_create(new_image_file, format,
                             state->old_bs->filename,
                             state->old_bs->drv->format_name,
@@ -3230,6 +3231,7 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device,
             goto out;
         }
         assert(bdrv_get_aio_context(base_bs) == aio_context);
+        bdrv_refresh_filename(base_bs);
         base_name = base_bs->filename;
     }
 
@@ -3349,6 +3351,10 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
             goto out;
         }
     } else if (has_top && top) {
+        /* This strcmp() is just a shortcut, there is no need to
+         * refresh @bs's filename.  If it mismatches,
+         * bdrv_find_backing_image() will do the refresh and may still
+         * return @bs. */
         if (strcmp(bs->filename, top) != 0) {
             top_bs = bdrv_find_backing_image(bs, top);
         }
@@ -3509,6 +3515,7 @@ static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,
     if (backup->mode != NEW_IMAGE_MODE_EXISTING) {
         assert(backup->format);
         if (source) {
+            bdrv_refresh_filename(source);
             bdrv_img_create(backup->target, backup->format, source->filename,
                             source->drv->format_name, NULL,
                             size, flags, false, &local_err);
@@ -3889,6 +3896,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
             break;
         case NEW_IMAGE_MODE_ABSOLUTE_PATHS:
             /* create new image with backing file */
+            bdrv_refresh_filename(source);
             bdrv_img_create(arg->target, format,
                             source->filename,
                             source->drv->format_name,
diff --git a/qemu-img.c b/qemu-img.c
index eb5045c742..d2fc28c987 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -2790,6 +2790,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
     BlockDriverState *file;
     bool has_offset;
     int64_t map;
+    char *filename = NULL;
 
     /* As an optimization, we could cache the current range of unallocated
      * clusters in each file of the chain, and avoid querying the same
@@ -2817,6 +2818,11 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
 
     has_offset = !!(ret & BDRV_BLOCK_OFFSET_VALID);
 
+    if (file && has_offset) {
+        bdrv_refresh_filename(file);
+        filename = file->filename;
+    }
+
     *e = (MapEntry) {
         .start = offset,
         .length = bytes,
@@ -2825,8 +2831,8 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
         .offset = map,
         .has_offset = has_offset,
         .depth = depth,
-        .has_filename = file && has_offset,
-        .filename = file && has_offset ? file->filename : NULL,
+        .has_filename = filename,
+        .filename = filename,
     };
 
     return 0;
@@ -3334,6 +3340,7 @@ static int img_rebase(int argc, char **argv)
                 qdict_put_bool(options, BDRV_OPT_FORCE_SHARE, true);
             }
 
+            bdrv_refresh_filename(bs);
             overlay_filename = bs->exact_filename[0] ? bs->exact_filename
                                                      : bs->filename;
             out_real_path = g_malloc(PATH_MAX);
-- 
2.20.1

[Qemu-devel] [PULL 28/71] block: Use children list in bdrv_refresh_filename

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

bdrv_refresh_filename() should invoke itself recursively on all
children, not just on file.

With that change, we can remove the manual invocations in blkverify,
quorum, commit, mirror, and blklogwrites.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-3-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c              | 9 +++++----
 block/blklogwrites.c | 3 ---
 block/blkverify.c    | 3 ---
 block/commit.c       | 1 -
 block/mirror.c       | 1 -
 block/quorum.c       | 1 -
 6 files changed, 5 insertions(+), 13 deletions(-)

diff --git a/block.c b/block.c
index 88aa687a68..d97f5e7084 100644
--- a/block.c
+++ b/block.c
@@ -5536,16 +5536,17 @@ static bool append_open_options(QDict *d, BlockDriverState *bs)
 void bdrv_refresh_filename(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
+    BdrvChild *child;
     QDict *opts;
 
     if (!drv) {
         return;
     }
 
-    /* This BDS's file name will most probably depend on its file's name, so
-     * refresh that first */
-    if (bs->file) {
-        bdrv_refresh_filename(bs->file->bs);
+    /* This BDS's file name may depend on any of its children's file names, so
+     * refresh those first */
+    QLIST_FOREACH(child, &bs->children, next) {
+        bdrv_refresh_filename(child->bs);
     }
 
     if (drv->bdrv_refresh_filename) {
diff --git a/block/blklogwrites.c b/block/blklogwrites.c
index d2e01bdb1d..36e3d0f822 100644
--- a/block/blklogwrites.c
+++ b/block/blklogwrites.c
@@ -285,9 +285,6 @@ static void blk_log_writes_refresh_filename(BlockDriverState *bs,
 {
     BDRVBlkLogWritesState *s = bs->opaque;
 
-    /* bs->file->bs has already been refreshed */
-    bdrv_refresh_filename(s->log_file->bs);
-
     if (bs->file->bs->full_open_options
         && s->log_file->bs->full_open_options)
     {
diff --git a/block/blkverify.c b/block/blkverify.c
index 89bf4386e3..035d77b64a 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -285,9 +285,6 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
 {
     BDRVBlkverifyState *s = bs->opaque;
 
-    /* bs->file->bs has already been refreshed */
-    bdrv_refresh_filename(s->test_file->bs);
-
     if (bs->file->bs->full_open_options
         && s->test_file->bs->full_open_options)
     {
diff --git a/block/commit.c b/block/commit.c
index 5deb05925b..614a8ca374 100644
--- a/block/commit.c
+++ b/block/commit.c
@@ -232,7 +232,6 @@ static int coroutine_fn bdrv_commit_top_preadv(BlockDriverState *bs,
 
 static void bdrv_commit_top_refresh_filename(BlockDriverState *bs, QDict *opts)
 {
-    bdrv_refresh_filename(bs->backing->bs);
     pstrcpy(bs->exact_filename, sizeof(bs->exact_filename),
             bs->backing->bs->filename);
 }
diff --git a/block/mirror.c b/block/mirror.c
index b67b0120f8..031c1aeaeb 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -1438,7 +1438,6 @@ static void bdrv_mirror_top_refresh_filename(BlockDriverState *bs, QDict *opts)
          * bdrv_set_backing_hd */
         return;
     }
-    bdrv_refresh_filename(bs->backing->bs);
     pstrcpy(bs->exact_filename, sizeof(bs->exact_filename),
             bs->backing->bs->filename);
 }
diff --git a/block/quorum.c b/block/quorum.c
index 16b3c8067c..cf9d7c16c2 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -1073,7 +1073,6 @@ static void quorum_refresh_filename(BlockDriverState *bs, QDict *options)
     int i;
 
     for (i = 0; i < s->num_children; i++) {
-        bdrv_refresh_filename(s->children[i]->bs);
         if (!s->children[i]->bs->full_open_options) {
             return;
         }
-- 
2.20.1

[Qemu-devel] [PULL 29/71] block: Skip implicit nodes for filename info

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

bdrv_refresh_filename() should simply skip all implicit nodes.  They are
supposed to be invisible to the user, so they should not appear in
filename information.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-4-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/block.c b/block.c
index d97f5e7084..31e4664a42 100644
--- a/block.c
+++ b/block.c
@@ -5549,6 +5549,20 @@ void bdrv_refresh_filename(BlockDriverState *bs)
         bdrv_refresh_filename(child->bs);
     }
 
+    if (bs->implicit) {
+        /* For implicit nodes, just copy everything from the single child */
+        child = QLIST_FIRST(&bs->children);
+        assert(QLIST_NEXT(child, next) == NULL);
+
+        pstrcpy(bs->exact_filename, sizeof(bs->exact_filename),
+                child->bs->exact_filename);
+        pstrcpy(bs->filename, sizeof(bs->filename), child->bs->filename);
+
+        bs->full_open_options = qobject_ref(child->bs->full_open_options);
+
+        return;
+    }
+
     if (drv->bdrv_refresh_filename) {
         /* Obsolete information is of no use here, so drop the old file name
          * information before refreshing it */
-- 
2.20.1

[Qemu-devel] [PULL 30/71] block: Add BDS.auto_backing_file

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

If the backing file is overridden, this most probably does change the
guest-visible data of a BDS.  Therefore, we will need to consider this
in bdrv_refresh_filename().

To see whether it has been overridden, we might want to compare
bs->backing_file and bs->backing->bs->filename.  However,
bs->backing_file is changed by bdrv_set_backing_hd() (which is just used
to change the backing child at runtime, without modifying the image
header), so bs->backing_file most of the time simply contains a copy of
bs->backing->bs->filename anyway, so it is useless for such a
comparison.

This patch adds an auto_backing_file BDS field which contains the
backing file path as indicated by the image header, which is not changed
by bdrv_set_backing_hd().

Because of bdrv_refresh_filename() magic, however, a BDS's filename may
differ from what has been specified during bdrv_open().  Then, the
comparison between bs->auto_backing_file and bs->backing->bs->filename
may fail even though bs->backing was opened from bs->auto_backing_file.
To mitigate this, we can copy the real BDS's filename (after the whole
bdrv_open() and bdrv_refresh_filename() process) into
bs->auto_backing_file, if we know the former has been opened based on
the latter.  This is only possible if no options modifying the backing
file's behavior have been specified, though.  To simplify things, this
patch only copies the filename from the backing file if no options have
been specified for it at all.

Furthermore, there are cases where an overlay is created by qemu which
already contains a BDS's filename (e.g. in blockdev-snapshot-sync).  We
do not need to worry about updating the overlay's bs->auto_backing_file
there, because we actually wrote a post-bdrv_refresh_filename() filename
into the image header.

So all in all, there will be false negatives where (as of a future
patch) bdrv_refresh_filename() will assume that the backing file differs
from what was specified in the image header, even though it really does
not.  However, these cases should be limited to where (1) the user
actually did override something in the backing chain (e.g. by specifying
options for the backing file), or (2) the user executed a QMP command to
change some node's backing file (e.g. change-backing-file or
block-commit with @backing-file given) where the given filename does not
happen to coincide with qemu's idea of the backing BDS's filename.

Then again, (1) really is limited to -drive.  With -blockdev or
blockdev-add, you have to adhere to the schema, so a user cannot give
partial "unimportant" options (e.g. by just setting backing.node-name
and leaving the rest to the image header).  Therefore, trying to fix
this would mean trying to fix something for -drive only.

To improve on (2), we would need a full infrastructure to "canonicalize"
an arbitrary filename (+ options), so it can be compared against
another.  That seems a bit over the top, considering that filenames
nowadays are there mostly for the user's entertainment.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-5-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block_int.h |  4 ++++
 block.c                   | 19 +++++++++++++++++++
 block/qcow.c              |  7 +++++--
 block/qcow2.c             | 10 +++++++---
 block/qed.c               |  7 +++++--
 block/vmdk.c              |  6 ++++--
 6 files changed, 44 insertions(+), 9 deletions(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index 8437df85a2..dd7276cde2 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -702,6 +702,10 @@ struct BlockDriverState {
     char filename[PATH_MAX];
     char backing_file[PATH_MAX]; /* if non zero, the image is a diff of
                                     this file image */
+    /* The backing filename indicated by the image header; if we ever
+     * open this file, then this is replaced by the resulting BDS's
+     * filename (i.e. after a bdrv_refresh_filename() run). */
+    char auto_backing_file[PATH_MAX];
     char backing_format[16]; /* if non-zero and backing_file exists */
 
     QDict *full_open_options;
diff --git a/block.c b/block.c
index 31e4664a42..21395b546c 100644
--- a/block.c
+++ b/block.c
@@ -2361,6 +2361,7 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *parent_options,
     char *bdref_key_dot;
     const char *reference = NULL;
     int ret = 0;
+    bool implicit_backing = false;
     BlockDriverState *backing_hd;
     QDict *options;
     QDict *tmp_parent_options = NULL;
@@ -2396,6 +2397,16 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *parent_options,
         qobject_unref(options);
         goto free_exit;
     } else {
+        if (qdict_size(options) == 0) {
+            /* If the user specifies options that do not modify the
+             * backing file's behavior, we might still consider it the
+             * implicit backing file.  But it's easier this way, and
+             * just specifying some of the backing BDS's options is
+             * only possible with -drive anyway (otherwise the QAPI
+             * schema forces the user to specify everything). */
+            implicit_backing = !strcmp(bs->auto_backing_file, bs->backing_file);
+        }
+
         bdrv_get_full_backing_filename(bs, backing_filename, PATH_MAX,
                                        &local_err);
         if (local_err) {
@@ -2429,6 +2440,12 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *parent_options,
     }
     bdrv_set_aio_context(backing_hd, bdrv_get_aio_context(bs));
 
+    if (implicit_backing) {
+        bdrv_refresh_filename(backing_hd);
+        pstrcpy(bs->auto_backing_file, sizeof(bs->auto_backing_file),
+                backing_hd->filename);
+    }
+
     /* Hook up the backing file link; drop our reference, bs owns the
      * backing_hd reference now */
     bdrv_set_backing_hd(bs, backing_hd, &local_err);
@@ -3848,6 +3865,8 @@ int bdrv_change_backing_file(BlockDriverState *bs,
     if (ret == 0) {
         pstrcpy(bs->backing_file, sizeof(bs->backing_file), backing_file ?: "");
         pstrcpy(bs->backing_format, sizeof(bs->backing_format), backing_fmt ?: "");
+        pstrcpy(bs->auto_backing_file, sizeof(bs->auto_backing_file),
+                backing_file ?: "");
     }
     return ret;
 }
diff --git a/block/qcow.c b/block/qcow.c
index 0a235bf393..d47515d3df 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -31,6 +31,7 @@
 #include "qemu/module.h"
 #include "qemu/option.h"
 #include "qemu/bswap.h"
+#include "qemu/cutils.h"
 #include <zlib.h>
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
@@ -295,11 +296,13 @@ static int qcow_open(BlockDriverState *bs, QDict *options, int flags,
             goto fail;
         }
         ret = bdrv_pread(bs->file, header.backing_file_offset,
-                   bs->backing_file, len);
+                   bs->auto_backing_file, len);
         if (ret < 0) {
             goto fail;
         }
-        bs->backing_file[len] = '\0';
+        bs->auto_backing_file[len] = '\0';
+        pstrcpy(bs->backing_file, sizeof(bs->backing_file),
+                bs->auto_backing_file);
     }
 
     /* Disable migration when qcow images are used */
diff --git a/block/qcow2.c b/block/qcow2.c
index 65a54c9ac6..3826ce7a39 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1474,13 +1474,15 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
             goto fail;
         }
         ret = bdrv_pread(bs->file, header.backing_file_offset,
-                         bs->backing_file, len);
+                         bs->auto_backing_file, len);
         if (ret < 0) {
             error_setg_errno(errp, -ret, "Could not read backing file name");
             goto fail;
         }
-        bs->backing_file[len] = '\0';
-        s->image_backing_file = g_strdup(bs->backing_file);
+        bs->auto_backing_file[len] = '\0';
+        pstrcpy(bs->backing_file, sizeof(bs->backing_file),
+                bs->auto_backing_file);
+        s->image_backing_file = g_strdup(bs->auto_backing_file);
     }
 
     /* Internal snapshots */
@@ -2518,6 +2520,8 @@ static int qcow2_change_backing_file(BlockDriverState *bs,
         return -EINVAL;
     }
 
+    pstrcpy(bs->auto_backing_file, sizeof(bs->auto_backing_file),
+            backing_file ?: "");
     pstrcpy(bs->backing_file, sizeof(bs->backing_file), backing_file ?: "");
     pstrcpy(bs->backing_format, sizeof(bs->backing_format), backing_fmt ?: "");
 
diff --git a/block/qed.c b/block/qed.c
index 1280870024..81a1bedd41 100644
--- a/block/qed.c
+++ b/block/qed.c
@@ -454,11 +454,14 @@ static int coroutine_fn bdrv_qed_do_open(BlockDriverState *bs, QDict *options,
         }
 
         ret = qed_read_string(bs->file, s->header.backing_filename_offset,
-                              s->header.backing_filename_size, bs->backing_file,
-                              sizeof(bs->backing_file));
+                              s->header.backing_filename_size,
+                              bs->auto_backing_file,
+                              sizeof(bs->auto_backing_file));
         if (ret < 0) {
             return ret;
         }
+        pstrcpy(bs->backing_file, sizeof(bs->backing_file),
+                bs->auto_backing_file);
 
         if (s->header.features & QED_F_BACKING_FORMAT_NO_PROBE) {
             pstrcpy(bs->backing_format, sizeof(bs->backing_format), "raw");
diff --git a/block/vmdk.c b/block/vmdk.c
index 117dc6adfe..464b718352 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -386,12 +386,14 @@ static int vmdk_parent_open(BlockDriverState *bs)
             ret = -EINVAL;
             goto out;
         }
-        if ((end_name - p_name) > sizeof(bs->backing_file) - 1) {
+        if ((end_name - p_name) > sizeof(bs->auto_backing_file) - 1) {
             ret = -EINVAL;
             goto out;
         }
 
-        pstrcpy(bs->backing_file, end_name - p_name + 1, p_name);
+        pstrcpy(bs->auto_backing_file, end_name - p_name + 1, p_name);
+        pstrcpy(bs->backing_file, sizeof(bs->backing_file),
+                bs->auto_backing_file);
     }
 
 out:
-- 
2.20.1

[Qemu-devel] [PULL 31/71] block: Respect backing bs in bdrv_refresh_filename

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Basically, bdrv_refresh_filename() should respect all children of a
BlockDriverState. However, generally those children are driver-specific,
so this function cannot handle the general case. On the other hand,
there are only few drivers which use other children than @file and
@backing (that being vmdk, quorum, and blkverify).

Most block drivers only use @file and/or @backing (if they use any
children at all). Both can be implemented directly in
bdrv_refresh_filename.

The user overriding the file's filename is already handled, however, the
user overriding the backing file is not. If this is done, opening the
BDS with the plain filename of its file will not be correct, so we may
not set bs->exact_filename in that case.

iotest 051 contains test cases for overriding the backing file, and so
its output changes with this patch applied.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-6-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c                       | 38 ++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/051.out    |  8 ++++----
 tests/qemu-iotests/051.pc.out |  8 ++++----
 3 files changed, 45 insertions(+), 9 deletions(-)

diff --git a/block.c b/block.c
index 21395b546c..3b33941c5b 100644
--- a/block.c
+++ b/block.c
@@ -5540,6 +5540,21 @@ static bool append_open_options(QDict *d, BlockDriverState *bs)
     return found_any;
 }
 
+/* Note: This function may return false positives; it may return true
+ * even if opening the backing file specified by bs's image header
+ * would result in exactly bs->backing. */
+static bool bdrv_backing_overridden(BlockDriverState *bs)
+{
+    if (bs->backing) {
+        return strcmp(bs->auto_backing_file,
+                      bs->backing->bs->filename);
+    } else {
+        /* No backing BDS, so if the image header reports any backing
+         * file, it must have been suppressed */
+        return bs->auto_backing_file[0] != '\0';
+    }
+}
+
 /* Updates the following BDS fields:
  *  - exact_filename: A filename which may be used for opening a block device
  *                    which (mostly) equals the given BDS (even without any
@@ -5557,6 +5572,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
     BlockDriver *drv = bs->drv;
     BdrvChild *child;
     QDict *opts;
+    bool backing_overridden;
 
     if (!drv) {
         return;
@@ -5582,6 +5598,16 @@ void bdrv_refresh_filename(BlockDriverState *bs)
         return;
     }
 
+    backing_overridden = bdrv_backing_overridden(bs);
+
+    if (bs->open_flags & BDRV_O_NO_IO) {
+        /* Without I/O, the backing file does not change anything.
+         * Therefore, in such a case (primarily qemu-img), we can
+         * pretend the backing file has not been overridden even if
+         * it technically has been. */
+        backing_overridden = false;
+    }
+
     if (drv->bdrv_refresh_filename) {
         /* Obsolete information is of no use here, so drop the old file name
          * information before refreshing it */
@@ -5607,6 +5633,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
 
         opts = qdict_new();
         has_open_options = append_open_options(opts, bs);
+        has_open_options |= backing_overridden;
 
         /* If no specific options have been given for this BDS, the filename of
          * the underlying file should suffice for this one as well */
@@ -5618,11 +5645,20 @@ void bdrv_refresh_filename(BlockDriverState *bs)
          * file BDS. The full options QDict of that file BDS should somehow
          * contain a representation of the filename, therefore the following
          * suffices without querying the (exact_)filename of this BDS. */
-        if (bs->file->bs->full_open_options) {
+        if (bs->file->bs->full_open_options &&
+            (!bs->backing || bs->backing->bs->full_open_options))
+        {
             qdict_put_str(opts, "driver", drv->format_name);
             qdict_put(opts, "file",
                       qobject_ref(bs->file->bs->full_open_options));
 
+            if (bs->backing) {
+                qdict_put(opts, "backing",
+                          qobject_ref(bs->backing->bs->full_open_options));
+            } else if (backing_overridden) {
+                qdict_put_null(opts, "backing");
+            }
+
             bs->full_open_options = opts;
         } else {
             qobject_unref(opts);
diff --git a/tests/qemu-iotests/051.out b/tests/qemu-iotests/051.out
index 793af2ab96..b900935fbc 100644
--- a/tests/qemu-iotests/051.out
+++ b/tests/qemu-iotests/051.out
@@ -82,7 +82,7 @@ QEMU X.Y.Z monitor - type 'help' for more information
 Testing: -drive file=TEST_DIR/t.qcow2,driver=qcow2,backing.file.filename=TEST_DIR/t.qcow2.orig,if=none,id=drive0 -nodefaults
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) info block
-drive0 (NODE_NAME): TEST_DIR/t.qcow2 (qcow2)
+drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2.orig"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2"}} (qcow2)
     Removable device: not locked, tray closed
     Cache mode:       writeback
     Backing file:     TEST_DIR/t.qcow2.orig (chain depth: 1)
@@ -172,7 +172,7 @@ QEMU_PROG: -drive driver=null-co,cache=invalid_value: invalid cache option
 Testing: -drive file=TEST_DIR/t.qcow2,cache=writeback,backing.file.filename=TEST_DIR/t.qcow2.base,backing.cache.no-flush=on,backing.node-name=backing,backing.file.node-name=backing-file,file.node-name=file,if=none,id=drive0 -nodefaults
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) info block
-drive0 (NODE_NAME): TEST_DIR/t.qcow2 (qcow2)
+drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2.base"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2"}} (qcow2)
     Removable device: not locked, tray closed
     Cache mode:       writeback
     Backing file:     TEST_DIR/t.qcow2.base (chain depth: 1)
@@ -192,7 +192,7 @@ backing-file: TEST_DIR/t.qcow2.base (file, read-only)
 Testing: -drive file=TEST_DIR/t.qcow2,cache=writethrough,backing.file.filename=TEST_DIR/t.qcow2.base,backing.cache.no-flush=on,backing.node-name=backing,backing.file.node-name=backing-file,file.node-name=file,if=none,id=drive0 -nodefaults
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) info block
-drive0 (NODE_NAME): TEST_DIR/t.qcow2 (qcow2)
+drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2.base"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2"}} (qcow2)
     Removable device: not locked, tray closed
     Cache mode:       writethrough
     Backing file:     TEST_DIR/t.qcow2.base (chain depth: 1)
@@ -212,7 +212,7 @@ backing-file: TEST_DIR/t.qcow2.base (file, read-only)
 Testing: -drive file=TEST_DIR/t.qcow2,cache=unsafe,backing.file.filename=TEST_DIR/t.qcow2.base,backing.cache.no-flush=on,backing.node-name=backing,backing.file.node-name=backing-file,file.node-name=file,if=none,id=drive0 -nodefaults
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) info block
-drive0 (NODE_NAME): TEST_DIR/t.qcow2 (qcow2)
+drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2.base"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2"}} (qcow2)
     Removable device: not locked, tray closed
     Cache mode:       writeback, ignore flushes
     Backing file:     TEST_DIR/t.qcow2.base (chain depth: 1)
diff --git a/tests/qemu-iotests/051.pc.out b/tests/qemu-iotests/051.pc.out
index ca64edae6a..8c5c735dfd 100644
--- a/tests/qemu-iotests/051.pc.out
+++ b/tests/qemu-iotests/051.pc.out
@@ -82,7 +82,7 @@ QEMU X.Y.Z monitor - type 'help' for more information
 Testing: -drive file=TEST_DIR/t.qcow2,driver=qcow2,backing.file.filename=TEST_DIR/t.qcow2.orig,if=none,id=drive0 -nodefaults
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) info block
-drive0 (NODE_NAME): TEST_DIR/t.qcow2 (qcow2)
+drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2.orig"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2"}} (qcow2)
     Removable device: not locked, tray closed
     Cache mode:       writeback
     Backing file:     TEST_DIR/t.qcow2.orig (chain depth: 1)
@@ -244,7 +244,7 @@ QEMU_PROG: -drive driver=null-co,cache=invalid_value: invalid cache option
 Testing: -drive file=TEST_DIR/t.qcow2,cache=writeback,backing.file.filename=TEST_DIR/t.qcow2.base,backing.cache.no-flush=on,backing.node-name=backing,backing.file.node-name=backing-file,file.node-name=file,if=none,id=drive0 -nodefaults
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) info block
-drive0 (NODE_NAME): TEST_DIR/t.qcow2 (qcow2)
+drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2.base"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2"}} (qcow2)
     Removable device: not locked, tray closed
     Cache mode:       writeback
     Backing file:     TEST_DIR/t.qcow2.base (chain depth: 1)
@@ -264,7 +264,7 @@ backing-file: TEST_DIR/t.qcow2.base (file, read-only)
 Testing: -drive file=TEST_DIR/t.qcow2,cache=writethrough,backing.file.filename=TEST_DIR/t.qcow2.base,backing.cache.no-flush=on,backing.node-name=backing,backing.file.node-name=backing-file,file.node-name=file,if=none,id=drive0 -nodefaults
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) info block
-drive0 (NODE_NAME): TEST_DIR/t.qcow2 (qcow2)
+drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2.base"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2"}} (qcow2)
     Removable device: not locked, tray closed
     Cache mode:       writethrough
     Backing file:     TEST_DIR/t.qcow2.base (chain depth: 1)
@@ -284,7 +284,7 @@ backing-file: TEST_DIR/t.qcow2.base (file, read-only)
 Testing: -drive file=TEST_DIR/t.qcow2,cache=unsafe,backing.file.filename=TEST_DIR/t.qcow2.base,backing.cache.no-flush=on,backing.node-name=backing,backing.file.node-name=backing-file,file.node-name=file,if=none,id=drive0 -nodefaults
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) info block
-drive0 (NODE_NAME): TEST_DIR/t.qcow2 (qcow2)
+drive0 (NODE_NAME): json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2.base"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/t.qcow2"}} (qcow2)
     Removable device: not locked, tray closed
     Cache mode:       writeback, ignore flushes
     Backing file:     TEST_DIR/t.qcow2.base (chain depth: 1)
-- 
2.20.1

[Qemu-devel] [PULL 32/71] iotests.py: Add filter_imgfmt()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-7-mreitz@redhat.com
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index b461f53abf..d80fd7fc2a 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -304,6 +304,16 @@ def filter_img_info(output, filename):
         lines.append(line)
     return '\n'.join(lines)
 
+def filter_imgfmt(msg):
+    return msg.replace(imgfmt, 'IMGFMT')
+
+def filter_qmp_imgfmt(qmsg):
+    def _filter(key, value):
+        if is_str(value):
+            return filter_imgfmt(value)
+        return value
+    return filter_qmp(qmsg, _filter)
+
 def log(msg, filters=[], indent=None):
     '''Logs either a string message or a JSON serializable message (like QMP).
     If indent is provided, JSON serializable messages are pretty-printed.'''
-- 
2.20.1

[Qemu-devel] [PULL 33/71] iotests.py: Add node_info()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

This function queries a node; since we cannot do that right now, it
executes query-named-block-nodes and returns the matching node's object.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20190201192935.18394-8-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index d80fd7fc2a..82dd096c6e 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -543,6 +543,13 @@ class VM(qtest.QEMUQtestMachine):
                 else:
                     iotests.log(ev)
 
+    def node_info(self, node_name):
+        nodes = self.qmp('query-named-block-nodes')
+        for x in nodes['return']:
+            if x['node-name'] == node_name:
+                return x
+        return None
+
 
 index_re = re.compile(r'([^\[]+)\[([^\]]+)\]')
 
-- 
2.20.1

[Qemu-devel] [PULL 34/71] iotests: Add test for backing file overrides

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-9-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/228     | 242 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/228.out |  84 +++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 327 insertions(+)
 create mode 100755 tests/qemu-iotests/228
 create mode 100644 tests/qemu-iotests/228.out

diff --git a/tests/qemu-iotests/228 b/tests/qemu-iotests/228
new file mode 100755
index 0000000000..2930f8442c
--- /dev/null
+++ b/tests/qemu-iotests/228
@@ -0,0 +1,242 @@
+#!/usr/bin/env python
+#
+# Test for when a backing file is considered overridden (thus, a
+# json:{} filename is generated for the overlay) and when it is not
+#
+# Copyright (C) 2018 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# Creator/Owner: Max Reitz <mreitz@redhat.com>
+
+import iotests
+from iotests import log, qemu_img, filter_testfiles, filter_imgfmt, \
+        filter_qmp_testfiles, filter_qmp_imgfmt
+
+# Need backing file and change-backing-file support
+iotests.verify_image_format(supported_fmts=['qcow2', 'qed'])
+iotests.verify_platform(['linux'])
+
+
+def log_node_info(node):
+    log('')
+
+    log('bs->filename: ' + node['image']['filename'],
+        filters=[filter_testfiles, filter_imgfmt])
+    log('bs->backing_file: ' + node['backing_file'],
+        filters=[filter_testfiles, filter_imgfmt])
+
+    if 'backing-image' in node['image']:
+        log('bs->backing->bs->filename: ' +
+            node['image']['backing-image']['filename'],
+            filters=[filter_testfiles, filter_imgfmt])
+    else:
+        log('bs->backing: (none)')
+
+    log('')
+
+
+with iotests.FilePath('base.img') as base_img_path, \
+     iotests.FilePath('top.img') as top_img_path, \
+     iotests.VM() as vm:
+
+    assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
+    # Choose a funny way to describe the backing filename
+    assert qemu_img('create', '-f', iotests.imgfmt, '-b',
+                    'file:' + base_img_path, top_img_path) == 0
+
+    vm.launch()
+
+    log('--- Implicit backing file ---')
+    log('')
+
+    vm.qmp_log('blockdev-add',
+                node_name='node0',
+                driver=iotests.imgfmt,
+                file={
+                    'driver': 'file',
+                    'filename': top_img_path
+                },
+                filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+
+    # Filename should be plain, and the backing filename should not
+    # contain the "file:" prefix
+    log_node_info(vm.node_info('node0'))
+
+    vm.qmp_log('blockdev-del', node_name='node0')
+
+    log('')
+    log('--- change-backing-file ---')
+    log('')
+
+    vm.qmp_log('blockdev-add',
+               node_name='node0',
+               driver=iotests.imgfmt,
+               file={
+                   'driver': 'file',
+                   'filename': top_img_path
+               },
+               filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+
+    # Changing the backing file to a qemu-reported filename should
+    # result in qemu accepting the corresponding BDS as the implicit
+    # backing BDS (and thus not generate a json:{} filename).
+    # So, first, query the backing filename.
+
+    backing_filename = \
+        vm.node_info('node0')['image']['backing-image']['filename']
+
+    # Next, change the backing file to something different
+
+    vm.qmp_log('change-backing-file',
+               image_node_name='node0',
+               device='node0',
+               backing_file='null-co://',
+               filters=[filter_qmp_testfiles])
+
+    # Now, verify that we get a json:{} filename
+    # (Image header says "null-co://", actual backing file still is
+    # base_img_path)
+
+    log_node_info(vm.node_info('node0'))
+
+    # Change it back
+    # (To get header and backing file in sync)
+
+    vm.qmp_log('change-backing-file',
+               image_node_name='node0',
+               device='node0',
+               backing_file=backing_filename,
+               filters=[filter_qmp_testfiles])
+
+    # And verify that we get our original results
+
+    log_node_info(vm.node_info('node0'))
+
+    # Finally, try a "file:" prefix.  While this is actually what we
+    # originally had in the image header, qemu will not reopen the
+    # backing file here, so it cannot verify that this filename
+    # "resolves" to the actual backing BDS's filename and will thus
+    # consider both to be different.
+    # (This may be fixed in the future.)
+
+    vm.qmp_log('change-backing-file',
+               image_node_name='node0',
+               device='node0',
+               backing_file=('file:' + backing_filename),
+               filters=[filter_qmp_testfiles])
+
+    # So now we should get a json:{} filename
+
+    log_node_info(vm.node_info('node0'))
+
+    # Remove and re-attach so we can see that (as in our first try),
+    # opening the image anew helps qemu resolve the header backing
+    # filename.
+
+    vm.qmp_log('blockdev-del', node_name='node0')
+
+    vm.qmp_log('blockdev-add',
+               node_name='node0',
+               driver=iotests.imgfmt,
+               file={
+                   'driver': 'file',
+                   'filename': top_img_path
+               },
+               filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+
+    log_node_info(vm.node_info('node0'))
+
+    vm.qmp_log('blockdev-del', node_name='node0')
+
+    log('')
+    log('--- Override backing file ---')
+    log('')
+
+    # For this test, we need the plain filename in the image header
+    # (because qemu cannot "canonicalize"/"resolve" the backing
+    # filename unless the backing file is opened implicitly with the
+    # overlay)
+    assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
+                    top_img_path) == 0
+
+    # You can only reliably override backing options by using a node
+    # reference (or by specifying file.filename, but, well...)
+    vm.qmp_log('blockdev-add', node_name='null', driver='null-co')
+
+    vm.qmp_log('blockdev-add',
+               node_name='node0',
+               driver=iotests.imgfmt,
+               file={
+                   'driver': 'file',
+                   'filename': top_img_path
+               },
+               backing='null',
+               filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+
+    # Should get a json:{} filename (and bs->backing_file is
+    # null-co://, because that field actually has not much to do
+    # with the header backing filename (except that it is changed by
+    # change-backing-file))
+
+    log_node_info(vm.node_info('node0'))
+
+    # Detach the backing file by reopening the whole thing
+
+    vm.qmp_log('blockdev-del', node_name='node0')
+    vm.qmp_log('blockdev-del', node_name='null')
+
+    vm.qmp_log('blockdev-add',
+               node_name='node0',
+               driver=iotests.imgfmt,
+               file={
+                   'driver': 'file',
+                   'filename': top_img_path
+               },
+               backing=None,
+               filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+
+    # Should get a json:{} filename (because we overrode the backing
+    # file to not be there)
+
+    log_node_info(vm.node_info('node0'))
+
+    # Open the original backing file
+
+    vm.qmp_log('blockdev-add',
+               node_name='original-backing',
+               driver=iotests.imgfmt,
+               file={
+                   'driver': 'file',
+                   'filename': base_img_path
+               },
+               filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+
+    # Attach the original backing file to its overlay
+
+    vm.qmp_log('blockdev-snapshot',
+               node='original-backing',
+               overlay='node0')
+
+    # This should give us the original plain result
+    # FIXME: Currently, the block layer considers the runtime backing
+    #        file to be different from the image header, which is
+    #        wrong.  This is fixed by a future patch.
+
+    log_node_info(vm.node_info('node0'))
+
+    vm.qmp_log('blockdev-del', node_name='node0')
+    vm.qmp_log('blockdev-del', node_name='original-backing')
+
+    vm.shutdown()
diff --git a/tests/qemu-iotests/228.out b/tests/qemu-iotests/228.out
new file mode 100644
index 0000000000..57fe97d4bc
--- /dev/null
+++ b/tests/qemu-iotests/228.out
@@ -0,0 +1,84 @@
+--- Implicit backing file ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "node0"}}
+{"return": {}}
+
+bs->filename: TEST_DIR/PID-top.img
+bs->backing_file: TEST_DIR/PID-base.img
+bs->backing->bs->filename: TEST_DIR/PID-base.img
+
+{"execute": "blockdev-del", "arguments": {"node-name": "node0"}}
+{"return": {}}
+
+--- change-backing-file ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "node0"}}
+{"return": {}}
+{"execute": "change-backing-file", "arguments": {"backing-file": "null-co://", "device": "node0", "image-node-name": "node0"}}
+{"return": {}}
+
+bs->filename: json:{"backing": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->backing_file: null-co://
+bs->backing->bs->filename: TEST_DIR/PID-base.img
+
+{"execute": "change-backing-file", "arguments": {"backing-file": "TEST_DIR/PID-base.img", "device": "node0", "image-node-name": "node0"}}
+{"return": {}}
+
+bs->filename: TEST_DIR/PID-top.img
+bs->backing_file: TEST_DIR/PID-base.img
+bs->backing->bs->filename: TEST_DIR/PID-base.img
+
+{"execute": "change-backing-file", "arguments": {"backing-file": "file:TEST_DIR/PID-base.img", "device": "node0", "image-node-name": "node0"}}
+{"return": {}}
+
+bs->filename: json:{"backing": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->backing_file: file:TEST_DIR/PID-base.img
+bs->backing->bs->filename: TEST_DIR/PID-base.img
+
+{"execute": "blockdev-del", "arguments": {"node-name": "node0"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "node0"}}
+{"return": {}}
+
+bs->filename: TEST_DIR/PID-top.img
+bs->backing_file: TEST_DIR/PID-base.img
+bs->backing->bs->filename: TEST_DIR/PID-base.img
+
+{"execute": "blockdev-del", "arguments": {"node-name": "node0"}}
+{"return": {}}
+
+--- Override backing file ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "null"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"backing": "null", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "node0"}}
+{"return": {}}
+
+bs->filename: json:{"backing": {"driver": "null-co"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->backing_file: null-co://
+bs->backing->bs->filename: null-co://
+
+{"execute": "blockdev-del", "arguments": {"node-name": "node0"}}
+{"return": {}}
+{"execute": "blockdev-del", "arguments": {"node-name": "null"}}
+{"return": {}}
+{"execute": "blockdev-add", "arguments": {"backing": null, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "node0"}}
+{"return": {}}
+
+bs->filename: json:{"backing": null, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->backing_file: TEST_DIR/PID-base.img
+bs->backing: (none)
+
+{"execute": "blockdev-add", "arguments": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}, "node-name": "original-backing"}}
+{"return": {}}
+{"execute": "blockdev-snapshot", "arguments": {"node": "original-backing", "overlay": "node0"}}
+{"return": {}}
+
+bs->filename: json:{"backing": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->backing_file: TEST_DIR/PID-base.img
+bs->backing->bs->filename: TEST_DIR/PID-base.img
+
+{"execute": "blockdev-del", "arguments": {"node-name": "node0"}}
+{"return": {}}
+{"execute": "blockdev-del", "arguments": {"node-name": "original-backing"}}
+{"return": {}}
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index fc4c416fa3..f701863cdb 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -227,6 +227,7 @@
 225 rw auto quick
 226 auto quick
 227 auto quick
+228 rw auto quick
 229 auto quick
 231 auto quick
 232 auto quick
-- 
2.20.1

[Qemu-devel] [PULL 35/71] block: Make path_combine() return the path

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Besides being safe for arbitrary path lengths, after some follow-up
patches all callers will want a freshly allocated buffer anyway.

In the meantime, path_combine_deprecated() is added which has the same
interface as path_combine() had before this patch. All callers to that
function will be converted in follow-up patches.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 20190201192935.18394-10-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block.h |  4 +-
 block.c               | 85 ++++++++++++++++++++++++-------------------
 block/vmdk.c          |  3 +-
 3 files changed, 49 insertions(+), 43 deletions(-)

diff --git a/include/block/block.h b/include/block/block.h
index aaae900925..e233372a3a 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -496,9 +496,7 @@ void bdrv_get_full_backing_filename_from_filename(const char *backed,
 
 int path_has_protocol(const char *path);
 int path_is_absolute(const char *path);
-void path_combine(char *dest, int dest_size,
-                  const char *base_path,
-                  const char *filename);
+char *path_combine(const char *base_path, const char *filename);
 
 int bdrv_readv_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos);
 int bdrv_writev_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos);
diff --git a/block.c b/block.c
index 3b33941c5b..91859a928d 100644
--- a/block.c
+++ b/block.c
@@ -152,53 +152,62 @@ int path_is_absolute(const char *path)
 #endif
 }
 
-/* if filename is absolute, just copy it to dest. Otherwise, build a
+/* if filename is absolute, just return its duplicate. Otherwise, build a
    path to it by considering it is relative to base_path. URL are
    supported. */
-void path_combine(char *dest, int dest_size,
-                  const char *base_path,
-                  const char *filename)
+char *path_combine(const char *base_path, const char *filename)
 {
+    const char *protocol_stripped = NULL;
     const char *p, *p1;
+    char *result;
     int len;
 
-    if (dest_size <= 0)
-        return;
     if (path_is_absolute(filename)) {
-        pstrcpy(dest, dest_size, filename);
-    } else {
-        const char *protocol_stripped = NULL;
+        return g_strdup(filename);
+    }
 
-        if (path_has_protocol(base_path)) {
-            protocol_stripped = strchr(base_path, ':');
-            if (protocol_stripped) {
-                protocol_stripped++;
-            }
+    if (path_has_protocol(base_path)) {
+        protocol_stripped = strchr(base_path, ':');
+        if (protocol_stripped) {
+            protocol_stripped++;
         }
-        p = protocol_stripped ?: base_path;
+    }
+    p = protocol_stripped ?: base_path;
 
-        p1 = strrchr(base_path, '/');
+    p1 = strrchr(base_path, '/');
 #ifdef _WIN32
-        {
-            const char *p2;
-            p2 = strrchr(base_path, '\\');
-            if (!p1 || p2 > p1)
-                p1 = p2;
+    {
+        const char *p2;
+        p2 = strrchr(base_path, '\\');
+        if (!p1 || p2 > p1) {
+            p1 = p2;
         }
+    }
 #endif
-        if (p1)
-            p1++;
-        else
-            p1 = base_path;
-        if (p1 > p)
-            p = p1;
-        len = p - base_path;
-        if (len > dest_size - 1)
-            len = dest_size - 1;
-        memcpy(dest, base_path, len);
-        dest[len] = '\0';
-        pstrcat(dest, dest_size, filename);
+    if (p1) {
+        p1++;
+    } else {
+        p1 = base_path;
+    }
+    if (p1 > p) {
+        p = p1;
     }
+    len = p - base_path;
+
+    result = g_malloc(len + strlen(filename) + 1);
+    memcpy(result, base_path, len);
+    strcpy(result + len, filename);
+
+    return result;
+}
+
+static void path_combine_deprecated(char *dest, int dest_size,
+                                    const char *base_path,
+                                    const char *filename)
+{
+    char *combined = path_combine(base_path, filename);
+    pstrcpy(dest, dest_size, combined);
+    g_free(combined);
 }
 
 /*
@@ -316,7 +325,7 @@ void bdrv_get_full_backing_filename_from_filename(const char *backed,
         error_setg(errp, "Cannot use relative backing file names for '%s'",
                    backed);
     } else {
-        path_combine(dest, sz, backed, backing);
+        path_combine_deprecated(dest, sz, backed, backing);
     }
 }
 
@@ -4657,8 +4666,8 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
         } else {
             /* If not an absolute filename path, make it relative to the current
              * image's filename path */
-            path_combine(filename_tmp, PATH_MAX, curr_bs->filename,
-                         backing_file);
+            path_combine_deprecated(filename_tmp, PATH_MAX, curr_bs->filename,
+                                    backing_file);
 
             /* We are going to compare absolute pathnames */
             if (!realpath(filename_tmp, filename_full)) {
@@ -4667,8 +4676,8 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
 
             /* We need to make sure the backing filename we are comparing against
              * is relative to the current image filename (or absolute) */
-            path_combine(filename_tmp, PATH_MAX, curr_bs->filename,
-                         curr_bs->backing_file);
+            path_combine_deprecated(filename_tmp, PATH_MAX, curr_bs->filename,
+                                    curr_bs->backing_file);
 
             if (!realpath(filename_tmp, backing_file_full)) {
                 continue;
diff --git a/block/vmdk.c b/block/vmdk.c
index 464b718352..32e4e7589a 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -873,8 +873,7 @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
             return -EINVAL;
         }
 
-        extent_path = g_malloc0(PATH_MAX);
-        path_combine(extent_path, PATH_MAX, desc_file_path, fname);
+        extent_path = path_combine(desc_file_path, fname);
 
         ret = snprintf(extent_opt_prefix, 32, "extents.%d", s->num_extents);
         assert(ret < 32);
-- 
2.20.1

[Qemu-devel] [PULL 36/71] block: bdrv_get_full_backing_filename_from_...'s ret. val.

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Make bdrv_get_full_backing_filename_from_filename() return an allocated
string instead of placing the result in a caller-provided buffer.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-11-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block.h |  7 +++---
 block.c               | 53 ++++++++++++++++++++++++++++++-------------
 block/vmdk.c          | 10 ++++----
 qemu-img.c            | 12 ++++------
 4 files changed, 49 insertions(+), 33 deletions(-)

diff --git a/include/block/block.h b/include/block/block.h
index e233372a3a..5f1650304d 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -489,10 +489,9 @@ void bdrv_get_backing_filename(BlockDriverState *bs,
                                char *filename, int filename_size);
 void bdrv_get_full_backing_filename(BlockDriverState *bs,
                                     char *dest, size_t sz, Error **errp);
-void bdrv_get_full_backing_filename_from_filename(const char *backed,
-                                                  const char *backing,
-                                                  char *dest, size_t sz,
-                                                  Error **errp);
+char *bdrv_get_full_backing_filename_from_filename(const char *backed,
+                                                   const char *backing,
+                                                   Error **errp);
 
 int path_has_protocol(const char *path);
 int path_is_absolute(const char *path);
diff --git a/block.c b/block.c
index 91859a928d..466d7887cc 100644
--- a/block.c
+++ b/block.c
@@ -312,20 +312,29 @@ fail:
     return -EACCES;
 }
 
-void bdrv_get_full_backing_filename_from_filename(const char *backed,
-                                                  const char *backing,
-                                                  char *dest, size_t sz,
-                                                  Error **errp)
+/*
+ * If @backing is empty, this function returns NULL without setting
+ * @errp.  In all other cases, NULL will only be returned with @errp
+ * set.
+ *
+ * Therefore, a return value of NULL without @errp set means that
+ * there is no backing file; if @errp is set, there is one but its
+ * absolute filename cannot be generated.
+ */
+char *bdrv_get_full_backing_filename_from_filename(const char *backed,
+                                                   const char *backing,
+                                                   Error **errp)
 {
-    if (backing[0] == '\0' || path_has_protocol(backing) ||
-        path_is_absolute(backing))
-    {
-        pstrcpy(dest, sz, backing);
+    if (backing[0] == '\0') {
+        return NULL;
+    } else if (path_has_protocol(backing) || path_is_absolute(backing)) {
+        return g_strdup(backing);
     } else if (backed[0] == '\0' || strstart(backed, "json:", NULL)) {
         error_setg(errp, "Cannot use relative backing file names for '%s'",
                    backed);
+        return NULL;
     } else {
-        path_combine_deprecated(dest, sz, backed, backing);
+        return path_combine(backed, backing);
     }
 }
 
@@ -333,12 +342,24 @@ void bdrv_get_full_backing_filename(BlockDriverState *bs, char *dest, size_t sz,
                                     Error **errp)
 {
     char *backed;
+    char *full_name;
+    Error *local_error = NULL;
 
     bdrv_refresh_filename(bs);
 
     backed = bs->exact_filename[0] ? bs->exact_filename : bs->filename;
-    bdrv_get_full_backing_filename_from_filename(backed, bs->backing_file,
-                                                 dest, sz, errp);
+
+    full_name = bdrv_get_full_backing_filename_from_filename(backed,
+                                                             bs->backing_file,
+                                                             &local_error);
+    if (full_name) {
+        pstrcpy(dest, sz, full_name);
+        g_free(full_name);
+    } else if (local_error) {
+        error_propagate(errp, local_error);
+    } else if (sz > 0) {
+        *dest = '\0';
+    }
 }
 
 void bdrv_register(BlockDriver *bdrv)
@@ -5179,17 +5200,17 @@ void bdrv_img_create(const char *filename, const char *fmt,
     size = qemu_opt_get_size(opts, BLOCK_OPT_SIZE, img_size);
     if (backing_file && !(flags & BDRV_O_NO_BACKING)) {
         BlockDriverState *bs;
-        char *full_backing = g_new0(char, PATH_MAX);
+        char *full_backing;
         int back_flags;
         QDict *backing_options = NULL;
 
-        bdrv_get_full_backing_filename_from_filename(filename, backing_file,
-                                                     full_backing, PATH_MAX,
-                                                     &local_err);
+        full_backing =
+            bdrv_get_full_backing_filename_from_filename(filename, backing_file,
+                                                         &local_err);
         if (local_err) {
-            g_free(full_backing);
             goto out;
         }
+        assert(full_backing);
 
         /* backing files always opened read-only */
         back_flags = flags;
diff --git a/block/vmdk.c b/block/vmdk.c
index 32e4e7589a..81c506cb69 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -2078,16 +2078,16 @@ static int coroutine_fn vmdk_co_do_create(int64_t size,
 
     if (backing_file) {
         BlockBackend *backing;
-        char *full_backing = g_new0(char, PATH_MAX);
-        bdrv_get_full_backing_filename_from_filename(blk_bs(blk)->filename, backing_file,
-                                                     full_backing, PATH_MAX,
-                                                     &local_err);
+        char *full_backing =
+            bdrv_get_full_backing_filename_from_filename(blk_bs(blk)->filename,
+                                                         backing_file,
+                                                         &local_err);
         if (local_err) {
-            g_free(full_backing);
             error_propagate(errp, local_err);
             ret = -ENOENT;
             goto exit;
         }
+        assert(full_backing);
 
         backing = blk_new_open(full_backing, NULL, NULL,
                                BDRV_O_NO_BACKING, errp);
diff --git a/qemu-img.c b/qemu-img.c
index d2fc28c987..ae0025926c 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -3343,18 +3343,14 @@ static int img_rebase(int argc, char **argv)
             bdrv_refresh_filename(bs);
             overlay_filename = bs->exact_filename[0] ? bs->exact_filename
                                                      : bs->filename;
-            out_real_path = g_malloc(PATH_MAX);
-
-            bdrv_get_full_backing_filename_from_filename(overlay_filename,
-                                                         out_baseimg,
-                                                         out_real_path,
-                                                         PATH_MAX,
-                                                         &local_err);
+            out_real_path =
+                bdrv_get_full_backing_filename_from_filename(overlay_filename,
+                                                             out_baseimg,
+                                                             &local_err);
             if (local_err) {
                 error_reportf_err(local_err,
                                   "Could not resolve backing filename: ");
                 ret = -1;
-                g_free(out_real_path);
                 goto out;
             }
 
-- 
2.20.1

[Qemu-devel] [PULL 37/71] block: bdrv_get_full_backing_filename's ret. val.

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Make bdrv_get_full_backing_filename() return an allocated string instead
of placing the result in a caller-provided buffer.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-12-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block.h |  3 +--
 block.c               | 48 +++++++++++++++----------------------------
 block/qapi.c          | 12 ++---------
 3 files changed, 20 insertions(+), 43 deletions(-)

diff --git a/include/block/block.h b/include/block/block.h
index 5f1650304d..42cc38f598 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -487,8 +487,7 @@ void bdrv_round_to_clusters(BlockDriverState *bs,
 
 void bdrv_get_backing_filename(BlockDriverState *bs,
                                char *filename, int filename_size);
-void bdrv_get_full_backing_filename(BlockDriverState *bs,
-                                    char *dest, size_t sz, Error **errp);
+char *bdrv_get_full_backing_filename(BlockDriverState *bs, Error **errp);
 char *bdrv_get_full_backing_filename_from_filename(const char *backed,
                                                    const char *backing,
                                                    Error **errp);
diff --git a/block.c b/block.c
index 466d7887cc..a2203f0cfe 100644
--- a/block.c
+++ b/block.c
@@ -338,28 +338,16 @@ char *bdrv_get_full_backing_filename_from_filename(const char *backed,
     }
 }
 
-void bdrv_get_full_backing_filename(BlockDriverState *bs, char *dest, size_t sz,
-                                    Error **errp)
+char *bdrv_get_full_backing_filename(BlockDriverState *bs, Error **errp)
 {
     char *backed;
-    char *full_name;
-    Error *local_error = NULL;
 
     bdrv_refresh_filename(bs);
 
     backed = bs->exact_filename[0] ? bs->exact_filename : bs->filename;
-
-    full_name = bdrv_get_full_backing_filename_from_filename(backed,
-                                                             bs->backing_file,
-                                                             &local_error);
-    if (full_name) {
-        pstrcpy(dest, sz, full_name);
-        g_free(full_name);
-    } else if (local_error) {
-        error_propagate(errp, local_error);
-    } else if (sz > 0) {
-        *dest = '\0';
-    }
+    return bdrv_get_full_backing_filename_from_filename(backed,
+                                                        bs->backing_file,
+                                                        errp);
 }
 
 void bdrv_register(BlockDriver *bdrv)
@@ -2387,7 +2375,7 @@ out:
 int bdrv_open_backing_file(BlockDriverState *bs, QDict *parent_options,
                            const char *bdref_key, Error **errp)
 {
-    char *backing_filename = g_malloc0(PATH_MAX);
+    char *backing_filename = NULL;
     char *bdref_key_dot;
     const char *reference = NULL;
     int ret = 0;
@@ -2422,7 +2410,7 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *parent_options,
      */
     reference = qdict_get_try_str(parent_options, bdref_key);
     if (reference || qdict_haskey(options, "file.filename")) {
-        backing_filename[0] = '\0';
+        /* keep backing_filename NULL */
     } else if (bs->backing_file[0] == '\0' && qdict_size(options) == 0) {
         qobject_unref(options);
         goto free_exit;
@@ -2437,8 +2425,7 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *parent_options,
             implicit_backing = !strcmp(bs->auto_backing_file, bs->backing_file);
         }
 
-        bdrv_get_full_backing_filename(bs, backing_filename, PATH_MAX,
-                                       &local_err);
+        backing_filename = bdrv_get_full_backing_filename(bs, &local_err);
         if (local_err) {
             ret = -EINVAL;
             error_propagate(errp, local_err);
@@ -2459,9 +2446,8 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *parent_options,
         qdict_put_str(options, "driver", bs->backing_format);
     }
 
-    backing_hd = bdrv_open_inherit(*backing_filename ? backing_filename : NULL,
-                                   reference, options, 0, bs, &child_backing,
-                                   errp);
+    backing_hd = bdrv_open_inherit(backing_filename, reference, options, 0, bs,
+                                   &child_backing, errp);
     if (!backing_hd) {
         bs->open_flags |= BDRV_O_NO_BACKING;
         error_prepend(errp, "Could not open backing file: ");
@@ -4648,7 +4634,6 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
     int is_protocol = 0;
     BlockDriverState *curr_bs = NULL;
     BlockDriverState *retval = NULL;
-    Error *local_error = NULL;
 
     if (!bs || !bs->drv || !backing_file) {
         return NULL;
@@ -4668,21 +4653,22 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
         /* If either of the filename paths is actually a protocol, then
          * compare unmodified paths; otherwise make paths relative */
         if (is_protocol || path_has_protocol(curr_bs->backing_file)) {
+            char *backing_file_full_ret;
+
             if (strcmp(backing_file, curr_bs->backing_file) == 0) {
                 retval = curr_bs->backing->bs;
                 break;
             }
             /* Also check against the full backing filename for the image */
-            bdrv_get_full_backing_filename(curr_bs, backing_file_full, PATH_MAX,
-                                           &local_error);
-            if (local_error == NULL) {
-                if (strcmp(backing_file, backing_file_full) == 0) {
+            backing_file_full_ret = bdrv_get_full_backing_filename(curr_bs,
+                                                                   NULL);
+            if (backing_file_full_ret) {
+                bool equal = strcmp(backing_file, backing_file_full_ret) == 0;
+                g_free(backing_file_full_ret);
+                if (equal) {
                     retval = curr_bs->backing->bs;
                     break;
                 }
-            } else {
-                error_free(local_error);
-                local_error = NULL;
             }
         } else {
             /* If not an absolute filename path, make it relative to the current
diff --git a/block/qapi.c b/block/qapi.c
index 4623de1d7b..6002a768f8 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -296,18 +296,10 @@ void bdrv_query_image_info(BlockDriverState *bs,
 
     backing_filename = bs->backing_file;
     if (backing_filename[0] != '\0') {
-        char *backing_filename2 = g_malloc0(PATH_MAX);
+        char *backing_filename2;
         info->backing_filename = g_strdup(backing_filename);
         info->has_backing_filename = true;
-        bdrv_get_full_backing_filename(bs, backing_filename2, PATH_MAX, &err);
-        if (err) {
-            /* Can't reconstruct the full backing filename, so we must omit
-             * this field and apply a Best Effort to this query. */
-            g_free(backing_filename2);
-            backing_filename2 = NULL;
-            error_free(err);
-            err = NULL;
-        }
+        backing_filename2 = bdrv_get_full_backing_filename(bs, NULL);
 
         /* Always report the full_backing_filename if present, even if it's the
          * same as backing_filename. That they are same is useful info. */
-- 
2.20.1

[Qemu-devel] [PULL 38/71] block: Add bdrv_make_absolute_filename()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

This is a general function for making a filename that is relative to a
certain BDS absolute.

It calls bdrv_get_full_backing_filename_from_filename() for now, but
that will be changed in a follow-up patch.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-13-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/block.c b/block.c
index a2203f0cfe..c66cd111e1 100644
--- a/block.c
+++ b/block.c
@@ -338,16 +338,29 @@ char *bdrv_get_full_backing_filename_from_filename(const char *backed,
     }
 }
 
-char *bdrv_get_full_backing_filename(BlockDriverState *bs, Error **errp)
+/*
+ * If @filename is empty or NULL, this function returns NULL without
+ * setting @errp.  In all other cases, NULL will only be returned with
+ * @errp set.
+ */
+static char *bdrv_make_absolute_filename(BlockDriverState *relative_to,
+                                         const char *filename, Error **errp)
 {
-    char *backed;
+    char *bs_filename;
 
-    bdrv_refresh_filename(bs);
+    bdrv_refresh_filename(relative_to);
+
+    bs_filename = relative_to->exact_filename[0]
+                      ? relative_to->exact_filename
+                      : relative_to->filename;
 
-    backed = bs->exact_filename[0] ? bs->exact_filename : bs->filename;
-    return bdrv_get_full_backing_filename_from_filename(backed,
-                                                        bs->backing_file,
-                                                        errp);
+    return bdrv_get_full_backing_filename_from_filename(bs_filename,
+                                                        filename ?: "", errp);
+}
+
+char *bdrv_get_full_backing_filename(BlockDriverState *bs, Error **errp)
+{
+    return bdrv_make_absolute_filename(bs, bs->backing_file, errp);
 }
 
 void bdrv_register(BlockDriver *bdrv)
-- 
2.20.1

[Qemu-devel] [PULL 39/71] block: Fix bdrv_find_backing_image()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

bdrv_find_backing_image() should use bdrv_get_full_backing_filename() or
bdrv_make_absolute_filename() instead of trying to do what those
functions do by itself.

path_combine_deprecated() can now be dropped, so let's do that.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-14-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 33 ++++++++++-----------------------
 1 file changed, 10 insertions(+), 23 deletions(-)

diff --git a/block.c b/block.c
index c66cd111e1..d14ce1c98c 100644
--- a/block.c
+++ b/block.c
@@ -201,15 +201,6 @@ char *path_combine(const char *base_path, const char *filename)
     return result;
 }
 
-static void path_combine_deprecated(char *dest, int dest_size,
-                                    const char *base_path,
-                                    const char *filename)
-{
-    char *combined = path_combine(base_path, filename);
-    pstrcpy(dest, dest_size, combined);
-    g_free(combined);
-}
-
 /*
  * Helper function for bdrv_parse_filename() implementations to remove optional
  * protocol prefixes (especially "file:") from a filename and for putting the
@@ -4654,13 +4645,9 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
 
     filename_full     = g_malloc(PATH_MAX);
     backing_file_full = g_malloc(PATH_MAX);
-    filename_tmp      = g_malloc(PATH_MAX);
 
     is_protocol = path_has_protocol(backing_file);
 
-    /* This will recursively refresh everything in the backing chain */
-    bdrv_refresh_filename(bs);
-
     for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) {
 
         /* If either of the filename paths is actually a protocol, then
@@ -4686,22 +4673,23 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
         } else {
             /* If not an absolute filename path, make it relative to the current
              * image's filename path */
-            path_combine_deprecated(filename_tmp, PATH_MAX, curr_bs->filename,
-                                    backing_file);
-
-            /* We are going to compare absolute pathnames */
-            if (!realpath(filename_tmp, filename_full)) {
+            filename_tmp = bdrv_make_absolute_filename(curr_bs, backing_file,
+                                                       NULL);
+            /* We are going to compare canonicalized absolute pathnames */
+            if (!filename_tmp || !realpath(filename_tmp, filename_full)) {
+                g_free(filename_tmp);
                 continue;
             }
+            g_free(filename_tmp);
 
             /* We need to make sure the backing filename we are comparing against
              * is relative to the current image filename (or absolute) */
-            path_combine_deprecated(filename_tmp, PATH_MAX, curr_bs->filename,
-                                    curr_bs->backing_file);
-
-            if (!realpath(filename_tmp, backing_file_full)) {
+            filename_tmp = bdrv_get_full_backing_filename(curr_bs, NULL);
+            if (!filename_tmp || !realpath(filename_tmp, backing_file_full)) {
+                g_free(filename_tmp);
                 continue;
             }
+            g_free(filename_tmp);
 
             if (strcmp(backing_file_full, filename_full) == 0) {
                 retval = curr_bs->backing->bs;
@@ -4712,7 +4700,6 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
 
     g_free(filename_full);
     g_free(backing_file_full);
-    g_free(filename_tmp);
     return retval;
 }
 
-- 
2.20.1

[Qemu-devel] [PULL 40/71] block: Add bdrv_dirname()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

This function may be implemented by block drivers to derive a directory
name from a BDS. Concatenating this g_free()-able string with a relative
filename must result in a valid (not necessarily existing) filename, so
this is a function that should generally be not implemented by format
drivers, because this is protocol-specific.

If a BDS's driver does not implement this function, bdrv_dirname() will
fall through to the BDS's file if it exists. If it does not, the
exact_filename field will be used to generate a directory name.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-15-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block.h     |  1 +
 include/block/block_int.h |  7 +++++++
 block.c                   | 27 +++++++++++++++++++++++++++
 3 files changed, 35 insertions(+)

diff --git a/include/block/block.h b/include/block/block.h
index 42cc38f598..5b5cf868df 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -491,6 +491,7 @@ char *bdrv_get_full_backing_filename(BlockDriverState *bs, Error **errp);
 char *bdrv_get_full_backing_filename_from_filename(const char *backed,
                                                    const char *backing,
                                                    Error **errp);
+char *bdrv_dirname(BlockDriverState *bs, Error **errp);
 
 int path_has_protocol(const char *path);
 int path_is_absolute(const char *path);
diff --git a/include/block/block_int.h b/include/block/block_int.h
index dd7276cde2..e4d4817ea6 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -141,6 +141,13 @@ struct BlockDriver {
 
     void (*bdrv_refresh_filename)(BlockDriverState *bs, QDict *options);
 
+    /*
+     * Returns an allocated string which is the directory name of this BDS: It
+     * will be used to make relative filenames absolute by prepending this
+     * function's return value to them.
+     */
+    char *(*bdrv_dirname)(BlockDriverState *bs, Error **errp);
+
     /* aio */
     BlockAIOCB *(*bdrv_aio_preadv)(BlockDriverState *bs,
         uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, int flags,
diff --git a/block.c b/block.c
index d14ce1c98c..9f655c0fd6 100644
--- a/block.c
+++ b/block.c
@@ -5715,6 +5715,33 @@ void bdrv_refresh_filename(BlockDriverState *bs)
     }
 }
 
+char *bdrv_dirname(BlockDriverState *bs, Error **errp)
+{
+    BlockDriver *drv = bs->drv;
+
+    if (!drv) {
+        error_setg(errp, "Node '%s' is ejected", bs->node_name);
+        return NULL;
+    }
+
+    if (drv->bdrv_dirname) {
+        return drv->bdrv_dirname(bs, errp);
+    }
+
+    if (bs->file) {
+        return bdrv_dirname(bs->file->bs, errp);
+    }
+
+    bdrv_refresh_filename(bs);
+    if (bs->exact_filename[0] != '\0') {
+        return path_combine(bs->exact_filename, "");
+    }
+
+    error_setg(errp, "Cannot generate a base directory for %s nodes",
+               drv->format_name);
+    return NULL;
+}
+
 /*
  * Hot add/remove a BDS's child. So the user can take a child offline when
  * it is broken and take a new child online
-- 
2.20.1

[Qemu-devel] [PULL 41/71] blkverify: Make bdrv_dirname() return NULL

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

blkverify's BDSs have a file BDS, but we do not want this to be
preferred over the raw node. There is no way to decide between the two
(and not really a reason to, either), so just return NULL in blkverify's
implementation of bdrv_dirname().

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-16-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/blkverify.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/block/blkverify.c b/block/blkverify.c
index 035d77b64a..3c7d4c8729 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -313,6 +313,15 @@ static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
     }
 }
 
+static char *blkverify_dirname(BlockDriverState *bs, Error **errp)
+{
+    /* In general, there are two BDSs with different dirnames below this one;
+     * so there is no unique dirname we could return (unless both are equal by
+     * chance). Therefore, to be consistent, just always return NULL. */
+    error_setg(errp, "Cannot generate a base directory for blkverify nodes");
+    return NULL;
+}
+
 static BlockDriver bdrv_blkverify = {
     .format_name                      = "blkverify",
     .protocol_name                    = "blkverify",
@@ -324,6 +333,7 @@ static BlockDriver bdrv_blkverify = {
     .bdrv_child_perm                  = bdrv_filter_default_perms,
     .bdrv_getlength                   = blkverify_getlength,
     .bdrv_refresh_filename            = blkverify_refresh_filename,
+    .bdrv_dirname                     = blkverify_dirname,
 
     .bdrv_co_preadv                   = blkverify_co_preadv,
     .bdrv_co_pwritev                  = blkverify_co_pwritev,
-- 
2.20.1

[Qemu-devel] [PULL 42/71] quorum: Make bdrv_dirname() return NULL

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

While the common implementation for bdrv_dirname() should return NULL
for quorum BDSs already (because they do not have a file node and their
exact_filename field should be empty), there is no reason not to make
that explicit.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-17-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/quorum.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/block/quorum.c b/block/quorum.c
index cf9d7c16c2..a890f21e85 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -1094,6 +1094,16 @@ static void quorum_refresh_filename(BlockDriverState *bs, QDict *options)
     bs->full_open_options = opts;
 }
 
+static char *quorum_dirname(BlockDriverState *bs, Error **errp)
+{
+    /* In general, there are multiple BDSs with different dirnames below this
+     * one; so there is no unique dirname we could return (unless all are equal
+     * by chance, or there is only one). Therefore, to be consistent, just
+     * always return NULL. */
+    error_setg(errp, "Cannot generate a base directory for quorum nodes");
+    return NULL;
+}
+
 static BlockDriver bdrv_quorum = {
     .format_name                        = "quorum",
 
@@ -1102,6 +1112,7 @@ static BlockDriver bdrv_quorum = {
     .bdrv_open                          = quorum_open,
     .bdrv_close                         = quorum_close,
     .bdrv_refresh_filename              = quorum_refresh_filename,
+    .bdrv_dirname                       = quorum_dirname,
 
     .bdrv_co_flush_to_disk              = quorum_co_flush,
 
-- 
2.20.1

[Qemu-devel] [PULL 43/71] block/nbd: Make bdrv_dirname() return NULL

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

The generic bdrv_dirname() implementation would be able to generate some
form of directory name for many NBD nodes, but it would be always wrong.
Therefore, we have to explicitly make it an error (until NBD has some
form of specification for export paths, if it ever will).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20190201192935.18394-18-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/nbd.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/block/nbd.c b/block/nbd.c
index 9db5eded89..83e6e9e442 100644
--- a/block/nbd.c
+++ b/block/nbd.c
@@ -528,6 +528,16 @@ static void nbd_refresh_filename(BlockDriverState *bs, QDict *options)
     bs->full_open_options = opts;
 }
 
+static char *nbd_dirname(BlockDriverState *bs, Error **errp)
+{
+    /* The generic bdrv_dirname() implementation is able to work out some
+     * directory name for NBD nodes, but that would be wrong. So far there is no
+     * specification for how "export paths" would work, so NBD does not have
+     * directory names. */
+    error_setg(errp, "Cannot generate a base directory for NBD nodes");
+    return NULL;
+}
+
 static BlockDriver bdrv_nbd = {
     .format_name                = "nbd",
     .protocol_name              = "nbd",
@@ -546,6 +556,7 @@ static BlockDriver bdrv_nbd = {
     .bdrv_attach_aio_context    = nbd_attach_aio_context,
     .bdrv_refresh_filename      = nbd_refresh_filename,
     .bdrv_co_block_status       = nbd_client_co_block_status,
+    .bdrv_dirname               = nbd_dirname,
 };
 
 static BlockDriver bdrv_nbd_tcp = {
@@ -566,6 +577,7 @@ static BlockDriver bdrv_nbd_tcp = {
     .bdrv_attach_aio_context    = nbd_attach_aio_context,
     .bdrv_refresh_filename      = nbd_refresh_filename,
     .bdrv_co_block_status       = nbd_client_co_block_status,
+    .bdrv_dirname               = nbd_dirname,
 };
 
 static BlockDriver bdrv_nbd_unix = {
@@ -586,6 +598,7 @@ static BlockDriver bdrv_nbd_unix = {
     .bdrv_attach_aio_context    = nbd_attach_aio_context,
     .bdrv_refresh_filename      = nbd_refresh_filename,
     .bdrv_co_block_status       = nbd_client_co_block_status,
+    .bdrv_dirname               = nbd_dirname,
 };
 
 static void bdrv_nbd_init(void)
-- 
2.20.1

[Qemu-devel] [PULL 44/71] block/nfs: Implement bdrv_dirname()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

While the basic idea is obvious and could be handled by the default
bdrv_dirname() implementation, we cannot generate a directory name if
the gid or uid are set, so we have to explicitly return NULL in those
cases.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-19-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/nfs.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/block/nfs.c b/block/nfs.c
index eab1a2c408..19ee07c321 100644
--- a/block/nfs.c
+++ b/block/nfs.c
@@ -855,6 +855,20 @@ static void nfs_refresh_filename(BlockDriverState *bs, QDict *options)
     bs->full_open_options = opts;
 }
 
+static char *nfs_dirname(BlockDriverState *bs, Error **errp)
+{
+    NFSClient *client = bs->opaque;
+
+    if (client->uid || client->gid) {
+        bdrv_refresh_filename(bs);
+        error_setg(errp, "Cannot generate a base directory for NFS node '%s'",
+                   bs->filename);
+        return NULL;
+    }
+
+    return g_strdup_printf("nfs://%s%s/", client->server->host, client->path);
+}
+
 #ifdef LIBNFS_FEATURE_PAGECACHE
 static void coroutine_fn nfs_co_invalidate_cache(BlockDriverState *bs,
                                                  Error **errp)
@@ -889,6 +903,7 @@ static BlockDriver bdrv_nfs = {
     .bdrv_detach_aio_context        = nfs_detach_aio_context,
     .bdrv_attach_aio_context        = nfs_attach_aio_context,
     .bdrv_refresh_filename          = nfs_refresh_filename,
+    .bdrv_dirname                   = nfs_dirname,
 
 #ifdef LIBNFS_FEATURE_PAGECACHE
     .bdrv_co_invalidate_cache       = nfs_co_invalidate_cache,
-- 
2.20.1

[Qemu-devel] [PULL 45/71] block: Use bdrv_dirname() for relative filenames

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

bdrv_get_full_backing_filename_from_filename() breaks down when it comes
to JSON filenames. Using bdrv_dirname() as the basis is better because
since we have BDS, we can descend through the BDS tree to the protocol
layer, which gives us a greater probability of finding a non-JSON name;
also, bdrv_dirname() is more correct as it allows block drivers to
override the generation of that directory name in a protocol-specific
way.

We still need to keep bdrv_get_full_backing_filename_from_filename(),
though, because it has valid callers which need it during image creation
when no BDS is available yet.

This makes a test case in qemu-iotest 110, which was supposed to fail,
work. That is actually good, but we need to change the reference output
(and the comment in 110) accordingly.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-20-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c                    | 20 +++++++++++++-------
 tests/qemu-iotests/110     |  3 ++-
 tests/qemu-iotests/110.out |  2 +-
 3 files changed, 16 insertions(+), 9 deletions(-)

diff --git a/block.c b/block.c
index 9f655c0fd6..185475ab00 100644
--- a/block.c
+++ b/block.c
@@ -337,16 +337,22 @@ char *bdrv_get_full_backing_filename_from_filename(const char *backed,
 static char *bdrv_make_absolute_filename(BlockDriverState *relative_to,
                                          const char *filename, Error **errp)
 {
-    char *bs_filename;
+    char *dir, *full_name;
 
-    bdrv_refresh_filename(relative_to);
+    if (!filename || filename[0] == '\0') {
+        return NULL;
+    } else if (path_has_protocol(filename) || path_is_absolute(filename)) {
+        return g_strdup(filename);
+    }
 
-    bs_filename = relative_to->exact_filename[0]
-                      ? relative_to->exact_filename
-                      : relative_to->filename;
+    dir = bdrv_dirname(relative_to, errp);
+    if (!dir) {
+        return NULL;
+    }
 
-    return bdrv_get_full_backing_filename_from_filename(bs_filename,
-                                                        filename ?: "", errp);
+    full_name = g_strconcat(dir, filename, NULL);
+    g_free(dir);
+    return full_name;
 }
 
 char *bdrv_get_full_backing_filename(BlockDriverState *bs, Error **errp)
diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
index b64b3b215a..3e9d72d302 100755
--- a/tests/qemu-iotests/110
+++ b/tests/qemu-iotests/110
@@ -60,7 +60,8 @@ echo '=== Non-reconstructable filename ==='
 echo
 
 # Across blkdebug without a config file, you cannot reconstruct filenames, so
-# qemu is incapable of knowing the directory of the top image
+# qemu is incapable of knowing the directory of the top image from the filename
+# alone. However, using bdrv_dirname(), it should still work.
 TEST_IMG="json:{
     'driver': '$IMGFMT',
     'file': {
diff --git a/tests/qemu-iotests/110.out b/tests/qemu-iotests/110.out
index b3584ff87f..5370bc1d26 100644
--- a/tests/qemu-iotests/110.out
+++ b/tests/qemu-iotests/110.out
@@ -14,7 +14,7 @@ backing file: t.IMGFMT.base (actual path: TEST_DIR/t.IMGFMT.base)
 image: json:{"driver": "IMGFMT", "file": {"set-state.0.event": "read_aio", "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "set-state.0.new_state": 42}}
 file format: IMGFMT
 virtual size: 64M (67108864 bytes)
-backing file: t.IMGFMT.base (cannot determine actual path)
+backing file: t.IMGFMT.base (actual path: TEST_DIR/t.IMGFMT.base)
 
 === Backing name is always relative to the backed image ===
 
-- 
2.20.1

[Qemu-devel] [PULL 46/71] iotests: Add quorum case to test 110

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Test 110 tests relative backing filenames for complex BDS trees.  Now
that the originally supposedly failing test passes, let us add a new
failing test: Quorum can never work automatically (without detecting
whether all child nodes have the same base directory, but that would be
rather inconsistent behavior).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-21-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/110     | 26 ++++++++++++++++++++++++++
 tests/qemu-iotests/110.out |  7 +++++++
 2 files changed, 33 insertions(+)

diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
index 3e9d72d302..185ad5437e 100755
--- a/tests/qemu-iotests/110
+++ b/tests/qemu-iotests/110
@@ -29,6 +29,7 @@ status=1	# failure is the default!
 _cleanup()
 {
 	_cleanup_test_img
+        rm -f "$TEST_IMG.copy"
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
 
@@ -86,6 +87,31 @@ echo
 # omit the image size; it should work anyway
 _make_test_img -b "$TEST_IMG_REL.base"
 
+echo
+echo '=== Nodes without a common directory ==='
+echo
+
+cp "$TEST_IMG" "$TEST_IMG.copy"
+
+# Should inform us that the actual path of the backing file cannot be determined
+TEST_IMG="json:{
+    'driver': '$IMGFMT',
+    'file': {
+        'driver': 'quorum',
+        'vote-threshold': 1,
+        'children': [
+            {
+                'driver': 'file',
+                'filename': '$TEST_IMG'
+            },
+            {
+                'driver': 'file',
+                'filename': '$TEST_IMG.copy'
+            }
+        ]
+    }
+}" _img_info | _filter_img_info
+
 
 # success, all done
 echo '*** done'
diff --git a/tests/qemu-iotests/110.out b/tests/qemu-iotests/110.out
index 5370bc1d26..1d0b2475cc 100644
--- a/tests/qemu-iotests/110.out
+++ b/tests/qemu-iotests/110.out
@@ -19,4 +19,11 @@ backing file: t.IMGFMT.base (actual path: TEST_DIR/t.IMGFMT.base)
 === Backing name is always relative to the backed image ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=t.IMGFMT.base
+
+=== Nodes without a common directory ===
+
+image: json:{"driver": "IMGFMT", "file": {"children": [{"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.copy"}], "driver": "quorum", "blkverify": false, "rewrite-corrupted": false, "vote-threshold": 1}}
+file format: IMGFMT
+virtual size: 64M (67108864 bytes)
+backing file: t.IMGFMT.base (cannot determine actual path)
 *** done
-- 
2.20.1

[Qemu-devel] [PULL 47/71] block: Add strong_runtime_opts to BlockDriver

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

This new field can be set by block drivers to list the runtime options
they accept that may influence the contents of the respective BDS. As of
a follow-up patch, this list will be used by the common
bdrv_refresh_filename() implementation to decide which options to put
into BDS.full_open_options (and consequently whether a JSON filename has
to be created), thus freeing the drivers of having to implement that
logic themselves.

Additionally, this patch adds the field to all of the block drivers that
need it and sets it accordingly.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-22-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block_int.h |  7 +++++++
 block/blkdebug.c          | 16 ++++++++++++++++
 block/blklogwrites.c      |  8 ++++++++
 block/crypto.c            |  8 ++++++++
 block/curl.c              | 21 +++++++++++++++++++++
 block/gluster.c           | 19 +++++++++++++++++++
 block/iscsi.c             | 18 ++++++++++++++++++
 block/nbd.c               | 14 ++++++++++++++
 block/nfs.c               | 11 +++++++++++
 block/null.c              |  9 +++++++++
 block/nvme.c              |  8 ++++++++
 block/qcow.c              |  7 +++++++
 block/qcow2.c             |  7 +++++++
 block/quorum.c            | 11 +++++++++++
 block/raw-format.c        | 10 +++++++++-
 block/rbd.c               | 14 ++++++++++++++
 block/replication.c       |  8 ++++++++
 block/sheepdog.c          | 12 ++++++++++++
 block/ssh.c               | 12 ++++++++++++
 block/throttle.c          |  7 +++++++
 block/vpc.c               |  7 +++++++
 block/vvfat.c             | 12 ++++++++++++
 block/vxhs.c              | 11 +++++++++++
 23 files changed, 256 insertions(+), 1 deletion(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index e4d4817ea6..160e8cac1f 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -517,6 +517,13 @@ struct BlockDriver {
     void (*bdrv_register_buf)(BlockDriverState *bs, void *host, size_t size);
     void (*bdrv_unregister_buf)(BlockDriverState *bs, void *host);
     QLIST_ENTRY(BlockDriver) list;
+
+    /* Pointer to a NULL-terminated array of names of strong options
+     * that can be specified for bdrv_open(). A strong option is one
+     * that changes the data of a BDS.
+     * If this pointer is NULL, the array is considered empty.
+     * "filename" and "driver" are always considered strong. */
+    const char *const *strong_runtime_opts;
 };
 
 typedef struct BlockLimits {
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 0759452925..71b4275b98 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -888,6 +888,20 @@ static int blkdebug_reopen_prepare(BDRVReopenState *reopen_state,
     return 0;
 }
 
+static const char *const blkdebug_strong_runtime_opts[] = {
+    "config",
+    "inject-error.",
+    "set-state.",
+    "align",
+    "max-transfer",
+    "opt-write-zero",
+    "max-write-zero",
+    "opt-discard",
+    "max-discard",
+
+    NULL
+};
+
 static BlockDriver bdrv_blkdebug = {
     .format_name            = "blkdebug",
     .protocol_name          = "blkdebug",
@@ -917,6 +931,8 @@ static BlockDriver bdrv_blkdebug = {
                                 = blkdebug_debug_remove_breakpoint,
     .bdrv_debug_resume          = blkdebug_debug_resume,
     .bdrv_debug_is_suspended    = blkdebug_debug_is_suspended,
+
+    .strong_runtime_opts        = blkdebug_strong_runtime_opts,
 };
 
 static void bdrv_blkdebug_init(void)
diff --git a/block/blklogwrites.c b/block/blklogwrites.c
index 36e3d0f822..5da5df112d 100644
--- a/block/blklogwrites.c
+++ b/block/blklogwrites.c
@@ -517,6 +517,13 @@ blk_log_writes_co_pdiscard(BlockDriverState *bs, int64_t offset, int count)
                                  LOG_DISCARD_FLAG, false);
 }
 
+static const char *const blk_log_writes_strong_runtime_opts[] = {
+    "log-append",
+    "log-sector-size",
+
+    NULL
+};
+
 static BlockDriver bdrv_blk_log_writes = {
     .format_name            = "blklogwrites",
     .instance_size          = sizeof(BDRVBlkLogWritesState),
@@ -536,6 +543,7 @@ static BlockDriver bdrv_blk_log_writes = {
     .bdrv_co_block_status   = bdrv_co_block_status_from_file,
 
     .is_filter              = true,
+    .strong_runtime_opts    = blk_log_writes_strong_runtime_opts,
 };
 
 static void bdrv_blk_log_writes_init(void)
diff --git a/block/crypto.c b/block/crypto.c
index d5b1da66a1..fd8c7cfac6 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -619,6 +619,12 @@ block_crypto_get_specific_info_luks(BlockDriverState *bs, Error **errp)
     return spec_info;
 }
 
+static const char *const block_crypto_strong_runtime_opts[] = {
+    BLOCK_CRYPTO_OPT_LUKS_KEY_SECRET,
+
+    NULL
+};
+
 BlockDriver bdrv_crypto_luks = {
     .format_name        = "luks",
     .instance_size      = sizeof(BlockCrypto),
@@ -640,6 +646,8 @@ BlockDriver bdrv_crypto_luks = {
     .bdrv_getlength     = block_crypto_getlength,
     .bdrv_get_info      = block_crypto_get_info_luks,
     .bdrv_get_specific_info = block_crypto_get_specific_info_luks,
+
+    .strong_runtime_opts = block_crypto_strong_runtime_opts,
 };
 
 static void block_crypto_init(void)
diff --git a/block/curl.c b/block/curl.c
index b7ac265d3a..1c9e4f6a64 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -947,6 +947,19 @@ static int64_t curl_getlength(BlockDriverState *bs)
     return s->len;
 }
 
+static const char *const curl_strong_runtime_opts[] = {
+    CURL_BLOCK_OPT_URL,
+    CURL_BLOCK_OPT_SSLVERIFY,
+    CURL_BLOCK_OPT_COOKIE,
+    CURL_BLOCK_OPT_COOKIE_SECRET,
+    CURL_BLOCK_OPT_USERNAME,
+    CURL_BLOCK_OPT_PASSWORD_SECRET,
+    CURL_BLOCK_OPT_PROXY_USERNAME,
+    CURL_BLOCK_OPT_PROXY_PASSWORD_SECRET,
+
+    NULL
+};
+
 static BlockDriver bdrv_http = {
     .format_name                = "http",
     .protocol_name              = "http",
@@ -961,6 +974,8 @@ static BlockDriver bdrv_http = {
 
     .bdrv_detach_aio_context    = curl_detach_aio_context,
     .bdrv_attach_aio_context    = curl_attach_aio_context,
+
+    .strong_runtime_opts        = curl_strong_runtime_opts,
 };
 
 static BlockDriver bdrv_https = {
@@ -977,6 +992,8 @@ static BlockDriver bdrv_https = {
 
     .bdrv_detach_aio_context    = curl_detach_aio_context,
     .bdrv_attach_aio_context    = curl_attach_aio_context,
+
+    .strong_runtime_opts        = curl_strong_runtime_opts,
 };
 
 static BlockDriver bdrv_ftp = {
@@ -993,6 +1010,8 @@ static BlockDriver bdrv_ftp = {
 
     .bdrv_detach_aio_context    = curl_detach_aio_context,
     .bdrv_attach_aio_context    = curl_attach_aio_context,
+
+    .strong_runtime_opts        = curl_strong_runtime_opts,
 };
 
 static BlockDriver bdrv_ftps = {
@@ -1009,6 +1028,8 @@ static BlockDriver bdrv_ftps = {
 
     .bdrv_detach_aio_context    = curl_detach_aio_context,
     .bdrv_attach_aio_context    = curl_attach_aio_context,
+
+    .strong_runtime_opts        = curl_strong_runtime_opts,
 };
 
 static void curl_block_init(void)
diff --git a/block/gluster.c b/block/gluster.c
index 72891060e3..af64330211 100644
--- a/block/gluster.c
+++ b/block/gluster.c
@@ -1495,6 +1495,21 @@ static int coroutine_fn qemu_gluster_co_block_status(BlockDriverState *bs,
 }
 
 
+static const char *const gluster_strong_open_opts[] = {
+    GLUSTER_OPT_VOLUME,
+    GLUSTER_OPT_PATH,
+    GLUSTER_OPT_TYPE,
+    GLUSTER_OPT_SERVER_PATTERN,
+    GLUSTER_OPT_HOST,
+    GLUSTER_OPT_PORT,
+    GLUSTER_OPT_TO,
+    GLUSTER_OPT_IPV4,
+    GLUSTER_OPT_IPV6,
+    GLUSTER_OPT_SOCKET,
+
+    NULL
+};
+
 static BlockDriver bdrv_gluster = {
     .format_name                  = "gluster",
     .protocol_name                = "gluster",
@@ -1522,6 +1537,7 @@ static BlockDriver bdrv_gluster = {
 #endif
     .bdrv_co_block_status         = qemu_gluster_co_block_status,
     .create_opts                  = &qemu_gluster_create_opts,
+    .strong_runtime_opts          = gluster_strong_open_opts,
 };
 
 static BlockDriver bdrv_gluster_tcp = {
@@ -1551,6 +1567,7 @@ static BlockDriver bdrv_gluster_tcp = {
 #endif
     .bdrv_co_block_status         = qemu_gluster_co_block_status,
     .create_opts                  = &qemu_gluster_create_opts,
+    .strong_runtime_opts          = gluster_strong_open_opts,
 };
 
 static BlockDriver bdrv_gluster_unix = {
@@ -1580,6 +1597,7 @@ static BlockDriver bdrv_gluster_unix = {
 #endif
     .bdrv_co_block_status         = qemu_gluster_co_block_status,
     .create_opts                  = &qemu_gluster_create_opts,
+    .strong_runtime_opts          = gluster_strong_open_opts,
 };
 
 /* rdma is deprecated (actually never supported for volfile fetch).
@@ -1615,6 +1633,7 @@ static BlockDriver bdrv_gluster_rdma = {
 #endif
     .bdrv_co_block_status         = qemu_gluster_co_block_status,
     .create_opts                  = &qemu_gluster_create_opts,
+    .strong_runtime_opts          = gluster_strong_open_opts,
 };
 
 static void bdrv_gluster_init(void)
diff --git a/block/iscsi.c b/block/iscsi.c
index ff473206e6..a0c0084837 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -2448,6 +2448,20 @@ static QemuOptsList iscsi_create_opts = {
     }
 };
 
+static const char *const iscsi_strong_runtime_opts[] = {
+    "transport",
+    "portal",
+    "target",
+    "user",
+    "password",
+    "password-secret",
+    "lun",
+    "initiator-name",
+    "header-digest",
+
+    NULL
+};
+
 static BlockDriver bdrv_iscsi = {
     .format_name     = "iscsi",
     .protocol_name   = "iscsi",
@@ -2482,6 +2496,8 @@ static BlockDriver bdrv_iscsi = {
 
     .bdrv_detach_aio_context = iscsi_detach_aio_context,
     .bdrv_attach_aio_context = iscsi_attach_aio_context,
+
+    .strong_runtime_opts = iscsi_strong_runtime_opts,
 };
 
 #if LIBISCSI_API_VERSION >= (20160603)
@@ -2519,6 +2535,8 @@ static BlockDriver bdrv_iser = {
 
     .bdrv_detach_aio_context = iscsi_detach_aio_context,
     .bdrv_attach_aio_context = iscsi_attach_aio_context,
+
+    .strong_runtime_opts = iscsi_strong_runtime_opts,
 };
 #endif
 
diff --git a/block/nbd.c b/block/nbd.c
index 83e6e9e442..318a58776c 100644
--- a/block/nbd.c
+++ b/block/nbd.c
@@ -538,6 +538,17 @@ static char *nbd_dirname(BlockDriverState *bs, Error **errp)
     return NULL;
 }
 
+static const char *const nbd_strong_runtime_opts[] = {
+    "path",
+    "host",
+    "port",
+    "export",
+    "tls-creds",
+    "server.",
+
+    NULL
+};
+
 static BlockDriver bdrv_nbd = {
     .format_name                = "nbd",
     .protocol_name              = "nbd",
@@ -557,6 +568,7 @@ static BlockDriver bdrv_nbd = {
     .bdrv_refresh_filename      = nbd_refresh_filename,
     .bdrv_co_block_status       = nbd_client_co_block_status,
     .bdrv_dirname               = nbd_dirname,
+    .strong_runtime_opts        = nbd_strong_runtime_opts,
 };
 
 static BlockDriver bdrv_nbd_tcp = {
@@ -578,6 +590,7 @@ static BlockDriver bdrv_nbd_tcp = {
     .bdrv_refresh_filename      = nbd_refresh_filename,
     .bdrv_co_block_status       = nbd_client_co_block_status,
     .bdrv_dirname               = nbd_dirname,
+    .strong_runtime_opts        = nbd_strong_runtime_opts,
 };
 
 static BlockDriver bdrv_nbd_unix = {
@@ -599,6 +612,7 @@ static BlockDriver bdrv_nbd_unix = {
     .bdrv_refresh_filename      = nbd_refresh_filename,
     .bdrv_co_block_status       = nbd_client_co_block_status,
     .bdrv_dirname               = nbd_dirname,
+    .strong_runtime_opts        = nbd_strong_runtime_opts,
 };
 
 static void bdrv_nbd_init(void)
diff --git a/block/nfs.c b/block/nfs.c
index 19ee07c321..6985a44b89 100644
--- a/block/nfs.c
+++ b/block/nfs.c
@@ -878,6 +878,15 @@ static void coroutine_fn nfs_co_invalidate_cache(BlockDriverState *bs,
 }
 #endif
 
+static const char *nfs_strong_runtime_opts[] = {
+    "path",
+    "user",
+    "group",
+    "server.",
+
+    NULL
+};
+
 static BlockDriver bdrv_nfs = {
     .format_name                    = "nfs",
     .protocol_name                  = "nfs",
@@ -905,6 +914,8 @@ static BlockDriver bdrv_nfs = {
     .bdrv_refresh_filename          = nfs_refresh_filename,
     .bdrv_dirname                   = nfs_dirname,
 
+    .strong_runtime_opts            = nfs_strong_runtime_opts,
+
 #ifdef LIBNFS_FEATURE_PAGECACHE
     .bdrv_co_invalidate_cache       = nfs_co_invalidate_cache,
 #endif
diff --git a/block/null.c b/block/null.c
index d442d3e901..858892f0c4 100644
--- a/block/null.c
+++ b/block/null.c
@@ -252,6 +252,13 @@ static void null_refresh_filename(BlockDriverState *bs, QDict *opts)
     bs->full_open_options = qobject_ref(opts);
 }
 
+static const char *const null_strong_runtime_opts[] = {
+    BLOCK_OPT_SIZE,
+    NULL_OPT_ZEROES,
+
+    NULL
+};
+
 static BlockDriver bdrv_null_co = {
     .format_name            = "null-co",
     .protocol_name          = "null-co",
@@ -269,6 +276,7 @@ static BlockDriver bdrv_null_co = {
     .bdrv_co_block_status   = null_co_block_status,
 
     .bdrv_refresh_filename  = null_refresh_filename,
+    .strong_runtime_opts    = null_strong_runtime_opts,
 };
 
 static BlockDriver bdrv_null_aio = {
@@ -288,6 +296,7 @@ static BlockDriver bdrv_null_aio = {
     .bdrv_co_block_status   = null_co_block_status,
 
     .bdrv_refresh_filename  = null_refresh_filename,
+    .strong_runtime_opts    = null_strong_runtime_opts,
 };
 
 static void bdrv_null_init(void)
diff --git a/block/nvme.c b/block/nvme.c
index 6c2ce7dfa5..bf656b2bba 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -1136,6 +1136,13 @@ static void nvme_unregister_buf(BlockDriverState *bs, void *host)
     qemu_vfio_dma_unmap(s->vfio, host);
 }
 
+static const char *const nvme_strong_runtime_opts[] = {
+    NVME_BLOCK_OPT_DEVICE,
+    NVME_BLOCK_OPT_NAMESPACE,
+
+    NULL
+};
+
 static BlockDriver bdrv_nvme = {
     .format_name              = "nvme",
     .protocol_name            = "nvme",
@@ -1153,6 +1160,7 @@ static BlockDriver bdrv_nvme = {
 
     .bdrv_refresh_filename    = nvme_refresh_filename,
     .bdrv_refresh_limits      = nvme_refresh_limits,
+    .strong_runtime_opts      = nvme_strong_runtime_opts,
 
     .bdrv_detach_aio_context  = nvme_detach_aio_context,
     .bdrv_attach_aio_context  = nvme_attach_aio_context,
diff --git a/block/qcow.c b/block/qcow.c
index d47515d3df..25d2025fd0 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -1186,6 +1186,12 @@ static QemuOptsList qcow_create_opts = {
     }
 };
 
+static const char *const qcow_strong_runtime_opts[] = {
+    "encrypt." BLOCK_CRYPTO_OPT_QCOW_KEY_SECRET,
+
+    NULL
+};
+
 static BlockDriver bdrv_qcow = {
     .format_name	= "qcow",
     .instance_size	= sizeof(BDRVQcowState),
@@ -1209,6 +1215,7 @@ static BlockDriver bdrv_qcow = {
     .bdrv_get_info          = qcow_get_info,
 
     .create_opts            = &qcow_create_opts,
+    .strong_runtime_opts    = qcow_strong_runtime_opts,
 };
 
 static void bdrv_qcow_init(void)
diff --git a/block/qcow2.c b/block/qcow2.c
index 3826ce7a39..92242fb14f 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -4936,6 +4936,12 @@ static QemuOptsList qcow2_create_opts = {
     }
 };
 
+static const char *const qcow2_strong_runtime_opts[] = {
+    "encrypt." BLOCK_CRYPTO_OPT_QCOW_KEY_SECRET,
+
+    NULL
+};
+
 BlockDriver bdrv_qcow2 = {
     .format_name        = "qcow2",
     .instance_size      = sizeof(BDRVQcow2State),
@@ -4984,6 +4990,7 @@ BlockDriver bdrv_qcow2 = {
     .bdrv_inactivate            = qcow2_inactivate,
 
     .create_opts         = &qcow2_create_opts,
+    .strong_runtime_opts = qcow2_strong_runtime_opts,
     .bdrv_co_check       = qcow2_co_check,
     .bdrv_amend_options  = qcow2_amend_options,
 
diff --git a/block/quorum.c b/block/quorum.c
index a890f21e85..1af6458dc4 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -1104,6 +1104,15 @@ static char *quorum_dirname(BlockDriverState *bs, Error **errp)
     return NULL;
 }
 
+static const char *const quorum_strong_runtime_opts[] = {
+    QUORUM_OPT_VOTE_THRESHOLD,
+    QUORUM_OPT_BLKVERIFY,
+    QUORUM_OPT_REWRITE,
+    QUORUM_OPT_READ_PATTERN,
+
+    NULL
+};
+
 static BlockDriver bdrv_quorum = {
     .format_name                        = "quorum",
 
@@ -1128,6 +1137,8 @@ static BlockDriver bdrv_quorum = {
 
     .is_filter                          = true,
     .bdrv_recurse_is_first_non_filter   = quorum_recurse_is_first_non_filter,
+
+    .strong_runtime_opts                = quorum_strong_runtime_opts,
 };
 
 static void bdrv_quorum_init(void)
diff --git a/block/raw-format.c b/block/raw-format.c
index d07bcdae62..e3e5ba2c8a 100644
--- a/block/raw-format.c
+++ b/block/raw-format.c
@@ -532,6 +532,13 @@ static int coroutine_fn raw_co_copy_range_to(BlockDriverState *bs,
                                  read_flags, write_flags);
 }
 
+static const char *const raw_strong_runtime_opts[] = {
+    "offset",
+    "size",
+
+    NULL
+};
+
 BlockDriver bdrv_raw = {
     .format_name          = "raw",
     .instance_size        = sizeof(BDRVRawState),
@@ -561,7 +568,8 @@ BlockDriver bdrv_raw = {
     .bdrv_lock_medium     = &raw_lock_medium,
     .bdrv_co_ioctl        = &raw_co_ioctl,
     .create_opts          = &raw_create_opts,
-    .bdrv_has_zero_init   = &raw_has_zero_init
+    .bdrv_has_zero_init   = &raw_has_zero_init,
+    .strong_runtime_opts  = raw_strong_runtime_opts,
 };
 
 static void bdrv_raw_init(void)
diff --git a/block/rbd.c b/block/rbd.c
index 8a1a9f4b6e..0c549c9935 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -1228,6 +1228,18 @@ static QemuOptsList qemu_rbd_create_opts = {
     }
 };
 
+static const char *const qemu_rbd_strong_runtime_opts[] = {
+    "pool",
+    "image",
+    "conf",
+    "snapshot",
+    "user",
+    "server.",
+    "password-secret",
+
+    NULL
+};
+
 static BlockDriver bdrv_rbd = {
     .format_name            = "rbd",
     .instance_size          = sizeof(BDRVRBDState),
@@ -1265,6 +1277,8 @@ static BlockDriver bdrv_rbd = {
 #ifdef LIBRBD_SUPPORTS_INVALIDATE
     .bdrv_co_invalidate_cache = qemu_rbd_co_invalidate_cache,
 #endif
+
+    .strong_runtime_opts    = qemu_rbd_strong_runtime_opts,
 };
 
 static void bdrv_rbd_init(void)
diff --git a/block/replication.c b/block/replication.c
index 9b332002ee..4c80b54daf 100644
--- a/block/replication.c
+++ b/block/replication.c
@@ -676,6 +676,13 @@ static void replication_stop(ReplicationState *rs, bool failover, Error **errp)
     aio_context_release(aio_context);
 }
 
+static const char *const replication_strong_runtime_opts[] = {
+    REPLICATION_MODE,
+    REPLICATION_TOP_ID,
+
+    NULL
+};
+
 BlockDriver bdrv_replication = {
     .format_name                = "replication",
     .instance_size              = sizeof(BDRVReplicationState),
@@ -692,6 +699,7 @@ BlockDriver bdrv_replication = {
     .bdrv_recurse_is_first_non_filter = replication_recurse_is_first_non_filter,
 
     .has_variable_length        = true,
+    .strong_runtime_opts        = replication_strong_runtime_opts,
 };
 
 static void bdrv_replication_init(void)
diff --git a/block/sheepdog.c b/block/sheepdog.c
index b916ba07bf..cbdfe9ab6e 100644
--- a/block/sheepdog.c
+++ b/block/sheepdog.c
@@ -3203,6 +3203,15 @@ static QemuOptsList sd_create_opts = {
     }
 };
 
+static const char *const sd_strong_runtime_opts[] = {
+    "vdi",
+    "snap-id",
+    "tag",
+    "server.",
+
+    NULL
+};
+
 static BlockDriver bdrv_sheepdog = {
     .format_name                  = "sheepdog",
     .protocol_name                = "sheepdog",
@@ -3238,6 +3247,7 @@ static BlockDriver bdrv_sheepdog = {
     .bdrv_attach_aio_context      = sd_attach_aio_context,
 
     .create_opts                  = &sd_create_opts,
+    .strong_runtime_opts          = sd_strong_runtime_opts,
 };
 
 static BlockDriver bdrv_sheepdog_tcp = {
@@ -3275,6 +3285,7 @@ static BlockDriver bdrv_sheepdog_tcp = {
     .bdrv_attach_aio_context      = sd_attach_aio_context,
 
     .create_opts                  = &sd_create_opts,
+    .strong_runtime_opts          = sd_strong_runtime_opts,
 };
 
 static BlockDriver bdrv_sheepdog_unix = {
@@ -3312,6 +3323,7 @@ static BlockDriver bdrv_sheepdog_unix = {
     .bdrv_attach_aio_context      = sd_attach_aio_context,
 
     .create_opts                  = &sd_create_opts,
+    .strong_runtime_opts          = sd_strong_runtime_opts,
 };
 
 static void bdrv_sheepdog_init(void)
diff --git a/block/ssh.c b/block/ssh.c
index bbc513e095..190ef95300 100644
--- a/block/ssh.c
+++ b/block/ssh.c
@@ -1254,6 +1254,17 @@ static int coroutine_fn ssh_co_truncate(BlockDriverState *bs, int64_t offset,
     return ssh_grow_file(s, offset, errp);
 }
 
+static const char *const ssh_strong_runtime_opts[] = {
+    "host",
+    "port",
+    "path",
+    "user",
+    "host_key_check",
+    "server.",
+
+    NULL
+};
+
 static BlockDriver bdrv_ssh = {
     .format_name                  = "ssh",
     .protocol_name                = "ssh",
@@ -1270,6 +1281,7 @@ static BlockDriver bdrv_ssh = {
     .bdrv_co_truncate             = ssh_co_truncate,
     .bdrv_co_flush_to_disk        = ssh_co_flush,
     .create_opts                  = &ssh_create_opts,
+    .strong_runtime_opts          = ssh_strong_runtime_opts,
 };
 
 static void bdrv_ssh_init(void)
diff --git a/block/throttle.c b/block/throttle.c
index 636c9764aa..f64dcc27b9 100644
--- a/block/throttle.c
+++ b/block/throttle.c
@@ -227,6 +227,12 @@ static void coroutine_fn throttle_co_drain_end(BlockDriverState *bs)
     atomic_dec(&tgm->io_limits_disabled);
 }
 
+static const char *const throttle_strong_runtime_opts[] = {
+    QEMU_OPT_THROTTLE_GROUP_NAME,
+
+    NULL
+};
+
 static BlockDriver bdrv_throttle = {
     .format_name                        =   "throttle",
     .instance_size                      =   sizeof(ThrottleGroupMember),
@@ -259,6 +265,7 @@ static BlockDriver bdrv_throttle = {
     .bdrv_co_drain_end                  =   throttle_co_drain_end,
 
     .is_filter                          =   true,
+    .strong_runtime_opts                =   throttle_strong_runtime_opts,
 };
 
 static void bdrv_throttle_init(void)
diff --git a/block/vpc.c b/block/vpc.c
index 52ab717642..a902a4c54d 100644
--- a/block/vpc.c
+++ b/block/vpc.c
@@ -1218,6 +1218,12 @@ static QemuOptsList vpc_create_opts = {
     }
 };
 
+static const char *const vpc_strong_runtime_opts[] = {
+    VPC_OPT_SIZE_CALC,
+
+    NULL
+};
+
 static BlockDriver bdrv_vpc = {
     .format_name    = "vpc",
     .instance_size  = sizeof(BDRVVPCState),
@@ -1238,6 +1244,7 @@ static BlockDriver bdrv_vpc = {
 
     .create_opts            = &vpc_create_opts,
     .bdrv_has_zero_init     = vpc_has_zero_init,
+    .strong_runtime_opts    = vpc_strong_runtime_opts,
 };
 
 static void bdrv_vpc_init(void)
diff --git a/block/vvfat.c b/block/vvfat.c
index b7b61ea8b7..5f66787890 100644
--- a/block/vvfat.c
+++ b/block/vvfat.c
@@ -3253,6 +3253,16 @@ static void vvfat_close(BlockDriverState *bs)
     }
 }
 
+static const char *const vvfat_strong_runtime_opts[] = {
+    "dir",
+    "fat-type",
+    "floppy",
+    "label",
+    "rw",
+
+    NULL
+};
+
 static BlockDriver bdrv_vvfat = {
     .format_name            = "vvfat",
     .protocol_name          = "fat",
@@ -3267,6 +3277,8 @@ static BlockDriver bdrv_vvfat = {
     .bdrv_co_preadv         = vvfat_co_preadv,
     .bdrv_co_pwritev        = vvfat_co_pwritev,
     .bdrv_co_block_status   = vvfat_co_block_status,
+
+    .strong_runtime_opts    = vvfat_strong_runtime_opts,
 };
 
 static void bdrv_vvfat_init(void)
diff --git a/block/vxhs.c b/block/vxhs.c
index 0cb0a007e9..2e18229ba4 100644
--- a/block/vxhs.c
+++ b/block/vxhs.c
@@ -556,6 +556,16 @@ static int64_t vxhs_getlength(BlockDriverState *bs)
     return vdisk_size;
 }
 
+static const char *const vxhs_strong_runtime_opts[] = {
+    VXHS_OPT_VDISK_ID,
+    "tls-creds",
+    VXHS_OPT_HOST,
+    VXHS_OPT_PORT,
+    VXHS_OPT_SERVER".",
+
+    NULL
+};
+
 static BlockDriver bdrv_vxhs = {
     .format_name                  = "vxhs",
     .protocol_name                = "vxhs",
@@ -567,6 +577,7 @@ static BlockDriver bdrv_vxhs = {
     .bdrv_getlength               = vxhs_getlength,
     .bdrv_aio_preadv              = vxhs_aio_preadv,
     .bdrv_aio_pwritev             = vxhs_aio_pwritev,
+    .strong_runtime_opts          = vxhs_strong_runtime_opts,
 };
 
 static void bdrv_vxhs_init(void)
-- 
2.20.1

[Qemu-devel] [PULL 48/71] block: Add BlockDriver.bdrv_gather_child_options

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Some follow-up patches will rework the way bs->full_open_options is
refreshed in bdrv_refresh_filename(). The new implementation will remove
the need for the block drivers' bdrv_refresh_filename() implementations
to set bs->full_open_options; instead, it will be generic and use static
information from each block driver.

However, by implementing bdrv_gather_child_options(), block drivers will
still be able to override the way the full_open_options of their
children are incorporated into their own.

We need to implement this function for VMDK because we have to prevent
the generic implementation from gathering the options of all children:
It is not possible to specify options for the extents through the
runtime options.

For quorum, the child names that would be used by the generic
implementation and the ones that we actually (currently) want to use
differ. See quorum_gather_child_options() for more information.

Note that both of these are cases which are not ideal: In case of VMDK
it would probably be nice to be able to specify options for all extents.
In case of quorum, the current runtime option structure is simply broken
and needs to be fixed (but that is left for another patch).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-23-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block_int.h | 24 +++++++++++++++++++++++
 block/quorum.c            | 40 +++++++++++++++++++++++++++++++++++++++
 block/vmdk.c              | 19 +++++++++++++++++++
 3 files changed, 83 insertions(+)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index 160e8cac1f..ab4cf2df07 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -141,6 +141,30 @@ struct BlockDriver {
 
     void (*bdrv_refresh_filename)(BlockDriverState *bs, QDict *options);
 
+    /*
+     * Gathers the open options for all children into @target.
+     * A simple format driver (without backing file support) might
+     * implement this function like this:
+     *
+     *     QINCREF(bs->file->bs->full_open_options);
+     *     qdict_put(target, "file", bs->file->bs->full_open_options);
+     *
+     * If not specified, the generic implementation will simply put
+     * all children's options under their respective name.
+     *
+     * @backing_overridden is true when bs->backing seems not to be
+     * the child that would result from opening bs->backing_file.
+     * Therefore, if it is true, the backing child's options should be
+     * gathered; otherwise, there is no need since the backing child
+     * is the one implied by the image header.
+     *
+     * Note that ideally this function would not be needed.  Every
+     * block driver which implements it is probably doing something
+     * shady regarding its runtime option structure.
+     */
+    void (*bdrv_gather_child_options)(BlockDriverState *bs, QDict *target,
+                                      bool backing_overridden);
+
     /*
      * Returns an allocated string which is the directory name of this BDS: It
      * will be used to make relative filenames absolute by prepending this
diff --git a/block/quorum.c b/block/quorum.c
index 1af6458dc4..3984f0aa4f 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -1094,6 +1094,45 @@ static void quorum_refresh_filename(BlockDriverState *bs, QDict *options)
     bs->full_open_options = opts;
 }
 
+static void quorum_gather_child_options(BlockDriverState *bs, QDict *target,
+                                        bool backing_overridden)
+{
+    BDRVQuorumState *s = bs->opaque;
+    QList *children_list;
+    int i;
+
+    /*
+     * The generic implementation for gathering child options in
+     * bdrv_refresh_filename() would use the names of the children
+     * as specified for bdrv_open_child() or bdrv_attach_child(),
+     * which is "children.%u" with %u being a value
+     * (s->next_child_index) that is incremented each time a new child
+     * is added (and never decremented).  Since children can be
+     * deleted at runtime, there may be gaps in that enumeration.
+     * When creating a new quorum BDS and specifying the children for
+     * it through runtime options, the enumeration used there may not
+     * have any gaps, though.
+     *
+     * Therefore, we have to create a new gap-less enumeration here
+     * (which we can achieve by simply putting all of the children's
+     * full_open_options into a QList).
+     *
+     * XXX: Note that there are issues with the current child option
+     *      structure quorum uses (such as the fact that children do
+     *      not really have unique permanent names).  Therefore, this
+     *      is going to have to change in the future and ideally we
+     *      want quorum to be covered by the generic implementation.
+     */
+
+    children_list = qlist_new();
+    qdict_put(target, "children", children_list);
+
+    for (i = 0; i < s->num_children; i++) {
+        qlist_append(children_list,
+                     qobject_ref(s->children[i]->bs->full_open_options));
+    }
+}
+
 static char *quorum_dirname(BlockDriverState *bs, Error **errp)
 {
     /* In general, there are multiple BDSs with different dirnames below this
@@ -1121,6 +1160,7 @@ static BlockDriver bdrv_quorum = {
     .bdrv_open                          = quorum_open,
     .bdrv_close                         = quorum_close,
     .bdrv_refresh_filename              = quorum_refresh_filename,
+    .bdrv_gather_child_options          = quorum_gather_child_options,
     .bdrv_dirname                       = quorum_dirname,
 
     .bdrv_co_flush_to_disk              = quorum_co_flush,
diff --git a/block/vmdk.c b/block/vmdk.c
index 81c506cb69..91345babb5 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -27,6 +27,7 @@
 #include "qapi/error.h"
 #include "block/block_int.h"
 #include "sysemu/block-backend.h"
+#include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qerror.h"
 #include "qemu/error-report.h"
 #include "qemu/module.h"
@@ -2608,6 +2609,23 @@ static int vmdk_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
     return 0;
 }
 
+static void vmdk_gather_child_options(BlockDriverState *bs, QDict *target,
+                                      bool backing_overridden)
+{
+    /* No children but file and backing can be explicitly specified (TODO) */
+    qdict_put(target, "file",
+              qobject_ref(bs->file->bs->full_open_options));
+
+    if (backing_overridden) {
+        if (bs->backing) {
+            qdict_put(target, "backing",
+                      qobject_ref(bs->backing->bs->full_open_options));
+        } else {
+            qdict_put_null(target, "backing");
+        }
+    }
+}
+
 static QemuOptsList vmdk_create_opts = {
     .name = "vmdk-create-opts",
     .head = QTAILQ_HEAD_INITIALIZER(vmdk_create_opts.head),
@@ -2679,6 +2697,7 @@ static BlockDriver bdrv_vmdk = {
     .bdrv_get_specific_info       = vmdk_get_specific_info,
     .bdrv_refresh_limits          = vmdk_refresh_limits,
     .bdrv_get_info                = vmdk_get_info,
+    .bdrv_gather_child_options    = vmdk_gather_child_options,
 
     .supports_backing             = true,
     .create_opts                  = &vmdk_create_opts,
-- 
2.20.1

[Qemu-devel] [PULL 49/71] block: Generically refresh runtime options

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Instead of having every block driver which implements
bdrv_refresh_filename() copy all of the strong runtime options over to
bs->full_open_options, implement this process generically in
bdrv_refresh_filename().

This patch only adds this new generic implementation, it does not remove
the old functionality. This is done in a follow-up patch.

With this patch, some superfluous information (that should never have
been there) may be removed from some JSON filenames, as can be seen in
the change to iotests 110's and 228's reference outputs.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-24-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c                    | 116 ++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/110.out |   2 +-
 tests/qemu-iotests/228     |   7 ++-
 tests/qemu-iotests/228.out |   2 +-
 4 files changed, 121 insertions(+), 6 deletions(-)

diff --git a/block.c b/block.c
index 185475ab00..a6e523059d 100644
--- a/block.c
+++ b/block.c
@@ -5535,6 +5535,92 @@ out:
     return to_replace_bs;
 }
 
+/**
+ * Iterates through the list of runtime option keys that are said to
+ * be "strong" for a BDS.  An option is called "strong" if it changes
+ * a BDS's data.  For example, the null block driver's "size" and
+ * "read-zeroes" options are strong, but its "latency-ns" option is
+ * not.
+ *
+ * If a key returned by this function ends with a dot, all options
+ * starting with that prefix are strong.
+ */
+static const char *const *strong_options(BlockDriverState *bs,
+                                         const char *const *curopt)
+{
+    static const char *const global_options[] = {
+        "driver", "filename", NULL
+    };
+
+    if (!curopt) {
+        return &global_options[0];
+    }
+
+    curopt++;
+    if (curopt == &global_options[ARRAY_SIZE(global_options) - 1] && bs->drv) {
+        curopt = bs->drv->strong_runtime_opts;
+    }
+
+    return (curopt && *curopt) ? curopt : NULL;
+}
+
+/**
+ * Copies all strong runtime options from bs->options to the given
+ * QDict.  The set of strong option keys is determined by invoking
+ * strong_options().
+ *
+ * Returns true iff any strong option was present in bs->options (and
+ * thus copied to the target QDict) with the exception of "filename"
+ * and "driver".  The caller is expected to use this value to decide
+ * whether the existence of strong options prevents the generation of
+ * a plain filename.
+ */
+static bool append_strong_runtime_options(QDict *d, BlockDriverState *bs)
+{
+    bool found_any = false;
+    const char *const *option_name = NULL;
+
+    if (!bs->drv) {
+        return false;
+    }
+
+    while ((option_name = strong_options(bs, option_name))) {
+        bool option_given = false;
+
+        assert(strlen(*option_name) > 0);
+        if ((*option_name)[strlen(*option_name) - 1] != '.') {
+            QObject *entry = qdict_get(bs->options, *option_name);
+            if (!entry) {
+                continue;
+            }
+
+            qdict_put_obj(d, *option_name, qobject_ref(entry));
+            option_given = true;
+        } else {
+            const QDictEntry *entry;
+            for (entry = qdict_first(bs->options); entry;
+                 entry = qdict_next(bs->options, entry))
+            {
+                if (strstart(qdict_entry_key(entry), *option_name, NULL)) {
+                    qdict_put_obj(d, qdict_entry_key(entry),
+                                  qobject_ref(qdict_entry_value(entry)));
+                    option_given = true;
+                }
+            }
+        }
+
+        /* While "driver" and "filename" need to be included in a JSON filename,
+         * their existence does not prohibit generation of a plain filename. */
+        if (!found_any && option_given &&
+            strcmp(*option_name, "driver") && strcmp(*option_name, "filename"))
+        {
+            found_any = true;
+        }
+    }
+
+    return found_any;
+}
+
 static bool append_open_options(QDict *d, BlockDriverState *bs)
 {
     const QDictEntry *entry;
@@ -5711,9 +5797,37 @@ void bdrv_refresh_filename(BlockDriverState *bs)
         bs->full_open_options = opts;
     }
 
+    /* Gather the options QDict */
+    opts = qdict_new();
+    append_strong_runtime_options(opts, bs);
+
+    if (drv->bdrv_gather_child_options) {
+        /* Some block drivers may not want to present all of their children's
+         * options, or name them differently from BdrvChild.name */
+        drv->bdrv_gather_child_options(bs, opts, backing_overridden);
+    } else {
+        QLIST_FOREACH(child, &bs->children, next) {
+            if (child->role == &child_backing && !backing_overridden) {
+                /* We can skip the backing BDS if it has not been overridden */
+                continue;
+            }
+
+            qdict_put(opts, child->name,
+                      qobject_ref(child->bs->full_open_options));
+        }
+
+        if (backing_overridden && !bs->backing) {
+            /* Force no backing file */
+            qdict_put_null(opts, "backing");
+        }
+    }
+
+    qobject_unref(bs->full_open_options);
+    bs->full_open_options = opts;
+
     if (bs->exact_filename[0]) {
         pstrcpy(bs->filename, sizeof(bs->filename), bs->exact_filename);
-    } else if (bs->full_open_options) {
+    } else {
         QString *json = qobject_to_json(QOBJECT(bs->full_open_options));
         snprintf(bs->filename, sizeof(bs->filename), "json:%s",
                  qstring_get_str(json));
diff --git a/tests/qemu-iotests/110.out b/tests/qemu-iotests/110.out
index 1d0b2475cc..46e6a60510 100644
--- a/tests/qemu-iotests/110.out
+++ b/tests/qemu-iotests/110.out
@@ -22,7 +22,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=t.IMGFMT.b
 
 === Nodes without a common directory ===
 
-image: json:{"driver": "IMGFMT", "file": {"children": [{"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.copy"}], "driver": "quorum", "blkverify": false, "rewrite-corrupted": false, "vote-threshold": 1}}
+image: json:{"driver": "IMGFMT", "file": {"children": [{"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.copy"}], "driver": "quorum", "vote-threshold": 1}}
 file format: IMGFMT
 virtual size: 64M (67108864 bytes)
 backing file: t.IMGFMT.base (cannot determine actual path)
diff --git a/tests/qemu-iotests/228 b/tests/qemu-iotests/228
index 2930f8442c..7a57c41e29 100755
--- a/tests/qemu-iotests/228
+++ b/tests/qemu-iotests/228
@@ -230,9 +230,10 @@ with iotests.FilePath('base.img') as base_img_path, \
                overlay='node0')
 
     # This should give us the original plain result
-    # FIXME: Currently, the block layer considers the runtime backing
-    #        file to be different from the image header, which is
-    #        wrong.  This is fixed by a future patch.
+    # FIXME: Currently, it yields a json:{} filename even though it
+    #        only contains a @driver and a @file entry, so a plain
+    #        filename would obviously suffice.  This is fixed by a
+    #        future patch.
 
     log_node_info(vm.node_info('node0'))
 
diff --git a/tests/qemu-iotests/228.out b/tests/qemu-iotests/228.out
index 57fe97d4bc..393ab09164 100644
--- a/tests/qemu-iotests/228.out
+++ b/tests/qemu-iotests/228.out
@@ -74,7 +74,7 @@ bs->backing: (none)
 {"execute": "blockdev-snapshot", "arguments": {"node": "original-backing", "overlay": "node0"}}
 {"return": {}}
 
-bs->filename: json:{"backing": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->filename: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
 bs->backing_file: TEST_DIR/PID-base.img
 bs->backing->bs->filename: TEST_DIR/PID-base.img
 
-- 
2.20.1

[Qemu-devel] [PULL 50/71] block: Purify .bdrv_refresh_filename()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Currently, BlockDriver.bdrv_refresh_filename() is supposed to both
refresh the filename (BDS.exact_filename) and set BDS.full_open_options.
Now that we have generic code in the central bdrv_refresh_filename() for
creating BDS.full_open_options, we can drop the latter part from all
BlockDriver.bdrv_refresh_filename() implementations.

This also means that we can drop all of the existing default code for
this from the global bdrv_refresh_filename() itself.

Furthermore, we now have to call BlockDriver.bdrv_refresh_filename()
after having set BDS.full_open_options, because the block driver's
implementation should now be allowed to depend on BDS.full_open_options
being set correctly.

Finally, with this patch we can drop the @options parameter from
BlockDriver.bdrv_refresh_filename(); also, add a comment on this
function's purpose in block/block_int.h while touching its interface.

This completely obsoletes blklogwrite's implementation of
.bdrv_refresh_filename().

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-25-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block_int.h  |   6 +-
 block.c                    | 131 +++++++------------------------------
 block/blkdebug.c           |  54 ++++++---------
 block/blklogwrites.c       |  22 -------
 block/blkverify.c          |  16 +----
 block/commit.c             |   2 +-
 block/mirror.c             |   2 +-
 block/nbd.c                |  23 +------
 block/nfs.c                |  36 +---------
 block/null.c               |  22 ++++---
 block/nvme.c               |  22 ++++---
 block/quorum.c             |  30 ---------
 tests/qemu-iotests/228     |   4 --
 tests/qemu-iotests/228.out |   2 +-
 14 files changed, 81 insertions(+), 291 deletions(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index ab4cf2df07..836d67c1ae 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -139,7 +139,11 @@ struct BlockDriver {
                                             Error **errp);
     int (*bdrv_make_empty)(BlockDriverState *bs);
 
-    void (*bdrv_refresh_filename)(BlockDriverState *bs, QDict *options);
+    /*
+     * Refreshes the bs->exact_filename field. If that is impossible,
+     * bs->exact_filename has to be left empty.
+     */
+    void (*bdrv_refresh_filename)(BlockDriverState *bs);
 
     /*
      * Gathers the open options for all children into @target.
diff --git a/block.c b/block.c
index a6e523059d..23869623ea 100644
--- a/block.c
+++ b/block.c
@@ -5621,33 +5621,6 @@ static bool append_strong_runtime_options(QDict *d, BlockDriverState *bs)
     return found_any;
 }
 
-static bool append_open_options(QDict *d, BlockDriverState *bs)
-{
-    const QDictEntry *entry;
-    QemuOptDesc *desc;
-    bool found_any = false;
-
-    for (entry = qdict_first(bs->options); entry;
-         entry = qdict_next(bs->options, entry))
-    {
-        /* Exclude all non-driver-specific options */
-        for (desc = bdrv_runtime_opts.desc; desc->name; desc++) {
-            if (!strcmp(qdict_entry_key(entry), desc->name)) {
-                break;
-            }
-        }
-        if (desc->name) {
-            continue;
-        }
-
-        qdict_put_obj(d, qdict_entry_key(entry),
-                      qobject_ref(qdict_entry_value(entry)));
-        found_any = true;
-    }
-
-    return found_any;
-}
-
 /* Note: This function may return false positives; it may return true
  * even if opening the backing file specified by bs's image header
  * would result in exactly bs->backing. */
@@ -5681,6 +5654,8 @@ void bdrv_refresh_filename(BlockDriverState *bs)
     BdrvChild *child;
     QDict *opts;
     bool backing_overridden;
+    bool generate_json_filename; /* Whether our default implementation should
+                                    fill exact_filename (false) or not (true) */
 
     if (!drv) {
         return;
@@ -5716,90 +5691,10 @@ void bdrv_refresh_filename(BlockDriverState *bs)
         backing_overridden = false;
     }
 
-    if (drv->bdrv_refresh_filename) {
-        /* Obsolete information is of no use here, so drop the old file name
-         * information before refreshing it */
-        bs->exact_filename[0] = '\0';
-        if (bs->full_open_options) {
-            qobject_unref(bs->full_open_options);
-            bs->full_open_options = NULL;
-        }
-
-        opts = qdict_new();
-        append_open_options(opts, bs);
-        drv->bdrv_refresh_filename(bs, opts);
-        qobject_unref(opts);
-    } else if (bs->file) {
-        /* Try to reconstruct valid information from the underlying file */
-        bool has_open_options;
-
-        bs->exact_filename[0] = '\0';
-        if (bs->full_open_options) {
-            qobject_unref(bs->full_open_options);
-            bs->full_open_options = NULL;
-        }
-
-        opts = qdict_new();
-        has_open_options = append_open_options(opts, bs);
-        has_open_options |= backing_overridden;
-
-        /* If no specific options have been given for this BDS, the filename of
-         * the underlying file should suffice for this one as well */
-        if (bs->file->bs->exact_filename[0] && !has_open_options) {
-            strcpy(bs->exact_filename, bs->file->bs->exact_filename);
-        }
-        /* Reconstructing the full options QDict is simple for most format block
-         * drivers, as long as the full options are known for the underlying
-         * file BDS. The full options QDict of that file BDS should somehow
-         * contain a representation of the filename, therefore the following
-         * suffices without querying the (exact_)filename of this BDS. */
-        if (bs->file->bs->full_open_options &&
-            (!bs->backing || bs->backing->bs->full_open_options))
-        {
-            qdict_put_str(opts, "driver", drv->format_name);
-            qdict_put(opts, "file",
-                      qobject_ref(bs->file->bs->full_open_options));
-
-            if (bs->backing) {
-                qdict_put(opts, "backing",
-                          qobject_ref(bs->backing->bs->full_open_options));
-            } else if (backing_overridden) {
-                qdict_put_null(opts, "backing");
-            }
-
-            bs->full_open_options = opts;
-        } else {
-            qobject_unref(opts);
-        }
-    } else if (!bs->full_open_options && qdict_size(bs->options)) {
-        /* There is no underlying file BDS (at least referenced by BDS.file),
-         * so the full options QDict should be equal to the options given
-         * specifically for this block device when it was opened (plus the
-         * driver specification).
-         * Because those options don't change, there is no need to update
-         * full_open_options when it's already set. */
-
-        opts = qdict_new();
-        append_open_options(opts, bs);
-        qdict_put_str(opts, "driver", drv->format_name);
-
-        if (bs->exact_filename[0]) {
-            /* This may not work for all block protocol drivers (some may
-             * require this filename to be parsed), but we have to find some
-             * default solution here, so just include it. If some block driver
-             * does not support pure options without any filename at all or
-             * needs some special format of the options QDict, it needs to
-             * implement the driver-specific bdrv_refresh_filename() function.
-             */
-            qdict_put_str(opts, "filename", bs->exact_filename);
-        }
-
-        bs->full_open_options = opts;
-    }
-
     /* Gather the options QDict */
     opts = qdict_new();
-    append_strong_runtime_options(opts, bs);
+    generate_json_filename = append_strong_runtime_options(opts, bs);
+    generate_json_filename |= backing_overridden;
 
     if (drv->bdrv_gather_child_options) {
         /* Some block drivers may not want to present all of their children's
@@ -5825,6 +5720,24 @@ void bdrv_refresh_filename(BlockDriverState *bs)
     qobject_unref(bs->full_open_options);
     bs->full_open_options = opts;
 
+    if (drv->bdrv_refresh_filename) {
+        /* Obsolete information is of no use here, so drop the old file name
+         * information before refreshing it */
+        bs->exact_filename[0] = '\0';
+
+        drv->bdrv_refresh_filename(bs);
+    } else if (bs->file) {
+        /* Try to reconstruct valid information from the underlying file */
+
+        bs->exact_filename[0] = '\0';
+
+        /* If no specific options have been given for this BDS, the filename of
+         * the underlying file should suffice for this one as well */
+        if (bs->file->bs->exact_filename[0] && !generate_json_filename) {
+            strcpy(bs->exact_filename, bs->file->bs->exact_filename);
+        }
+    }
+
     if (bs->exact_filename[0]) {
         pstrcpy(bs->filename, sizeof(bs->filename), bs->exact_filename);
     } else {
diff --git a/block/blkdebug.c b/block/blkdebug.c
index 71b4275b98..1ea835c2b9 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -811,51 +811,37 @@ static int64_t blkdebug_getlength(BlockDriverState *bs)
     return bdrv_getlength(bs->file->bs);
 }
 
-static void blkdebug_refresh_filename(BlockDriverState *bs, QDict *options)
+static void blkdebug_refresh_filename(BlockDriverState *bs)
 {
     BDRVBlkdebugState *s = bs->opaque;
-    QDict *opts;
     const QDictEntry *e;
-    bool force_json = false;
-
-    for (e = qdict_first(options); e; e = qdict_next(options, e)) {
-        if (strcmp(qdict_entry_key(e), "config") &&
-            strcmp(qdict_entry_key(e), "x-image"))
-        {
-            force_json = true;
-            break;
-        }
-    }
+    int ret;
 
-    if (force_json && !bs->file->bs->full_open_options) {
-        /* The config file cannot be recreated, so creating a plain filename
-         * is impossible */
+    if (!bs->file->bs->exact_filename[0]) {
         return;
     }
 
-    if (!force_json && bs->file->bs->exact_filename[0]) {
-        int ret = snprintf(bs->exact_filename, sizeof(bs->exact_filename),
-                           "blkdebug:%s:%s", s->config_file ?: "",
-                           bs->file->bs->exact_filename);
-        if (ret >= sizeof(bs->exact_filename)) {
-            /* An overflow makes the filename unusable, so do not report any */
-            bs->exact_filename[0] = 0;
+    for (e = qdict_first(bs->full_open_options); e;
+         e = qdict_next(bs->full_open_options, e))
+    {
+        /* Real child options are under "image", but "x-image" may
+         * contain a filename */
+        if (strcmp(qdict_entry_key(e), "config") &&
+            strcmp(qdict_entry_key(e), "image") &&
+            strcmp(qdict_entry_key(e), "x-image") &&
+            strcmp(qdict_entry_key(e), "driver"))
+        {
+            return;
         }
     }
 
-    opts = qdict_new();
-    qdict_put_str(opts, "driver", "blkdebug");
-
-    qdict_put(opts, "image", qobject_ref(bs->file->bs->full_open_options));
-
-    for (e = qdict_first(options); e; e = qdict_next(options, e)) {
-        if (strcmp(qdict_entry_key(e), "x-image")) {
-            qdict_put_obj(opts, qdict_entry_key(e),
-                          qobject_ref(qdict_entry_value(e)));
-        }
+    ret = snprintf(bs->exact_filename, sizeof(bs->exact_filename),
+                   "blkdebug:%s:%s",
+                   s->config_file ?: "", bs->file->bs->exact_filename);
+    if (ret >= sizeof(bs->exact_filename)) {
+        /* An overflow makes the filename unusable, so do not report any */
+        bs->exact_filename[0] = 0;
     }
-
-    bs->full_open_options = opts;
 }
 
 static void blkdebug_refresh_limits(BlockDriverState *bs, Error **errp)
diff --git a/block/blklogwrites.c b/block/blklogwrites.c
index 5da5df112d..eb2b4901a5 100644
--- a/block/blklogwrites.c
+++ b/block/blklogwrites.c
@@ -280,27 +280,6 @@ static int64_t blk_log_writes_getlength(BlockDriverState *bs)
     return bdrv_getlength(bs->file->bs);
 }
 
-static void blk_log_writes_refresh_filename(BlockDriverState *bs,
-                                            QDict *options)
-{
-    BDRVBlkLogWritesState *s = bs->opaque;
-
-    if (bs->file->bs->full_open_options
-        && s->log_file->bs->full_open_options)
-    {
-        QDict *opts = qdict_new();
-        qdict_put_str(opts, "driver", "blklogwrites");
-
-        qobject_ref(bs->file->bs->full_open_options);
-        qdict_put(opts, "file", bs->file->bs->full_open_options);
-        qobject_ref(s->log_file->bs->full_open_options);
-        qdict_put(opts, "log", s->log_file->bs->full_open_options);
-        qdict_put_int(opts, "log-sector-size", s->sectorsize);
-
-        bs->full_open_options = opts;
-    }
-}
-
 static void blk_log_writes_child_perm(BlockDriverState *bs, BdrvChild *c,
                                       const BdrvChildRole *role,
                                       BlockReopenQueue *ro_q,
@@ -531,7 +510,6 @@ static BlockDriver bdrv_blk_log_writes = {
     .bdrv_open              = blk_log_writes_open,
     .bdrv_close             = blk_log_writes_close,
     .bdrv_getlength         = blk_log_writes_getlength,
-    .bdrv_refresh_filename  = blk_log_writes_refresh_filename,
     .bdrv_child_perm        = blk_log_writes_child_perm,
     .bdrv_refresh_limits    = blk_log_writes_refresh_limits,
 
diff --git a/block/blkverify.c b/block/blkverify.c
index 3c7d4c8729..3ff77ff49a 100644
--- a/block/blkverify.c
+++ b/block/blkverify.c
@@ -281,24 +281,10 @@ static bool blkverify_recurse_is_first_non_filter(BlockDriverState *bs,
     return bdrv_recurse_is_first_non_filter(s->test_file->bs, candidate);
 }
 
-static void blkverify_refresh_filename(BlockDriverState *bs, QDict *options)
+static void blkverify_refresh_filename(BlockDriverState *bs)
 {
     BDRVBlkverifyState *s = bs->opaque;
 
-    if (bs->file->bs->full_open_options
-        && s->test_file->bs->full_open_options)
-    {
-        QDict *opts = qdict_new();
-        qdict_put_str(opts, "driver", "blkverify");
-
-        qdict_put(opts, "raw",
-                  qobject_ref(bs->file->bs->full_open_options));
-        qdict_put(opts, "test",
-                  qobject_ref(s->test_file->bs->full_open_options));
-
-        bs->full_open_options = opts;
-    }
-
     if (bs->file->bs->exact_filename[0]
         && s->test_file->bs->exact_filename[0])
     {
diff --git a/block/commit.c b/block/commit.c
index 614a8ca374..385fb98527 100644
--- a/block/commit.c
+++ b/block/commit.c
@@ -230,7 +230,7 @@ static int coroutine_fn bdrv_commit_top_preadv(BlockDriverState *bs,
     return bdrv_co_preadv(bs->backing, offset, bytes, qiov, flags);
 }
 
-static void bdrv_commit_top_refresh_filename(BlockDriverState *bs, QDict *opts)
+static void bdrv_commit_top_refresh_filename(BlockDriverState *bs)
 {
     pstrcpy(bs->exact_filename, sizeof(bs->exact_filename),
             bs->backing->bs->filename);
diff --git a/block/mirror.c b/block/mirror.c
index 031c1aeaeb..726d3c27fb 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -1431,7 +1431,7 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs,
                                     NULL, 0);
 }
 
-static void bdrv_mirror_top_refresh_filename(BlockDriverState *bs, QDict *opts)
+static void bdrv_mirror_top_refresh_filename(BlockDriverState *bs)
 {
     if (bs->backing == NULL) {
         /* we can be here after failed bdrv_attach_child in
diff --git a/block/nbd.c b/block/nbd.c
index 318a58776c..2e72df528a 100644
--- a/block/nbd.c
+++ b/block/nbd.c
@@ -477,12 +477,9 @@ static void nbd_attach_aio_context(BlockDriverState *bs,
     nbd_client_attach_aio_context(bs, new_context);
 }
 
-static void nbd_refresh_filename(BlockDriverState *bs, QDict *options)
+static void nbd_refresh_filename(BlockDriverState *bs)
 {
     BDRVNBDState *s = bs->opaque;
-    QDict *opts = qdict_new();
-    QObject *saddr_qdict;
-    Visitor *ov;
     const char *host = NULL, *port = NULL, *path = NULL;
 
     if (s->saddr->type == SOCKET_ADDRESS_TYPE_INET) {
@@ -495,8 +492,6 @@ static void nbd_refresh_filename(BlockDriverState *bs, QDict *options)
         path = s->saddr->u.q_unix.path;
     } /* else can't represent as pseudo-filename */
 
-    qdict_put_str(opts, "driver", "nbd");
-
     if (path && s->export) {
         snprintf(bs->exact_filename, sizeof(bs->exact_filename),
                  "nbd+unix:///%s?socket=%s", s->export, path);
@@ -510,22 +505,6 @@ static void nbd_refresh_filename(BlockDriverState *bs, QDict *options)
         snprintf(bs->exact_filename, sizeof(bs->exact_filename),
                  "nbd://%s:%s", host, port);
     }
-
-    ov = qobject_output_visitor_new(&saddr_qdict);
-    visit_type_SocketAddress(ov, NULL, &s->saddr, &error_abort);
-    visit_complete(ov, &saddr_qdict);
-    visit_free(ov);
-    qdict_put_obj(opts, "server", saddr_qdict);
-
-    if (s->export) {
-        qdict_put_str(opts, "export", s->export);
-    }
-    if (s->tlscredsid) {
-        qdict_put_str(opts, "tls-creds", s->tlscredsid);
-    }
-
-    qdict_flatten(opts);
-    bs->full_open_options = opts;
 }
 
 static char *nbd_dirname(BlockDriverState *bs, Error **errp)
diff --git a/block/nfs.c b/block/nfs.c
index 6985a44b89..531903610b 100644
--- a/block/nfs.c
+++ b/block/nfs.c
@@ -799,14 +799,9 @@ static int nfs_reopen_prepare(BDRVReopenState *state,
     return 0;
 }
 
-static void nfs_refresh_filename(BlockDriverState *bs, QDict *options)
+static void nfs_refresh_filename(BlockDriverState *bs)
 {
     NFSClient *client = bs->opaque;
-    QDict *opts = qdict_new();
-    QObject *server_qdict;
-    Visitor *ov;
-
-    qdict_put_str(opts, "driver", "nfs");
 
     if (client->uid && !client->gid) {
         snprintf(bs->exact_filename, sizeof(bs->exact_filename),
@@ -824,35 +819,6 @@ static void nfs_refresh_filename(BlockDriverState *bs, QDict *options)
         snprintf(bs->exact_filename, sizeof(bs->exact_filename),
                  "nfs://%s%s", client->server->host, client->path);
     }
-
-    ov = qobject_output_visitor_new(&server_qdict);
-    visit_type_NFSServer(ov, NULL, &client->server, &error_abort);
-    visit_complete(ov, &server_qdict);
-    qdict_put_obj(opts, "server", server_qdict);
-    qdict_put_str(opts, "path", client->path);
-
-    if (client->uid) {
-        qdict_put_int(opts, "user", client->uid);
-    }
-    if (client->gid) {
-        qdict_put_int(opts, "group", client->gid);
-    }
-    if (client->tcp_syncnt) {
-        qdict_put_int(opts, "tcp-syn-cnt", client->tcp_syncnt);
-    }
-    if (client->readahead) {
-        qdict_put_int(opts, "readahead-size", client->readahead);
-    }
-    if (client->pagecache) {
-        qdict_put_int(opts, "page-cache-size", client->pagecache);
-    }
-    if (client->debug) {
-        qdict_put_int(opts, "debug", client->debug);
-    }
-
-    visit_free(ov);
-    qdict_flatten(opts);
-    bs->full_open_options = opts;
 }
 
 static char *nfs_dirname(BlockDriverState *bs, Error **errp)
diff --git a/block/null.c b/block/null.c
index 858892f0c4..1c56a0ef01 100644
--- a/block/null.c
+++ b/block/null.c
@@ -239,17 +239,23 @@ static int coroutine_fn null_co_block_status(BlockDriverState *bs,
     return ret;
 }
 
-static void null_refresh_filename(BlockDriverState *bs, QDict *opts)
+static void null_refresh_filename(BlockDriverState *bs)
 {
-    qdict_del(opts, "filename");
-
-    if (!qdict_size(opts)) {
-        snprintf(bs->exact_filename, sizeof(bs->exact_filename), "%s://",
-                 bs->drv->format_name);
+    const QDictEntry *e;
+
+    for (e = qdict_first(bs->full_open_options); e;
+         e = qdict_next(bs->full_open_options, e))
+    {
+        /* These options can be ignored */
+        if (strcmp(qdict_entry_key(e), "filename") &&
+            strcmp(qdict_entry_key(e), "driver"))
+        {
+            return;
+        }
     }
 
-    qdict_put_str(opts, "driver", bs->drv->format_name);
-    bs->full_open_options = qobject_ref(opts);
+    snprintf(bs->exact_filename, sizeof(bs->exact_filename), "%s://",
+             bs->drv->format_name);
 }
 
 static const char *const null_strong_runtime_opts[] = {
diff --git a/block/nvme.c b/block/nvme.c
index bf656b2bba..6b5845644b 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -1053,17 +1053,23 @@ static int nvme_reopen_prepare(BDRVReopenState *reopen_state,
     return 0;
 }
 
-static void nvme_refresh_filename(BlockDriverState *bs, QDict *opts)
+static void nvme_refresh_filename(BlockDriverState *bs)
 {
-    qdict_del(opts, "filename");
-
-    if (!qdict_size(opts)) {
-        snprintf(bs->exact_filename, sizeof(bs->exact_filename), "%s://",
-                 bs->drv->format_name);
+    const QDictEntry *e;
+
+    for (e = qdict_first(bs->full_open_options); e;
+         e = qdict_next(bs->full_open_options, e))
+    {
+        /* These options can be ignored */
+        if (strcmp(qdict_entry_key(e), "filename") &&
+            strcmp(qdict_entry_key(e), "driver"))
+        {
+            return;
+        }
     }
 
-    qdict_put_str(opts, "driver", bs->drv->format_name);
-    bs->full_open_options = qobject_ref(opts);
+    snprintf(bs->exact_filename, sizeof(bs->exact_filename), "%s://",
+             bs->drv->format_name);
 }
 
 static void nvme_refresh_limits(BlockDriverState *bs, Error **errp)
diff --git a/block/quorum.c b/block/quorum.c
index 3984f0aa4f..352f729136 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -1065,35 +1065,6 @@ static void quorum_del_child(BlockDriverState *bs, BdrvChild *child,
     bdrv_drained_end(bs);
 }
 
-static void quorum_refresh_filename(BlockDriverState *bs, QDict *options)
-{
-    BDRVQuorumState *s = bs->opaque;
-    QDict *opts;
-    QList *children;
-    int i;
-
-    for (i = 0; i < s->num_children; i++) {
-        if (!s->children[i]->bs->full_open_options) {
-            return;
-        }
-    }
-
-    children = qlist_new();
-    for (i = 0; i < s->num_children; i++) {
-        qlist_append(children,
-                     qobject_ref(s->children[i]->bs->full_open_options));
-    }
-
-    opts = qdict_new();
-    qdict_put_str(opts, "driver", "quorum");
-    qdict_put_int(opts, QUORUM_OPT_VOTE_THRESHOLD, s->threshold);
-    qdict_put_bool(opts, QUORUM_OPT_BLKVERIFY, s->is_blkverify);
-    qdict_put_bool(opts, QUORUM_OPT_REWRITE, s->rewrite_corrupted);
-    qdict_put(opts, "children", children);
-
-    bs->full_open_options = opts;
-}
-
 static void quorum_gather_child_options(BlockDriverState *bs, QDict *target,
                                         bool backing_overridden)
 {
@@ -1159,7 +1130,6 @@ static BlockDriver bdrv_quorum = {
 
     .bdrv_open                          = quorum_open,
     .bdrv_close                         = quorum_close,
-    .bdrv_refresh_filename              = quorum_refresh_filename,
     .bdrv_gather_child_options          = quorum_gather_child_options,
     .bdrv_dirname                       = quorum_dirname,
 
diff --git a/tests/qemu-iotests/228 b/tests/qemu-iotests/228
index 7a57c41e29..9a50afd205 100755
--- a/tests/qemu-iotests/228
+++ b/tests/qemu-iotests/228
@@ -230,10 +230,6 @@ with iotests.FilePath('base.img') as base_img_path, \
                overlay='node0')
 
     # This should give us the original plain result
-    # FIXME: Currently, it yields a json:{} filename even though it
-    #        only contains a @driver and a @file entry, so a plain
-    #        filename would obviously suffice.  This is fixed by a
-    #        future patch.
 
     log_node_info(vm.node_info('node0'))
 
diff --git a/tests/qemu-iotests/228.out b/tests/qemu-iotests/228.out
index 393ab09164..4217df24fe 100644
--- a/tests/qemu-iotests/228.out
+++ b/tests/qemu-iotests/228.out
@@ -74,7 +74,7 @@ bs->backing: (none)
 {"execute": "blockdev-snapshot", "arguments": {"node": "original-backing", "overlay": "node0"}}
 {"return": {}}
 
-bs->filename: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->filename: TEST_DIR/PID-top.img
 bs->backing_file: TEST_DIR/PID-base.img
 bs->backing->bs->filename: TEST_DIR/PID-base.img
 
-- 
2.20.1

[Qemu-devel] [PULL 51/71] block: Do not copy exact_filename from format file

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

If a format BDS's file BDS is in turn a format BDS, we cannot simply use
the same filename, because when opening a BDS tree based on a filename
alone, qemu will create only one format node on top of one protocol node
(disregarding a potential backing file).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-26-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/block.c b/block.c
index 23869623ea..9d9929e1a2 100644
--- a/block.c
+++ b/block.c
@@ -5731,9 +5731,21 @@ void bdrv_refresh_filename(BlockDriverState *bs)
 
         bs->exact_filename[0] = '\0';
 
-        /* If no specific options have been given for this BDS, the filename of
-         * the underlying file should suffice for this one as well */
-        if (bs->file->bs->exact_filename[0] && !generate_json_filename) {
+        /*
+         * We can use the underlying file's filename if:
+         * - it has a filename,
+         * - the file is a protocol BDS, and
+         * - opening that file (as this BDS's format) will automatically create
+         *   the BDS tree we have right now, that is:
+         *   - the user did not significantly change this BDS's behavior with
+         *     some explicit (strong) options
+         *   - no non-file child of this BDS has been overridden by the user
+         *   Both of these conditions are represented by generate_json_filename.
+         */
+        if (bs->file->bs->exact_filename[0] &&
+            bs->file->bs->drv->bdrv_file_open &&
+            !generate_json_filename)
+        {
             strcpy(bs->exact_filename, bs->file->bs->exact_filename);
         }
     }
-- 
2.20.1

[Qemu-devel] [PULL 52/71] block/nvme: Fix bdrv_refresh_filename()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Currently, nvme's bdrv_refresh_filename() is an exact copy of null's
implementation.  However, for null, "null-co://" and "null-aio://" are
indeed valid filenames -- for nvme, they are not, as a device address is
still required.

The correct implementation should generate a filename of the form
"nvme://[PCI address]/[namespace]" (as the comment above
nvme_parse_filename() describes).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-27-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/nvme.c | 23 +++++++++--------------
 1 file changed, 9 insertions(+), 14 deletions(-)

diff --git a/block/nvme.c b/block/nvme.c
index 6b5845644b..0684bbd077 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -111,6 +111,9 @@ typedef struct {
 
     /* Total size of mapped qiov, accessed under dma_map_lock */
     int dma_map_count;
+
+    /* PCI address (required for nvme_refresh_filename()) */
+    char *device;
 } BDRVNVMeState;
 
 #define NVME_BLOCK_OPT_DEVICE "device"
@@ -557,6 +560,7 @@ static int nvme_init(BlockDriverState *bs, const char *device, int namespace,
 
     qemu_co_mutex_init(&s->dma_map_lock);
     qemu_co_queue_init(&s->dma_flush_queue);
+    s->device = g_strdup(device);
     s->nsid = namespace;
     s->aio_context = bdrv_get_aio_context(bs);
     ret = event_notifier_init(&s->irq_notifier, 0);
@@ -729,6 +733,8 @@ static void nvme_close(BlockDriverState *bs)
     event_notifier_cleanup(&s->irq_notifier);
     qemu_vfio_pci_unmap_bar(s->vfio, 0, (void *)s->regs, 0, NVME_BAR_SIZE);
     qemu_vfio_close(s->vfio);
+
+    g_free(s->device);
 }
 
 static int nvme_file_open(BlockDriverState *bs, QDict *options, int flags,
@@ -1055,21 +1061,10 @@ static int nvme_reopen_prepare(BDRVReopenState *reopen_state,
 
 static void nvme_refresh_filename(BlockDriverState *bs)
 {
-    const QDictEntry *e;
-
-    for (e = qdict_first(bs->full_open_options); e;
-         e = qdict_next(bs->full_open_options, e))
-    {
-        /* These options can be ignored */
-        if (strcmp(qdict_entry_key(e), "filename") &&
-            strcmp(qdict_entry_key(e), "driver"))
-        {
-            return;
-        }
-    }
+    BDRVNVMeState *s = bs->opaque;
 
-    snprintf(bs->exact_filename, sizeof(bs->exact_filename), "%s://",
-             bs->drv->format_name);
+    snprintf(bs->exact_filename, sizeof(bs->exact_filename), "nvme://%s/%i",
+             s->device, s->nsid);
 }
 
 static void nvme_refresh_limits(BlockDriverState *bs, Error **errp)
-- 
2.20.1

[Qemu-devel] [PULL 53/71] block/curl: Harmonize option defaults

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Both of the defaults we currently have in the curl driver are named
based on a slightly different schema, let's unify that and call both
CURL_BLOCK_OPT_${NAME}_DEFAULT.

While at it, we can add a macro for the third option for which a default
exists, namely "sslverify".

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-28-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index 1c9e4f6a64..2c07a694d8 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -61,8 +61,6 @@ static CURLMcode __curl_multi_socket_action(CURLM *multi_handle,
 
 #define CURL_NUM_STATES 8
 #define CURL_NUM_ACB    8
-#define READ_AHEAD_DEFAULT (256 * 1024)
-#define CURL_TIMEOUT_DEFAULT 5
 #define CURL_TIMEOUT_MAX 10000
 
 #define CURL_BLOCK_OPT_URL       "url"
@@ -76,6 +74,10 @@ static CURLMcode __curl_multi_socket_action(CURLM *multi_handle,
 #define CURL_BLOCK_OPT_PROXY_USERNAME "proxy-username"
 #define CURL_BLOCK_OPT_PROXY_PASSWORD_SECRET "proxy-password-secret"
 
+#define CURL_BLOCK_OPT_READAHEAD_DEFAULT (256 * 1024)
+#define CURL_BLOCK_OPT_SSLVERIFY_DEFAULT true
+#define CURL_BLOCK_OPT_TIMEOUT_DEFAULT 5
+
 struct BDRVCURLState;
 
 static bool libcurl_initialized;
@@ -696,7 +698,7 @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
     }
 
     s->readahead_size = qemu_opt_get_size(opts, CURL_BLOCK_OPT_READAHEAD,
-                                          READ_AHEAD_DEFAULT);
+                                          CURL_BLOCK_OPT_READAHEAD_DEFAULT);
     if ((s->readahead_size & 0x1ff) != 0) {
         error_setg(errp, "HTTP_READAHEAD_SIZE %zd is not a multiple of 512",
                    s->readahead_size);
@@ -704,13 +706,14 @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
     }
 
     s->timeout = qemu_opt_get_number(opts, CURL_BLOCK_OPT_TIMEOUT,
-                                     CURL_TIMEOUT_DEFAULT);
+                                     CURL_BLOCK_OPT_TIMEOUT_DEFAULT);
     if (s->timeout > CURL_TIMEOUT_MAX) {
         error_setg(errp, "timeout parameter is too large or negative");
         goto out_noclean;
     }
 
-    s->sslverify = qemu_opt_get_bool(opts, CURL_BLOCK_OPT_SSLVERIFY, true);
+    s->sslverify = qemu_opt_get_bool(opts, CURL_BLOCK_OPT_SSLVERIFY,
+                                     CURL_BLOCK_OPT_SSLVERIFY_DEFAULT);
 
     cookie = qemu_opt_get(opts, CURL_BLOCK_OPT_COOKIE);
     cookie_secret = qemu_opt_get(opts, CURL_BLOCK_OPT_COOKIE_SECRET);
-- 
2.20.1

[Qemu-devel] [PULL 54/71] block/curl: Implement bdrv_refresh_filename()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-29-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/block/curl.c b/block/curl.c
index 2c07a694d8..606709fea4 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -950,6 +950,23 @@ static int64_t curl_getlength(BlockDriverState *bs)
     return s->len;
 }
 
+static void curl_refresh_filename(BlockDriverState *bs)
+{
+    BDRVCURLState *s = bs->opaque;
+
+    /* "readahead" and "timeout" do not change the guest-visible data,
+     * so ignore them */
+    if (s->sslverify != CURL_BLOCK_OPT_SSLVERIFY_DEFAULT ||
+        s->cookie || s->username || s->password || s->proxyusername ||
+        s->proxypassword)
+    {
+        return;
+    }
+
+    pstrcpy(bs->exact_filename, sizeof(bs->exact_filename), s->url);
+}
+
+
 static const char *const curl_strong_runtime_opts[] = {
     CURL_BLOCK_OPT_URL,
     CURL_BLOCK_OPT_SSLVERIFY,
@@ -978,6 +995,7 @@ static BlockDriver bdrv_http = {
     .bdrv_detach_aio_context    = curl_detach_aio_context,
     .bdrv_attach_aio_context    = curl_attach_aio_context,
 
+    .bdrv_refresh_filename      = curl_refresh_filename,
     .strong_runtime_opts        = curl_strong_runtime_opts,
 };
 
@@ -996,6 +1014,7 @@ static BlockDriver bdrv_https = {
     .bdrv_detach_aio_context    = curl_detach_aio_context,
     .bdrv_attach_aio_context    = curl_attach_aio_context,
 
+    .bdrv_refresh_filename      = curl_refresh_filename,
     .strong_runtime_opts        = curl_strong_runtime_opts,
 };
 
@@ -1014,6 +1033,7 @@ static BlockDriver bdrv_ftp = {
     .bdrv_detach_aio_context    = curl_detach_aio_context,
     .bdrv_attach_aio_context    = curl_attach_aio_context,
 
+    .bdrv_refresh_filename      = curl_refresh_filename,
     .strong_runtime_opts        = curl_strong_runtime_opts,
 };
 
@@ -1032,6 +1052,7 @@ static BlockDriver bdrv_ftps = {
     .bdrv_detach_aio_context    = curl_detach_aio_context,
     .bdrv_attach_aio_context    = curl_attach_aio_context,
 
+    .bdrv_refresh_filename      = curl_refresh_filename,
     .strong_runtime_opts        = curl_strong_runtime_opts,
 };
 
-- 
2.20.1

[Qemu-devel] [PULL 55/71] block/null: Generate filename even with latency-ns

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

While we cannot represent the latency-ns option in a filename, it is not
a strong option so not being able to should not stop us from generating
a filename nonetheless.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-30-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/null.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/block/null.c b/block/null.c
index 1c56a0ef01..a322929478 100644
--- a/block/null.c
+++ b/block/null.c
@@ -248,7 +248,8 @@ static void null_refresh_filename(BlockDriverState *bs)
     {
         /* These options can be ignored */
         if (strcmp(qdict_entry_key(e), "filename") &&
-            strcmp(qdict_entry_key(e), "driver"))
+            strcmp(qdict_entry_key(e), "driver") &&
+            strcmp(qdict_entry_key(e), NULL_OPT_LATENCY))
         {
             return;
         }
-- 
2.20.1

[Qemu-devel] [PULL 56/71] block: BDS options may lack the "driver" option

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

When BDSs are created by qemu itself (e.g. as filters in block jobs),
they may not have a "driver" option in their options QDict.  When
generating a json:{} filename, however, it must always be present.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-31-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/block.c b/block.c
index 9d9929e1a2..35e78e2172 100644
--- a/block.c
+++ b/block.c
@@ -5618,6 +5618,12 @@ static bool append_strong_runtime_options(QDict *d, BlockDriverState *bs)
         }
     }
 
+    if (!qdict_haskey(d, "driver")) {
+        /* Drivers created with bdrv_new_open_driver() may not have a
+         * @driver option.  Add it here. */
+        qdict_put_str(d, "driver", bs->drv->format_name);
+    }
+
     return found_any;
 }
 
-- 
2.20.1

[Qemu-devel] [PULL 57/71] iotests: Test json:{} filenames of internal BDSs

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-32-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/224     | 139 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/224.out |  18 +++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 158 insertions(+)
 create mode 100755 tests/qemu-iotests/224
 create mode 100644 tests/qemu-iotests/224.out

diff --git a/tests/qemu-iotests/224 b/tests/qemu-iotests/224
new file mode 100755
index 0000000000..b4dfaa639f
--- /dev/null
+++ b/tests/qemu-iotests/224
@@ -0,0 +1,139 @@
+#!/usr/bin/env python
+#
+# Test json:{} filenames with qemu-internal BDSs
+# (the one of commit, to be precise)
+#
+# Copyright (C) 2018 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# Creator/Owner: Max Reitz <mreitz@redhat.com>
+
+import iotests
+from iotests import log, qemu_img, qemu_io_silent, filter_qmp_testfiles, \
+                    filter_qmp_imgfmt
+import json
+
+# Need backing file support (for arbitrary backing formats)
+iotests.verify_image_format(supported_fmts=['qcow2', 'qcow', 'qed'])
+iotests.verify_platform(['linux'])
+
+
+# There are two variations of this test:
+# (1) We do not set filter_node_name.  In that case, the commit_top
+#     driver should not appear anywhere.
+# (2) We do set filter_node_name.  In that case, it should appear.
+#
+# This for loop executes both.
+for filter_node_name in False, True:
+    log('')
+    log('--- filter_node_name: %s ---' % filter_node_name)
+    log('')
+
+    with iotests.FilePath('base.img') as base_img_path, \
+         iotests.FilePath('mid.img') as mid_img_path, \
+         iotests.FilePath('top.img') as top_img_path, \
+         iotests.VM() as vm:
+
+        assert qemu_img('create', '-f', iotests.imgfmt,
+                        base_img_path, '64M') == 0
+        assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
+                        mid_img_path) == 0
+        assert qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path,
+                        top_img_path) == 0
+
+        # Something to commit
+        assert qemu_io_silent(mid_img_path, '-c', 'write -P 1 0 1M') == 0
+
+        vm.launch()
+
+        # Change the bottom-most image's backing file (to null-co://)
+        # to enforce json:{} filenames
+        vm.qmp_log('blockdev-add',
+                    node_name='top',
+                    driver=iotests.imgfmt,
+                    file={
+                        'driver': 'file',
+                        'filename': top_img_path
+                    },
+                    backing={
+                        'node-name': 'mid',
+                        'driver': iotests.imgfmt,
+                        'file': {
+                            'driver': 'file',
+                            'filename': mid_img_path
+                        },
+                        'backing': {
+                            'node-name': 'base',
+                            'driver': iotests.imgfmt,
+                            'file': {
+                                'driver': 'file',
+                                'filename': base_img_path
+                            },
+                            'backing': {
+                                'driver': 'null-co'
+                            }
+                        }
+                    },
+                    filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+
+        # As long as block-commit does not accept node names, we have to
+        # get our mid/base filenames here
+        mid_name = vm.node_info('mid')['image']['filename']
+        base_name = vm.node_info('base')['image']['filename']
+
+        assert mid_name[:5] == 'json:'
+        assert base_name[:5] == 'json:'
+
+        # Start the block job
+        if filter_node_name:
+            vm.qmp_log('block-commit',
+                        job_id='commit',
+                        device='top',
+                        filter_node_name='filter_node',
+                        top=mid_name,
+                        base=base_name,
+                        speed=1,
+                        filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+        else:
+            vm.qmp_log('block-commit',
+                        job_id='commit',
+                        device='top',
+                        top=mid_name,
+                        base=base_name,
+                        speed=1,
+                        filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+
+        vm.qmp_log('job-pause', id='commit')
+
+        # Get and parse top's json:{} filename
+        top_name = vm.node_info('top')['image']['filename']
+
+        vm.shutdown()
+
+        assert top_name[:5] == 'json:'
+        top_options = json.loads(top_name[5:])
+
+        if filter_node_name:
+            # This should be present and set
+            assert top_options['backing']['driver'] == 'commit_top'
+            # And the mid image is commit_top's backing image
+            mid_options = top_options['backing']['backing']
+        else:
+            # The mid image should appear as the immediate backing BDS
+            # of top
+            mid_options = top_options['backing']
+
+        assert mid_options['driver'] == iotests.imgfmt
+        assert mid_options['file']['filename'] == mid_img_path
diff --git a/tests/qemu-iotests/224.out b/tests/qemu-iotests/224.out
new file mode 100644
index 0000000000..23374a1d29
--- /dev/null
+++ b/tests/qemu-iotests/224.out
@@ -0,0 +1,18 @@
+
+--- filter_node_name: False ---
+
+{"execute": "blockdev-add", "arguments": {"backing": {"backing": {"backing": {"driver": "null-co"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}, "node-name": "base"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-mid.img"}, "node-name": "mid"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "top"}}
+{"return": {}}
+{"execute": "block-commit", "arguments": {"base": "json:{\"backing\": {\"driver\": \"null-co\"}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-base.img\"}}", "device": "top", "job-id": "commit", "speed": 1, "top": "json:{\"backing\": {\"backing\": {\"driver\": \"null-co\"}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-base.img\"}}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-mid.img\"}}"}}
+{"return": {}}
+{"execute": "job-pause", "arguments": {"id": "commit"}}
+{"return": {}}
+
+--- filter_node_name: True ---
+
+{"execute": "blockdev-add", "arguments": {"backing": {"backing": {"backing": {"driver": "null-co"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}, "node-name": "base"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-mid.img"}, "node-name": "mid"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "top"}}
+{"return": {}}
+{"execute": "block-commit", "arguments": {"base": "json:{\"backing\": {\"driver\": \"null-co\"}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-base.img\"}}", "device": "top", "filter-node-name": "filter_node", "job-id": "commit", "speed": 1, "top": "json:{\"backing\": {\"backing\": {\"driver\": \"null-co\"}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-base.img\"}}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-mid.img\"}}"}}
+{"return": {}}
+{"execute": "job-pause", "arguments": {"id": "commit"}}
+{"return": {}}
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index f701863cdb..b5ca63cf72 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -224,6 +224,7 @@
 221 rw auto quick
 222 rw auto quick
 223 rw auto quick
+224 rw auto quick
 225 rw auto quick
 226 auto quick
 227 auto quick
-- 
2.20.1

[Qemu-devel] [PULL 58/71] iotests: Re-add filename filters

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

A previous commit removed the default filters for qmp_log with the
intention to make them explicit; but this happened only for test 206.
There are more tests (for more exotic image formats than qcow2) which
require the filename filter, though.

Note that 237 is still broken for Python 2.x, which is fixed in the next
commit.

Fixes: f8ca8609d8549def45b28e82ecac64adaeee9f12
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-2-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/210 | 5 +++--
 tests/qemu-iotests/211 | 5 +++--
 tests/qemu-iotests/212 | 5 +++--
 tests/qemu-iotests/213 | 5 +++--
 tests/qemu-iotests/237 | 5 +++--
 5 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/tests/qemu-iotests/210 b/tests/qemu-iotests/210
index d142841e2b..565e3b7b9b 100755
--- a/tests/qemu-iotests/210
+++ b/tests/qemu-iotests/210
@@ -27,7 +27,8 @@ iotests.verify_image_format(supported_fmts=['luks'])
 iotests.verify_protocol(supported=['file'])
 
 def blockdev_create(vm, options):
-    result = vm.qmp_log('blockdev-create', job_id='job0', options=options)
+    result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
+                        filters=[iotests.filter_qmp_testfiles])
 
     if 'return' in result:
         assert result['return'] == {}
@@ -53,7 +54,7 @@ with iotests.FilePath('t.luks') as disk_path, \
                           'size': 0 })
 
     vm.qmp_log('blockdev-add', driver='file', filename=disk_path,
-               node_name='imgfile')
+               node_name='imgfile', filters=[iotests.filter_qmp_testfiles])
 
     blockdev_create(vm, { 'driver': imgfmt,
                           'file': 'imgfile',
diff --git a/tests/qemu-iotests/211 b/tests/qemu-iotests/211
index 7b7985db6c..5d285450b5 100755
--- a/tests/qemu-iotests/211
+++ b/tests/qemu-iotests/211
@@ -27,7 +27,8 @@ iotests.verify_image_format(supported_fmts=['vdi'])
 iotests.verify_protocol(supported=['file'])
 
 def blockdev_create(vm, options):
-    result = vm.qmp_log('blockdev-create', job_id='job0', options=options)
+    result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
+                        filters=[iotests.filter_qmp_testfiles])
 
     if 'return' in result:
         assert result['return'] == {}
@@ -51,7 +52,7 @@ with iotests.FilePath('t.vdi') as disk_path, \
                           'size': 0 })
 
     vm.qmp_log('blockdev-add', driver='file', filename=disk_path,
-               node_name='imgfile')
+               node_name='imgfile', filters=[iotests.filter_qmp_testfiles])
 
     blockdev_create(vm, { 'driver': imgfmt,
                           'file': 'imgfile',
diff --git a/tests/qemu-iotests/212 b/tests/qemu-iotests/212
index 95c8810d83..42b74f208b 100755
--- a/tests/qemu-iotests/212
+++ b/tests/qemu-iotests/212
@@ -27,7 +27,8 @@ iotests.verify_image_format(supported_fmts=['parallels'])
 iotests.verify_protocol(supported=['file'])
 
 def blockdev_create(vm, options):
-    result = vm.qmp_log('blockdev-create', job_id='job0', options=options)
+    result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
+                        filters=[iotests.filter_qmp_testfiles])
 
     if 'return' in result:
         assert result['return'] == {}
@@ -51,7 +52,7 @@ with iotests.FilePath('t.parallels') as disk_path, \
                           'size': 0 })
 
     vm.qmp_log('blockdev-add', driver='file', filename=disk_path,
-               node_name='imgfile')
+               node_name='imgfile', filters=[iotests.filter_qmp_testfiles])
 
     blockdev_create(vm, { 'driver': imgfmt,
                           'file': 'imgfile',
diff --git a/tests/qemu-iotests/213 b/tests/qemu-iotests/213
index 4054439e3c..5604f3cebb 100755
--- a/tests/qemu-iotests/213
+++ b/tests/qemu-iotests/213
@@ -27,7 +27,8 @@ iotests.verify_image_format(supported_fmts=['vhdx'])
 iotests.verify_protocol(supported=['file'])
 
 def blockdev_create(vm, options):
-    result = vm.qmp_log('blockdev-create', job_id='job0', options=options)
+    result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
+                        filters=[iotests.filter_qmp_testfiles])
 
     if 'return' in result:
         assert result['return'] == {}
@@ -51,7 +52,7 @@ with iotests.FilePath('t.vhdx') as disk_path, \
                           'size': 0 })
 
     vm.qmp_log('blockdev-add', driver='file', filename=disk_path,
-               node_name='imgfile')
+               node_name='imgfile', filters=[iotests.filter_qmp_testfiles])
 
     blockdev_create(vm, { 'driver': imgfmt,
                           'file': 'imgfile',
diff --git a/tests/qemu-iotests/237 b/tests/qemu-iotests/237
index 251771d7fb..fe0dd0f461 100755
--- a/tests/qemu-iotests/237
+++ b/tests/qemu-iotests/237
@@ -27,7 +27,8 @@ from iotests import imgfmt
 iotests.verify_image_format(supported_fmts=['vmdk'])
 
 def blockdev_create(vm, options):
-    result = vm.qmp_log('blockdev-create', job_id='job0', options=options)
+    result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
+                        filters=[iotests.filter_qmp_testfiles])
 
     if 'return' in result:
         assert result['return'] == {}
@@ -54,7 +55,7 @@ with iotests.FilePath('t.vmdk') as disk_path, \
                           'size': 0 })
 
     vm.qmp_log('blockdev-add', driver='file', filename=disk_path,
-               node_name='imgfile')
+               node_name='imgfile', filters=[iotests.filter_qmp_testfiles])
 
     blockdev_create(vm, { 'driver': imgfmt,
                           'file': 'imgfile',
-- 
2.20.1

[Qemu-devel] [PULL 59/71] iotests: Fix 237 for Python 2.x

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

math.ceil() returns an integer on Python 3.x, but a float on Python 2.x.
range() always needs integers, so we need an explicit conversion on 2.x
(which does not hurt on 3.x).

It is not quite clear whether we want to support Python 2.x for any
prolonged time, but this may as well be fixed along with the other
issues some iotests have right now.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20190210145736.1486-3-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/237 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/237 b/tests/qemu-iotests/237
index fe0dd0f461..06897f8c87 100755
--- a/tests/qemu-iotests/237
+++ b/tests/qemu-iotests/237
@@ -224,7 +224,7 @@ with iotests.FilePath('t.vmdk') as disk_path, \
             iotests.log("= %s %d =" % (subfmt, size))
             iotests.log("")
 
-            num_extents = math.ceil(size / 2.0**31)
+            num_extents = int(math.ceil(size / 2.0**31))
             extents = [ "ext%d" % (i) for i in range(1, num_extents + 1) ]
 
             vm.launch()
-- 
2.20.1

[Qemu-devel] [PULL 60/71] iotests: Remove superfluous rm from 232

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

This test creates no such file.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-4-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/232 | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/qemu-iotests/232 b/tests/qemu-iotests/232
index 0708b8b155..93e5d641a3 100755
--- a/tests/qemu-iotests/232
+++ b/tests/qemu-iotests/232
@@ -29,7 +29,6 @@ status=1	# failure is the default!
 _cleanup()
 {
     _cleanup_test_img
-    rm -f $TEST_IMG.snap
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
 
-- 
2.20.1

[Qemu-devel] [PULL 61/71] iotests: Fix 232 for LUKS

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

With IMGOPTSSYNTAX, $TEST_IMG is useless for this test (it only tests
the file-posix protocol driver).  Therefore, if $TEST_IMG_FILE is set,
use that instead.

Because this test requires the file protocol, $TEST_IMG_FILE will always
be set if $IMGOPTSSYNTAX is true.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-5-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/232 | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/qemu-iotests/232 b/tests/qemu-iotests/232
index 93e5d641a3..e48bc8f5db 100755
--- a/tests/qemu-iotests/232
+++ b/tests/qemu-iotests/232
@@ -69,6 +69,10 @@ size=128M
 
 _make_test_img $size
 
+if [ -n "$TEST_IMG_FILE" ]; then
+    TEST_IMG=$TEST_IMG_FILE
+fi
+
 echo
 echo "=== -drive with read-write image: read-only/auto-read-only combinations ==="
 echo
-- 
2.20.1

[Qemu-devel] [PULL 62/71] iotests: Fix 207 to use QMP filters for qmp_log

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Fixes: 08fcd6111e1949f456e1b232ebeeb0cc17019a92
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-6-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/207     | 10 +++++++---
 tests/qemu-iotests/207.out |  4 ++--
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
index c617ee7453..dfd3c51bd1 100755
--- a/tests/qemu-iotests/207
+++ b/tests/qemu-iotests/207
@@ -27,12 +27,16 @@ import re
 iotests.verify_image_format(supported_fmts=['raw'])
 iotests.verify_protocol(supported=['ssh'])
 
-def filter_hash(msg):
-    return re.sub('"hash": "[0-9a-f]+"', '"hash": HASH', msg)
+def filter_hash(qmsg):
+    def _filter(key, value):
+        if key == 'hash' and re.match('[0-9a-f]+', value):
+            return 'HASH'
+        return value
+    return iotests.filter_qmp(qmsg, _filter)
 
 def blockdev_create(vm, options):
     result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
-                        filters=[iotests.filter_testfiles, filter_hash])
+                        filters=[iotests.filter_qmp_testfiles, filter_hash])
 
     if 'return' in result:
         assert result['return'] == {}
diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
index 45ac7c2a8f..88d2240f54 100644
--- a/tests/qemu-iotests/207.out
+++ b/tests/qemu-iotests/207.out
@@ -40,7 +40,7 @@ Job failed: remote host key does not match host_key_check 'wrong'
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": HASH, "mode": "hash", "type": "md5"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 8388608}}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "HASH", "mode": "hash", "type": "md5"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 8388608}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -55,7 +55,7 @@ Job failed: remote host key does not match host_key_check 'wrong'
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": HASH, "mode": "hash", "type": "sha1"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "HASH", "mode": "hash", "type": "sha1"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
-- 
2.20.1

[Qemu-devel] [PULL 63/71] iotests.py: Add is_str()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

On Python 2.x, strings are not always unicode strings.  This function
checks whether a given value is a plain string, or a unicode string (if
there is a difference).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-7-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 82dd096c6e..52fc77563c 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -236,6 +236,12 @@ def image_size(img):
     r = qemu_img_pipe('info', '--output=json', '-f', imgfmt, img)
     return json.loads(r)['virtual-size']
 
+def is_str(val):
+    if sys.version_info.major >= 3:
+        return isinstance(val, str)
+    else:
+        return isinstance(val, str) or isinstance(val, unicode)
+
 test_dir_re = re.compile(r"%s" % test_dir)
 def filter_test_dir(msg):
     return test_dir_re.sub("TEST_DIR", msg)
-- 
2.20.1

[Qemu-devel] [PULL 64/71] iotests.py: Filter filename in any string value

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

filter_qmp_testfiles() currently filters the filename only for specific
keys.  However, there are more keys that take filenames (such as
block-commit's @top and @base, or ssh's @path), and it does not make
sense to list them all here.  "$TEST_DIR/$PID-" should have enough
entropy not to appear anywhere randomly.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-8-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 52fc77563c..cba91a9927 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -289,7 +289,7 @@ def filter_testfiles(msg):
 
 def filter_qmp_testfiles(qmsg):
     def _filter(key, value):
-        if key == 'filename' or key == 'backing-file':
+        if is_str(value):
             return filter_testfiles(value)
         return value
     return filter_qmp(qmsg, _filter)
-- 
2.20.1

[Qemu-devel] [PULL 65/71] iotests: Filter SSH paths

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

8908b253c4ad5f8874c8d13abec169c696a5cd32 has implemented filtering of
remote paths for NFS, but forgot SSH.  This patch takes care of that.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-9-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/common.rc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index e15e7a7c8e..09a27f02d0 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -145,6 +145,7 @@ else
         TEST_IMG="nbd:127.0.0.1:10810"
     elif [ "$IMGPROTO" = "ssh" ]; then
         TEST_IMG_FILE=$TEST_DIR/t.$IMGFMT
+        REMOTE_TEST_DIR="ssh://127.0.0.1$TEST_DIR"
         TEST_IMG="ssh://127.0.0.1$TEST_IMG_FILE"
     elif [ "$IMGPROTO" = "nfs" ]; then
         TEST_IMG_FILE=$TEST_DIR/t.$IMGFMT
-- 
2.20.1

[Qemu-devel] [PULL 66/71] iotests: Let 045 be run concurrently

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

Adding a telnet monitor for no real purpose on a fixed port is not so
great.  Just use a null monitor instead.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-10-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 scripts/qemu.py        | 5 ++---
 tests/qemu-iotests/045 | 2 +-
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 32b00af5cc..f7269eefbb 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -144,10 +144,9 @@ class QEMUMachine(object):
         return False
 
     # This can be used to add an unused monitor instance.
-    def add_monitor_telnet(self, ip, port):
-        args = 'tcp:%s:%d,server,nowait,telnet' % (ip, port)
+    def add_monitor_null(self):
         self._args.append('-monitor')
-        self._args.append(args)
+        self._args.append('null')
 
     def add_fd(self, fd, fdset, opaque, opts=''):
         """
diff --git a/tests/qemu-iotests/045 b/tests/qemu-iotests/045
index 55a5d31ca8..d5484a0ee1 100755
--- a/tests/qemu-iotests/045
+++ b/tests/qemu-iotests/045
@@ -132,7 +132,7 @@ class TestSCMFd(iotests.QMPTestCase):
         qemu_img('create', '-f', iotests.imgfmt, image0, '128K')
         # Add an unused monitor, to verify it works fine when two monitor
         # instances present
-        self.vm.add_monitor_telnet("0",4445)
+        self.vm.add_monitor_null()
         self.vm.launch()
 
     def tearDown(self):
-- 
2.20.1

[Qemu-devel] [PULL 67/71] iotests.py: s/_/-/g on keys in qmp_log()

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

This follows what qmp() does, so the output will correspond to the
actual QMP command.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190210145736.1486-11-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/206.out    | 56 +++++++++++++++++------------------
 tests/qemu-iotests/207.out    | 18 +++++------
 tests/qemu-iotests/210.out    | 28 +++++++++---------
 tests/qemu-iotests/211.out    | 26 ++++++++--------
 tests/qemu-iotests/212.out    | 44 +++++++++++++--------------
 tests/qemu-iotests/213.out    | 46 ++++++++++++++--------------
 tests/qemu-iotests/237.out    | 54 ++++++++++++++++-----------------
 tests/qemu-iotests/iotests.py |  6 ++--
 8 files changed, 140 insertions(+), 138 deletions(-)

diff --git a/tests/qemu-iotests/206.out b/tests/qemu-iotests/206.out
index 91f4db55d3..0f1c23babb 100644
--- a/tests/qemu-iotests/206.out
+++ b/tests/qemu-iotests/206.out
@@ -1,13 +1,13 @@
 === Successful image creation (defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "node_name": "imgfile"}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "node-name": "imgfile"}}
 {"return": {}}
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "imgfile", "size": 134217728}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "imgfile", "size": 134217728}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -24,12 +24,12 @@ Format specific information:
 
 === Successful image creation (inline blockdev-add, explicit defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "nocow": false, "preallocation": "off", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "nocow": false, "preallocation": "off", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 65536, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "lazy-refcounts": false, "preallocation": "off", "refcount-bits": 16, "size": 67108864, "version": "v3"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 65536, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "lazy-refcounts": false, "preallocation": "off", "refcount-bits": 16, "size": 67108864, "version": "v3"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -46,12 +46,12 @@ Format specific information:
 
 === Successful image creation (v3 non-default options) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "nocow": true, "preallocation": "falloc", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "nocow": true, "preallocation": "falloc", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 2097152, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "lazy-refcounts": true, "preallocation": "metadata", "refcount-bits": 1, "size": 33554432, "version": "v3"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 2097152, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "lazy-refcounts": true, "preallocation": "metadata", "refcount-bits": 1, "size": 33554432, "version": "v3"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -68,12 +68,12 @@ Format specific information:
 
 === Successful image creation (v2 non-default options) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"backing-file": "TEST_DIR/PID-t.qcow2.base", "backing-fmt": "qcow2", "cluster-size": 512, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "size": 33554432, "version": "v2"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"backing-file": "TEST_DIR/PID-t.qcow2.base", "backing-fmt": "qcow2", "cluster-size": 512, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "size": 33554432, "version": "v2"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -90,7 +90,7 @@ Format specific information:
 
 === Successful image creation (encrypted) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "encrypt": {"cipher-alg": "twofish-128", "cipher-mode": "ctr", "format": "luks", "hash-alg": "sha1", "iter-time": 10, "ivgen-alg": "plain64", "ivgen-hash-alg": "md5", "key-secret": "keysec0"}, "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "encrypt": {"cipher-alg": "twofish-128", "cipher-mode": "ctr", "format": "luks", "hash-alg": "sha1", "iter-time": 10, "ivgen-alg": "plain64", "ivgen-hash-alg": "md5", "key-secret": "keysec0"}, "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "size": 33554432}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -144,111 +144,111 @@ Format specific information:
 
 === Invalid BlockdevRef ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "this doesn't exist", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "this doesn't exist", "size": 33554432}}}
 {"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
 === Invalid sizes ===
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 1234}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 1234}}}
 {"return": {}}
 Job failed: Image size must be a multiple of 512 bytes
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 18446744073709551104}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 18446744073709551104}}}
 {"return": {}}
 Job failed: Could not resize image: Image size cannot be negative
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 9223372036854775808}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 9223372036854775808}}}
 {"return": {}}
 Job failed: Could not resize image: Image size cannot be negative
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 9223372036854775296}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 9223372036854775296}}}
 {"return": {}}
 Job failed: Could not resize image: Failed to grow the L1 table: File too large
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
 === Invalid version ===
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 67108864, "version": "v1"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 67108864, "version": "v1"}}}
 {"error": {"class": "GenericError", "desc": "Invalid parameter 'v1'"}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "lazy-refcounts": true, "size": 67108864, "version": "v2"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "lazy-refcounts": true, "size": 67108864, "version": "v2"}}}
 {"return": {}}
 Job failed: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 8, "size": 67108864, "version": "v2"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 8, "size": 67108864, "version": "v2"}}}
 {"return": {}}
 Job failed: Different refcount widths than 16 bits require compatibility level 1.1 or above (use version=v3 or greater)
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
 === Invalid backing file options ===
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"backing-file": "/dev/null", "driver": "qcow2", "file": "node0", "preallocation": "full", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"backing-file": "/dev/null", "driver": "qcow2", "file": "node0", "preallocation": "full", "size": 67108864}}}
 {"return": {}}
 Job failed: Backing file and preallocation cannot be used at the same time
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"backing-fmt": "qcow2", "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"backing-fmt": "qcow2", "driver": "qcow2", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Backing format cannot be used without backing file
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
 === Invalid cluster size ===
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 1234, "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 1234, "driver": "qcow2", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 128, "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 128, "driver": "qcow2", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 4194304, "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 4194304, "driver": "qcow2", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 0, "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 0, "driver": "qcow2", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 512, "driver": "qcow2", "file": "node0", "size": 281474976710656}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 512, "driver": "qcow2", "file": "node0", "size": 281474976710656}}}
 {"return": {}}
 Job failed: Could not resize image: Failed to grow the L1 table: File too large
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
 === Invalid refcount width ===
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 128, "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 128, "size": 67108864}}}
 {"return": {}}
 Job failed: Refcount width must be a power of two and may not exceed 64 bits
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 0, "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 0, "size": 67108864}}}
 {"return": {}}
 Job failed: Refcount width must be a power of two and may not exceed 64 bits
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 7, "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 7, "size": 67108864}}}
 {"return": {}}
 Job failed: Refcount width must be a power of two and may not exceed 64 bits
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
index 88d2240f54..568e8619d0 100644
--- a/tests/qemu-iotests/207.out
+++ b/tests/qemu-iotests/207.out
@@ -1,6 +1,6 @@
 === Successful image creation (defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -16,7 +16,7 @@ virtual size: 4.0M (4194304 bytes)
 
 === Test host-key-check options ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 8388608}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 8388608}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -25,7 +25,7 @@ image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.po
 file format: IMGFMT
 virtual size: 8.0M (8388608 bytes)
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "known_hosts"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "known_hosts"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -34,13 +34,13 @@ image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.po
 file format: IMGFMT
 virtual size: 4.0M (4194304 bytes)
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "wrong", "mode": "hash", "type": "md5"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 2097152}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "wrong", "mode": "hash", "type": "md5"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 2097152}}}
 {"return": {}}
 Job failed: remote host key does not match host_key_check 'wrong'
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "HASH", "mode": "hash", "type": "md5"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 8388608}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "HASH", "mode": "hash", "type": "md5"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 8388608}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -49,13 +49,13 @@ image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.po
 file format: IMGFMT
 virtual size: 8.0M (8388608 bytes)
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "wrong", "mode": "hash", "type": "sha1"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 2097152}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "wrong", "mode": "hash", "type": "sha1"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 2097152}}}
 {"return": {}}
 Job failed: remote host key does not match host_key_check 'wrong'
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "HASH", "mode": "hash", "type": "sha1"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "HASH", "mode": "hash", "type": "sha1"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -66,13 +66,13 @@ virtual size: 4.0M (4194304 bytes)
 
 === Invalid path and user ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
 {"return": {}}
 Job failed: failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}, "user": "invalid user"}, "size": 4194304}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}, "user": "invalid user"}, "size": 4194304}}}
 {"return": {}}
 Job failed: failed to authenticate using publickey authentication and the identities held by your ssh-agent
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
diff --git a/tests/qemu-iotests/210.out b/tests/qemu-iotests/210.out
index 923cb05117..a3692ce00d 100644
--- a/tests/qemu-iotests/210.out
+++ b/tests/qemu-iotests/210.out
@@ -1,13 +1,13 @@
 === Successful image creation (defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.luks", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.luks", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.luks", "node_name": "imgfile"}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.luks", "node-name": "imgfile"}}
 {"return": {}}
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "imgfile", "iter-time": 10, "key-secret": "keysec0", "size": 134217728}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "luks", "file": "imgfile", "iter-time": 10, "key-secret": "keysec0", "size": 134217728}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -54,12 +54,12 @@ Format specific information:
 
 === Successful image creation (with non-default options) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.luks", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.luks", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cipher-alg": "twofish-128", "cipher-mode": "ctr", "driver": "luks", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.luks"}, "hash-alg": "sha1", "iter-time": 10, "ivgen-alg": "plain64", "ivgen-hash-alg": "md5", "key-secret": "keysec0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cipher-alg": "twofish-128", "cipher-mode": "ctr", "driver": "luks", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.luks"}, "hash-alg": "sha1", "iter-time": 10, "ivgen-alg": "plain64", "ivgen-hash-alg": "md5", "key-secret": "keysec0", "size": 67108864}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -106,7 +106,7 @@ Format specific information:
 
 === Invalid BlockdevRef ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "this doesn't exist", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "luks", "file": "this doesn't exist", "size": 67108864}}}
 {"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -114,7 +114,7 @@ Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exi
 
 === Zero size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "node0", "iter-time": 10, "key-secret": "keysec0", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "luks", "file": "node0", "iter-time": 10, "key-secret": "keysec0", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -161,19 +161,19 @@ Format specific information:
 
 === Invalid sizes ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "node0", "key-secret": "keysec0", "size": 18446744073709551104}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "luks", "file": "node0", "key-secret": "keysec0", "size": 18446744073709551104}}}
 {"return": {}}
 Job failed: The requested file size is too large
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "node0", "key-secret": "keysec0", "size": 9223372036854775808}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "luks", "file": "node0", "key-secret": "keysec0", "size": 9223372036854775808}}}
 {"return": {}}
 Job failed: The requested file size is too large
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "node0", "key-secret": "keysec0", "size": 9223372036854775296}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "luks", "file": "node0", "key-secret": "keysec0", "size": 9223372036854775296}}}
 {"return": {}}
 Job failed: The requested file size is too large
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -181,13 +181,13 @@ Job failed: The requested file size is too large
 
 === Resize image with invalid sizes ===
 
-{"execute": "block_resize", "arguments": {"node_name": "node1", "size": 9223372036854775296}}
+{"execute": "block_resize", "arguments": {"node-name": "node1", "size": 9223372036854775296}}
 {"error": {"class": "GenericError", "desc": "The requested file size is too large"}}
-{"execute": "block_resize", "arguments": {"node_name": "node1", "size": 9223372036854775808}}
+{"execute": "block_resize", "arguments": {"node-name": "node1", "size": 9223372036854775808}}
 {"error": {"class": "GenericError", "desc": "Invalid parameter type for 'size', expected: integer"}}
-{"execute": "block_resize", "arguments": {"node_name": "node1", "size": 18446744073709551104}}
+{"execute": "block_resize", "arguments": {"node-name": "node1", "size": 18446744073709551104}}
 {"error": {"class": "GenericError", "desc": "Invalid parameter type for 'size', expected: integer"}}
-{"execute": "block_resize", "arguments": {"node_name": "node1", "size": -9223372036854775808}}
+{"execute": "block_resize", "arguments": {"node-name": "node1", "size": -9223372036854775808}}
 {"error": {"class": "GenericError", "desc": "Parameter 'size' expects a >0 size"}}
 image: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_IMG"}, "key-secret": "keysec0"}
 file format: IMGFMT
diff --git a/tests/qemu-iotests/211.out b/tests/qemu-iotests/211.out
index eebb0ea086..682adc2a10 100644
--- a/tests/qemu-iotests/211.out
+++ b/tests/qemu-iotests/211.out
@@ -1,13 +1,13 @@
 === Successful image creation (defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "node_name": "imgfile"}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "node-name": "imgfile"}}
 {"return": {}}
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "imgfile", "size": 134217728}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": "imgfile", "size": 134217728}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -21,12 +21,12 @@ cluster_size: 1048576
 
 === Successful image creation (explicit defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi"}, "preallocation": "off", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi"}, "preallocation": "off", "size": 67108864}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -40,12 +40,12 @@ cluster_size: 1048576
 
 === Successful image creation (with non-default options) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi"}, "preallocation": "metadata", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi"}, "preallocation": "metadata", "size": 33554432}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -60,7 +60,7 @@ cluster_size: 1048576
 
 === Invalid BlockdevRef ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "this doesn't exist", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": "this doesn't exist", "size": 33554432}}}
 {"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -68,7 +68,7 @@ Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exi
 
 === Zero size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -80,7 +80,7 @@ cluster_size: 1048576
 
 === Maximum size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 562949819203584}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 562949819203584}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -92,19 +92,19 @@ cluster_size: 1048576
 
 === Invalid sizes ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 18446744073709551104}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 18446744073709551104}}}
 {"return": {}}
 Job failed: Unsupported VDI image size (size is 0xfffffffffffffe00, max supported is 0x1fffff8000000)
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 9223372036854775808}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 9223372036854775808}}}
 {"return": {}}
 Job failed: Unsupported VDI image size (size is 0x8000000000000000, max supported is 0x1fffff8000000)
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 562949819203585}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 562949819203585}}}
 {"return": {}}
 Job failed: Unsupported VDI image size (size is 0x1fffff8000001, max supported is 0x1fffff8000000)
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
diff --git a/tests/qemu-iotests/212.out b/tests/qemu-iotests/212.out
index 01da467282..22810720cf 100644
--- a/tests/qemu-iotests/212.out
+++ b/tests/qemu-iotests/212.out
@@ -1,13 +1,13 @@
 === Successful image creation (defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "node_name": "imgfile"}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "node-name": "imgfile"}}
 {"return": {}}
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "imgfile", "size": 134217728}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "parallels", "file": "imgfile", "size": 134217728}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -18,12 +18,12 @@ virtual size: 128M (134217728 bytes)
 
 === Successful image creation (explicit defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 1048576, "driver": "parallels", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels"}, "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 1048576, "driver": "parallels", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels"}, "size": 67108864}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -34,12 +34,12 @@ virtual size: 64M (67108864 bytes)
 
 === Successful image creation (with non-default options) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 65536, "driver": "parallels", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels"}, "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 65536, "driver": "parallels", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels"}, "size": 33554432}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -50,7 +50,7 @@ virtual size: 32M (33554432 bytes)
 
 === Invalid BlockdevRef ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "this doesn't exist", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "parallels", "file": "this doesn't exist", "size": 33554432}}}
 {"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -58,7 +58,7 @@ Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exi
 
 === Zero size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -69,7 +69,7 @@ virtual size: 0 (0 bytes)
 
 === Maximum size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 4503599627369984}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 4503599627369984}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -80,31 +80,31 @@ virtual size: 4096T (4503599627369984 bytes)
 
 === Invalid sizes ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 1234}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 1234}}}
 {"return": {}}
 Job failed: Image size must be a multiple of 512 bytes
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 18446744073709551104}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 18446744073709551104}}}
 {"return": {}}
 Job failed: Image size is too large for this cluster size
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 9223372036854775808}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 9223372036854775808}}}
 {"return": {}}
 Job failed: Image size is too large for this cluster size
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 9223372036854775296}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 9223372036854775296}}}
 {"return": {}}
 Job failed: Image size is too large for this cluster size
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 4503599627370497}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 4503599627370497}}}
 {"return": {}}
 Job failed: Image size is too large for this cluster size
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -112,43 +112,43 @@ Job failed: Image size is too large for this cluster size
 
 === Invalid cluster size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 1234, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 1234, "driver": "parallels", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Cluster size must be a multiple of 512 bytes
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 128, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 128, "driver": "parallels", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Cluster size must be a multiple of 512 bytes
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 4294967296, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 4294967296, "driver": "parallels", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Cluster size is too large
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 9223372036854775808, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 9223372036854775808, "driver": "parallels", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Cluster size is too large
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 18446744073709551104, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 18446744073709551104, "driver": "parallels", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Cluster size is too large
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 0, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 0, "driver": "parallels", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Image size is too large for this cluster size
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 512, "driver": "parallels", "file": "node0", "size": 281474976710656}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"cluster-size": 512, "driver": "parallels", "file": "node0", "size": 281474976710656}}}
 {"return": {}}
 Job failed: Image size is too large for this cluster size
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
diff --git a/tests/qemu-iotests/213.out b/tests/qemu-iotests/213.out
index 0c9d65b2fe..169083e08e 100644
--- a/tests/qemu-iotests/213.out
+++ b/tests/qemu-iotests/213.out
@@ -1,13 +1,13 @@
 === Successful image creation (defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "node_name": "imgfile"}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "node-name": "imgfile"}}
 {"return": {}}
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "imgfile", "size": 134217728}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "imgfile", "size": 134217728}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -19,12 +19,12 @@ cluster_size: 8388608
 
 === Successful image creation (explicit defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 8388608, "block-state-zero": true, "driver": "vhdx", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx"}, "log-size": 1048576, "size": 67108864, "subformat": "dynamic"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"block-size": 8388608, "block-state-zero": true, "driver": "vhdx", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx"}, "log-size": 1048576, "size": 67108864, "subformat": "dynamic"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -36,12 +36,12 @@ cluster_size: 8388608
 
 === Successful image creation (with non-default options) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 268435456, "block-state-zero": false, "driver": "vhdx", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx"}, "log-size": 8388608, "size": 33554432, "subformat": "fixed"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"block-size": 268435456, "block-state-zero": false, "driver": "vhdx", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx"}, "log-size": 8388608, "size": 33554432, "subformat": "fixed"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -53,7 +53,7 @@ cluster_size: 268435456
 
 === Invalid BlockdevRef ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "this doesn't exist", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "this doesn't exist", "size": 33554432}}}
 {"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -61,7 +61,7 @@ Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exi
 
 === Zero size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -73,7 +73,7 @@ cluster_size: 8388608
 
 === Maximum size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 70368744177664}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 70368744177664}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -85,25 +85,25 @@ cluster_size: 67108864
 
 === Invalid sizes ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 18446744073709551104}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 18446744073709551104}}}
 {"return": {}}
 Job failed: Image size too large; max of 64TB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 9223372036854775808}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 9223372036854775808}}}
 {"return": {}}
 Job failed: Image size too large; max of 64TB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 9223372036854775296}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 9223372036854775296}}}
 {"return": {}}
 Job failed: Image size too large; max of 64TB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 70368744177665}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 70368744177665}}}
 {"return": {}}
 Job failed: Image size too large; max of 64TB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -111,31 +111,31 @@ Job failed: Image size too large; max of 64TB
 
 === Invalid block size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 1234567, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"block-size": 1234567, "driver": "vhdx", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Block size must be a multiple of 1 MB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 128, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"block-size": 128, "driver": "vhdx", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Block size must be a multiple of 1 MB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 3145728, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"block-size": 3145728, "driver": "vhdx", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Block size must be a power of two
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 536870912, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"block-size": 536870912, "driver": "vhdx", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Block size must not exceed 268435456
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 0, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"block-size": 0, "driver": "vhdx", "file": "node0", "size": 67108864}}}
 {"return": {}}
 Job failed: Block size must be a multiple of 1 MB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -143,25 +143,25 @@ Job failed: Block size must be a multiple of 1 MB
 
 === Invalid log size ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 1234567, "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 1234567, "size": 67108864}}}
 {"return": {}}
 Job failed: Log size must be a multiple of 1 MB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 128, "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 128, "size": 67108864}}}
 {"return": {}}
 Job failed: Log size must be a multiple of 1 MB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 4294967296, "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 4294967296, "size": 67108864}}}
 {"return": {}}
 Job failed: Log size must be smaller than 4 GB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 0, "size": 67108864}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 0, "size": 67108864}}}
 {"return": {}}
 Job failed: Log size must be a multiple of 1 MB
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
diff --git a/tests/qemu-iotests/237.out b/tests/qemu-iotests/237.out
index 241c864369..2aaa68f672 100644
--- a/tests/qemu-iotests/237.out
+++ b/tests/qemu-iotests/237.out
@@ -1,13 +1,13 @@
 === Successful image creation (defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk", "node_name": "imgfile"}}
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk", "node-name": "imgfile"}}
 {"return": {}}
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "file": "imgfile", "size": 5368709120}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "file": "imgfile", "size": 5368709120}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -29,12 +29,12 @@ Format specific information:
 
 === Successful image creation (inline blockdev-add, explicit defaults) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": "ide", "driver": "vmdk", "extents": [], "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk"}, "hwversion": "4", "size": 67108864, "subformat": "monolithicSparse", "zeroed-grain": false}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": "ide", "driver": "vmdk", "extents": [], "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk"}, "hwversion": "4", "size": 67108864, "subformat": "monolithicSparse", "zeroed-grain": false}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -56,12 +56,12 @@ Format specific information:
 
 === Successful image creation (with non-default options) ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk", "size": 0}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk", "size": 0}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": "buslogic", "driver": "vmdk", "extents": [], "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk"}, "size": 33554432, "subformat": "monolithicSparse", "zeroed-grain": true}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": "buslogic", "driver": "vmdk", "extents": [], "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vmdk"}, "size": 33554432, "subformat": "monolithicSparse", "zeroed-grain": true}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -83,7 +83,7 @@ Format specific information:
 
 === Invalid BlockdevRef ===
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "file": "this doesn't exist", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "file": "this doesn't exist", "size": 33554432}}}
 {"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -93,38 +93,38 @@ Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exi
 
 == Valid adapter types ==
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": "ide", "driver": "vmdk", "file": "node0", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": "ide", "driver": "vmdk", "file": "node0", "size": 33554432}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": "buslogic", "driver": "vmdk", "file": "node0", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": "buslogic", "driver": "vmdk", "file": "node0", "size": 33554432}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": "lsilogic", "driver": "vmdk", "file": "node0", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": "lsilogic", "driver": "vmdk", "file": "node0", "size": 33554432}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": "legacyESX", "driver": "vmdk", "file": "node0", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": "legacyESX", "driver": "vmdk", "file": "node0", "size": 33554432}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
 == Invalid adapter types ==
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": "foo", "driver": "vmdk", "file": "node0", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": "foo", "driver": "vmdk", "file": "node0", "size": 33554432}}}
 {"error": {"class": "GenericError", "desc": "Invalid parameter 'foo'"}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": "IDE", "driver": "vmdk", "file": "node0", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": "IDE", "driver": "vmdk", "file": "node0", "size": 33554432}}}
 {"error": {"class": "GenericError", "desc": "Invalid parameter 'IDE'"}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": "legacyesx", "driver": "vmdk", "file": "node0", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": "legacyesx", "driver": "vmdk", "file": "node0", "size": 33554432}}}
 {"error": {"class": "GenericError", "desc": "Invalid parameter 'legacyesx'"}}
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"adapter-type": 1, "driver": "vmdk", "file": "node0", "size": 33554432}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"adapter-type": 1, "driver": "vmdk", "file": "node0", "size": 33554432}}}
 {"error": {"class": "GenericError", "desc": "Invalid parameter type for 'options.adapter-type', expected: string"}}
 
 === Other subformats ===
@@ -137,7 +137,7 @@ Formatting 'TEST_DIR/PID-t.vmdk.3', fmt=vmdk size=0 compat6=off hwversion=undefi
 
 == Missing extent ==
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "file": "node0", "size": 33554432, "subformat": "monolithicFlat"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "file": "node0", "size": 33554432, "subformat": "monolithicFlat"}}}
 {"return": {}}
 Job failed: Extent [0] not specified
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -145,14 +145,14 @@ Job failed: Extent [0] not specified
 
 == Correct extent ==
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 33554432, "subformat": "monolithicFlat"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 33554432, "subformat": "monolithicFlat"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
 == Extra extent ==
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1", "ext2", "ext3"], "file": "node0", "size": 512, "subformat": "monolithicFlat"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1", "ext2", "ext3"], "file": "node0", "size": 512, "subformat": "monolithicFlat"}}}
 {"return": {}}
 Job failed: List of extents contains unused extents
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -162,7 +162,7 @@ Job failed: List of extents contains unused extents
 
 = twoGbMaxExtentFlat 512 =
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 512, "subformat": "twoGbMaxExtentFlat"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 512, "subformat": "twoGbMaxExtentFlat"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -182,7 +182,7 @@ Format specific information:
 
 = twoGbMaxExtentSparse 512 =
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 512, "subformat": "twoGbMaxExtentSparse"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 512, "subformat": "twoGbMaxExtentSparse"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -204,7 +204,7 @@ Format specific information:
 
 = twoGbMaxExtentFlat 1073741824 =
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 1073741824, "subformat": "twoGbMaxExtentFlat"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 1073741824, "subformat": "twoGbMaxExtentFlat"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -224,7 +224,7 @@ Format specific information:
 
 = twoGbMaxExtentSparse 1073741824 =
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 1073741824, "subformat": "twoGbMaxExtentSparse"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 1073741824, "subformat": "twoGbMaxExtentSparse"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -246,7 +246,7 @@ Format specific information:
 
 = twoGbMaxExtentFlat 2147483648 =
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 2147483648, "subformat": "twoGbMaxExtentFlat"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 2147483648, "subformat": "twoGbMaxExtentFlat"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -266,7 +266,7 @@ Format specific information:
 
 = twoGbMaxExtentSparse 2147483648 =
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 2147483648, "subformat": "twoGbMaxExtentSparse"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1"], "file": "node0", "size": 2147483648, "subformat": "twoGbMaxExtentSparse"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -288,7 +288,7 @@ Format specific information:
 
 = twoGbMaxExtentFlat 5368709120 =
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1", "ext2", "ext3"], "file": "node0", "size": 5368709120, "subformat": "twoGbMaxExtentFlat"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1", "ext2", "ext3"], "file": "node0", "size": 5368709120, "subformat": "twoGbMaxExtentFlat"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
@@ -316,7 +316,7 @@ Format specific information:
 
 = twoGbMaxExtentSparse 5368709120 =
 
-{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vmdk", "extents": ["ext1", "ext2", "ext3"], "file": "node0", "size": 5368709120, "subformat": "twoGbMaxExtentSparse"}}}
+{"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "extents": ["ext1", "ext2", "ext3"], "file": "node0", "size": 5368709120, "subformat": "twoGbMaxExtentSparse"}}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index cba91a9927..387e026556 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -76,14 +76,16 @@ def qemu_img(*args):
         sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
     return exitcode
 
-def ordered_qmp(qmsg):
+def ordered_qmp(qmsg, conv_keys=True):
     # Dictionaries are not ordered prior to 3.6, therefore:
     if isinstance(qmsg, list):
         return [ordered_qmp(atom) for atom in qmsg]
     if isinstance(qmsg, dict):
         od = OrderedDict()
         for k, v in sorted(qmsg.items()):
-            od[k] = ordered_qmp(v)
+            if conv_keys:
+                k = k.replace('_', '-')
+            od[k] = ordered_qmp(v, conv_keys=False)
         return od
     return qmsg
 
-- 
2.20.1

[Qemu-devel] [PULL 68/71] qcow2: include LUKS payload overhead in qemu-img measure

[Kevin Wolf]

From: Stefan Hajnoczi <stefanha@redhat.com>

LUKS encryption reserves clusters for its own payload data.  The size of
this area must be included in the qemu-img measure calculation so that
we arrive at the correct minimum required image size.

(Ab)use the qcrypto_block_create() API to determine the payload
overhead.  We discard the payload data that qcrypto thinks will be
written to the image.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190218104525.23674-2-stefanha@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/qcow2.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 71 insertions(+), 1 deletion(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index 92242fb14f..1de5e24613 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -4236,6 +4236,60 @@ static coroutine_fn int qcow2_co_flush_to_os(BlockDriverState *bs)
     return ret;
 }
 
+static ssize_t qcow2_measure_crypto_hdr_init_func(QCryptoBlock *block,
+        size_t headerlen, void *opaque, Error **errp)
+{
+    size_t *headerlenp = opaque;
+
+    /* Stash away the payload size */
+    *headerlenp = headerlen;
+    return 0;
+}
+
+static ssize_t qcow2_measure_crypto_hdr_write_func(QCryptoBlock *block,
+        size_t offset, const uint8_t *buf, size_t buflen,
+        void *opaque, Error **errp)
+{
+    /* Discard the bytes, we're not actually writing to an image */
+    return buflen;
+}
+
+/* Determine the number of bytes for the LUKS payload */
+static bool qcow2_measure_luks_headerlen(QemuOpts *opts, size_t *len,
+                                         Error **errp)
+{
+    QDict *opts_qdict;
+    QDict *cryptoopts_qdict;
+    QCryptoBlockCreateOptions *cryptoopts;
+    QCryptoBlock *crypto;
+
+    /* Extract "encrypt." options into a qdict */
+    opts_qdict = qemu_opts_to_qdict(opts, NULL);
+    qdict_extract_subqdict(opts_qdict, &cryptoopts_qdict, "encrypt.");
+    qobject_unref(opts_qdict);
+
+    /* Build QCryptoBlockCreateOptions object from qdict */
+    qdict_put_str(cryptoopts_qdict, "format", "luks");
+    cryptoopts = block_crypto_create_opts_init(cryptoopts_qdict, errp);
+    qobject_unref(cryptoopts_qdict);
+    if (!cryptoopts) {
+        return false;
+    }
+
+    /* Fake LUKS creation in order to determine the payload size */
+    crypto = qcrypto_block_create(cryptoopts, "encrypt.",
+                                  qcow2_measure_crypto_hdr_init_func,
+                                  qcow2_measure_crypto_hdr_write_func,
+                                  len, errp);
+    qapi_free_QCryptoBlockCreateOptions(cryptoopts);
+    if (!crypto) {
+        return false;
+    }
+
+    qcrypto_block_free(crypto);
+    return true;
+}
+
 static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
                                        Error **errp)
 {
@@ -4245,11 +4299,13 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
     uint64_t virtual_size; /* disk size as seen by guest */
     uint64_t refcount_bits;
     uint64_t l2_tables;
+    uint64_t luks_payload_size = 0;
     size_t cluster_size;
     int version;
     char *optstr;
     PreallocMode prealloc;
     bool has_backing_file;
+    bool has_luks;
 
     /* Parse image creation options */
     cluster_size = qcow2_opt_get_cluster_size_del(opts, &local_err);
@@ -4279,6 +4335,20 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
     has_backing_file = !!optstr;
     g_free(optstr);
 
+    optstr = qemu_opt_get_del(opts, BLOCK_OPT_ENCRYPT_FORMAT);
+    has_luks = optstr && strcmp(optstr, "luks") == 0;
+    g_free(optstr);
+
+    if (has_luks) {
+        size_t headerlen;
+
+        if (!qcow2_measure_luks_headerlen(opts, &headerlen, &local_err)) {
+            goto err;
+        }
+
+        luks_payload_size = ROUND_UP(headerlen, cluster_size);
+    }
+
     virtual_size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
     virtual_size = ROUND_UP(virtual_size, cluster_size);
 
@@ -4349,7 +4419,7 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
     info = g_new(BlockMeasureInfo, 1);
     info->fully_allocated =
         qcow2_calc_prealloc_size(virtual_size, cluster_size,
-                                 ctz32(refcount_bits));
+                                 ctz32(refcount_bits)) + luks_payload_size;
 
     /* Remove data clusters that are not required.  This overestimates the
      * required size because metadata needed for the fully allocated file is
-- 
2.20.1

[Qemu-devel] [PULL 69/71] iotests: add LUKS payload overhead to 178 qemu-img measure test

[Kevin Wolf]

From: Stefan Hajnoczi <stefanha@redhat.com>

The previous patch includes the LUKS payload overhead into the qemu-img
measure calculation for qcow2.  Update qemu-iotests 178 to exercise this
new code path.

Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20190218104525.23674-3-stefanha@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/178           |  8 ++++++++
 tests/qemu-iotests/178.out.qcow2 | 24 ++++++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178
index 3f4b4a4564..927bf06e4d 100755
--- a/tests/qemu-iotests/178
+++ b/tests/qemu-iotests/178
@@ -142,6 +142,14 @@ for ofmt in human json; do
             # The backing file doesn't need to exist :)
             $QEMU_IMG measure --output=$ofmt -o backing_file=x \
                               -f "$fmt" -O "$IMGFMT" "$TEST_IMG"
+
+            echo
+            echo "== $fmt input image and LUKS encryption =="
+            echo
+            $QEMU_IMG measure --output=$ofmt \
+                              --object secret,id=sec0,data=base \
+                              -o encrypt.format=luks,encrypt.key-secret=sec0,encrypt.iter-time=10 \
+                              -f "$fmt" -O "$IMGFMT" "$TEST_IMG"
         fi
 
         echo
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
index d42d4a4597..55a8dc926f 100644
--- a/tests/qemu-iotests/178.out.qcow2
+++ b/tests/qemu-iotests/178.out.qcow2
@@ -68,6 +68,11 @@ converted image file size in bytes: 458752
 required size: 1074135040
 fully allocated size: 1074135040
 
+== qcow2 input image and LUKS encryption ==
+
+required size: 2686976
+fully allocated size: 1076232192
+
 == qcow2 input image and preallocation (human) ==
 
 required size: 1074135040
@@ -114,6 +119,11 @@ converted image file size in bytes: 524288
 required size: 1074135040
 fully allocated size: 1074135040
 
+== raw input image and LUKS encryption ==
+
+required size: 2686976
+fully allocated size: 1076232192
+
 == raw input image and preallocation (human) ==
 
 required size: 1074135040
@@ -205,6 +215,13 @@ converted image file size in bytes: 458752
     "fully-allocated": 1074135040
 }
 
+== qcow2 input image and LUKS encryption ==
+
+{
+    "required": 2686976,
+    "fully-allocated": 1076232192
+}
+
 == qcow2 input image and preallocation (json) ==
 
 {
@@ -263,6 +280,13 @@ converted image file size in bytes: 524288
     "fully-allocated": 1074135040
 }
 
+== raw input image and LUKS encryption ==
+
+{
+    "required": 2686976,
+    "fully-allocated": 1076232192
+}
+
 == raw input image and preallocation (json) ==
 
 {
-- 
2.20.1

[Qemu-devel] [PULL 70/71] vmdk: false positive of compat6 with hwversion not set

[Kevin Wolf]

From: yuchenlin <yuchenlin@synology.com>

In vmdk_co_create_opts, when it finds hw_version is undefined, it will
set it to 4, which misleading the compat6 and hwversion in
vmdk_co_do_create. Simply set hw_version to NULL after free, let
the logic in vmdk_co_do_create to decide the value of hw_version.

This bug can be reproduced by:

$ qemu-img convert -O vmdk -o subformat=streamOptimized,compat6
/home/yuchenlin/syno.qcow2 /home/yuchenlin/syno.vmdk

qemu-img: /home/yuchenlin/syno.vmdk: error while converting vmdk:
compat6 cannot be enabled with hwversion set

Signed-off-by: yuchenlin <yuchenlin@synology.com>
Message-id: 20190221110805.28239-1-yuchenlin@synology.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index 91345babb5..f4e68aa00b 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -2267,7 +2267,7 @@ static int coroutine_fn vmdk_co_create_opts(const char *filename, QemuOpts *opts
     compat6 = qemu_opt_get_bool_del(opts, BLOCK_OPT_COMPAT6, false);
     if (strcmp(hw_version, "undefined") == 0) {
         g_free(hw_version);
-        hw_version = g_strdup("4");
+        hw_version = NULL;
     }
     fmt = qemu_opt_get_del(opts, BLOCK_OPT_SUBFMT);
     zeroed_grain = qemu_opt_get_bool_del(opts, BLOCK_OPT_ZEROED_GRAIN, false);
-- 
2.20.1

[Qemu-devel] [PULL 71/71] iotests: Skip 211 on insufficient memory

[Kevin Wolf]

From: Max Reitz <mreitz@redhat.com>

VDI keeps the whole bitmap in memory, and the maximum size (which is
tested here) is 2 GB.  This may not be available on all machines, and it
rarely is available when running a 32 bit build.

Fix this by making VM.run_job() return the error string if an error
occurred, and checking whether that contains "Could not allocate bmap"
in 211.  If so, the test is skipped.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190218180646.30282-1-mreitz@redhat.com
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/211        | 4 +++-
 tests/qemu-iotests/iotests.py | 5 ++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/211 b/tests/qemu-iotests/211
index 5d285450b5..6afc894f76 100755
--- a/tests/qemu-iotests/211
+++ b/tests/qemu-iotests/211
@@ -32,7 +32,9 @@ def blockdev_create(vm, options):
 
     if 'return' in result:
         assert result['return'] == {}
-        vm.run_job('job0')
+        error = vm.run_job('job0')
+        if error and 'Could not allocate bmap' in error:
+            iotests.notrun('Insufficient memory')
     iotests.log("")
 
 with iotests.FilePath('t.vdi') as disk_path, \
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 387e026556..4910fb2005 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -532,7 +532,9 @@ class VM(qtest.QEMUQtestMachine):
         log(result, filters, indent=indent)
         return result
 
+    # Returns None on success, and an error string on failure
     def run_job(self, job, auto_finalize=True, auto_dismiss=False):
+        error = None
         while True:
             for ev in self.get_qmp_events_filtered(wait=True):
                 if ev['event'] == 'JOB_STATUS_CHANGE':
@@ -541,13 +543,14 @@ class VM(qtest.QEMUQtestMachine):
                         result = self.qmp('query-jobs')
                         for j in result['return']:
                             if j['id'] == job:
+                                error = j['error']
                                 log('Job failed: %s' % (j['error']))
                     elif status == 'pending' and not auto_finalize:
                         self.qmp_log('job-finalize', id=job)
                     elif status == 'concluded' and not auto_dismiss:
                         self.qmp_log('job-dismiss', id=job)
                     elif status == 'null':
-                        return
+                        return error
                 else:
                     iotests.log(ev)
 
-- 
2.20.1

Re: [Qemu-devel] [PULL 00/71] Block layer patches

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190225152053.15976-1-kwolf@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20190225152053.15976-1-kwolf@redhat.com
Subject: [Qemu-devel] [PULL 00/71] Block layer patches
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
10b32f5be0 iotests: Skip 211 on insufficient memory
a05be02caa vmdk: false positive of compat6 with hwversion not set
c64025e1f5 iotests: add LUKS payload overhead to 178 qemu-img measure test
c80e628e71 qcow2: include LUKS payload overhead in qemu-img measure
6beaf9e3ab iotests.py: s/_/-/g on keys in qmp_log()
62dbee3637 iotests: Let 045 be run concurrently
24ce684632 iotests: Filter SSH paths
95eecc9714 iotests.py: Filter filename in any string value
7889a8c497 iotests.py: Add is_str()
d505dbed82 iotests: Fix 207 to use QMP filters for qmp_log
96b6f085bb iotests: Fix 232 for LUKS
dd6602fadd iotests: Remove superfluous rm from 232
10dac5f479 iotests: Fix 237 for Python 2.x
c2f383ed9a iotests: Re-add filename filters
470731a3b6 iotests: Test json:{} filenames of internal BDSs
d580ab483f block: BDS options may lack the "driver" option
64f4152927 block/null: Generate filename even with latency-ns
ee9187663b block/curl: Implement bdrv_refresh_filename()
9db1762a1a block/curl: Harmonize option defaults
f66bd55226 block/nvme: Fix bdrv_refresh_filename()
630f7fffac block: Do not copy exact_filename from format file
4e48b99d44 block: Purify .bdrv_refresh_filename()
8592be931f block: Generically refresh runtime options
78eaed8ed7 block: Add BlockDriver.bdrv_gather_child_options
8ecb70ac62 block: Add strong_runtime_opts to BlockDriver
2c2612b1ad iotests: Add quorum case to test 110
3606fa8493 block: Use bdrv_dirname() for relative filenames
c2211aa7d0 block/nfs: Implement bdrv_dirname()
2405785d83 block/nbd: Make bdrv_dirname() return NULL
976dead85d quorum: Make bdrv_dirname() return NULL
da78f0d647 blkverify: Make bdrv_dirname() return NULL
d0cdd12eeb block: Add bdrv_dirname()
ce5fba1b39 block: Fix bdrv_find_backing_image()
980761fa98 block: Add bdrv_make_absolute_filename()
685b9b0e60 block: bdrv_get_full_backing_filename's ret. val.
5d9bc79de6 block: bdrv_get_full_backing_filename_from_...'s ret. val.
6d6f623112 block: Make path_combine() return the path
f487ae593b iotests: Add test for backing file overrides
b567781927 iotests.py: Add node_info()
cad0e50e0b iotests.py: Add filter_imgfmt()
62bdce1333 block: Respect backing bs in bdrv_refresh_filename
bee568962b block: Add BDS.auto_backing_file
c501017de0 block: Skip implicit nodes for filename info
c683acac55 block: Use children list in bdrv_refresh_filename
b5a7a2b366 block: Use bdrv_refresh_filename() to pull
c5dad9864a block/nvme: Remove QEMU_PACKED from naturally aligned NVMeRegs struct
1f4652d66b qcow2: Assert that L2 table offsets fit in the L1 table
be9088d03b tests: add test-bdrv-graph-mod
d0ccc31d4b block: fix bdrv_check_perm for non-tree subgraph
101679f23a block: improve should_update_child
7aca78a67d aio-posix: Assert that aio_poll() is always called in home thread
3eb02d8d80 block: Use normal drain for bdrv_set_aio_context()
a544d77ae2 test-bdrv-drain: AioContext switch in drained section
e92fa43053 block: Fix AioContext switch for drained node
5309365338 block: Don't poll in bdrv_set_aio_context()
a36da7b9b7 nbd: Increase bs->in_flight during AioContext switch
375468b200 nbd: Use low-level QIOChannel API in nbd_read_eof()
16cef5961a nbd: Move nbd_read_eof() to nbd/client.c
00f72b298c io: Remove redundant read/write_coroutine assignments
2c8cf32209 io: Make qio_channel_yield() interruptible
20b541225e nbd: Restrict connection_co reentrance
cdad3c5441 virtio-blk: Increase in_flight for request restart BH
733e2c4b4e block-backend: Make blk_inc/dec_in_flight public
b5b7f641ba qemu-img: fix error reporting for -object
bcf8a8589e commit: Replace commit_top_bs on failure after deleting the block job
84d8dbd9a7 block: don't set the same context
7425640d2e qcow2-snapshot: remove redundant find_snapshot_by_id_and_name call
328390d7f3 block/snapshot: remove bdrv_snapshot_delete_by_id_or_name
e1c3af4a98 block/snapshot.c: eliminate use of ID input in snapshot operations
24361188ea MAINTAINERS: Remove myself as block maintainer
325eaaa06e MAINTAINERS: Replace myself with John Snow for block jobs

=== OUTPUT BEGIN ===
1/71 Checking commit 325eaaa06ec1 (MAINTAINERS: Replace myself with John Snow for block jobs)
2/71 Checking commit 24361188ea3d (MAINTAINERS: Remove myself as block maintainer)
3/71 Checking commit e1c3af4a9878 (block/snapshot.c: eliminate use of ID input in snapshot operations)
4/71 Checking commit 328390d7f386 (block/snapshot: remove bdrv_snapshot_delete_by_id_or_name)
5/71 Checking commit 7425640d2e76 (qcow2-snapshot: remove redundant find_snapshot_by_id_and_name call)
6/71 Checking commit 84d8dbd9a77c (block: don't set the same context)
7/71 Checking commit bcf8a8589e6d (commit: Replace commit_top_bs on failure after deleting the block job)
WARNING: Block comments use a leading /* on a separate line
#28: FILE: block/commit.c:373:
+    /* commit_top_bs has to be replaced after deleting the block job,

WARNING: Block comments use a trailing */ on a separate line
#29: FILE: block/commit.c:374:
+     * otherwise this would fail because of lack of permissions. */

total: 0 errors, 2 warnings, 13 lines checked

Patch 7/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
8/71 Checking commit b5b7f641ba57 (qemu-img: fix error reporting for -object)
9/71 Checking commit 733e2c4b4e0a (block-backend: Make blk_inc/dec_in_flight public)
10/71 Checking commit cdad3c54417d (virtio-blk: Increase in_flight for request restart BH)
WARNING: Block comments use a leading /* on a separate line
#39: FILE: hw/block/virtio-blk.c:856:
+        /* FIXME The data plane is not started yet, so these requests are

WARNING: Block comments use a trailing */ on a separate line
#40: FILE: hw/block/virtio-blk.c:857:
+         * processed in the main thread. */

total: 0 errors, 2 warnings, 18 lines checked

Patch 10/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
11/71 Checking commit 20b541225e54 (nbd: Restrict connection_co reentrance)
12/71 Checking commit 2c8cf32209e3 (io: Make qio_channel_yield() interruptible)
WARNING: Block comments use a leading /* on a separate line
#43: FILE: io/channel.c:473:
+    /* Allow interrupting the operation by reentering the coroutine other than

WARNING: Block comments use a trailing */ on a separate line
#44: FILE: io/channel.c:474:
+     * through the aio_fd_handlers. */

total: 0 errors, 2 warnings, 32 lines checked

Patch 12/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
13/71 Checking commit 00f72b298cfc (io: Remove redundant read/write_coroutine assignments)
14/71 Checking commit 16cef5961a11 (nbd: Move nbd_read_eof() to nbd/client.c)
WARNING: Block comments use a leading /* on a separate line
#38: FILE: nbd/client.c:1390:
+/* nbd_read_eof

total: 0 errors, 1 warnings, 67 lines checked

Patch 14/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
15/71 Checking commit 375468b20082 (nbd: Use low-level QIOChannel API in nbd_read_eof())
ERROR: "(foo*)" should be "(foo *)"
#100: FILE: nbd/client.c:1427:
+        buffer = (uint8_t*) buffer + len;

total: 1 errors, 0 warnings, 93 lines checked

Patch 15/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

16/71 Checking commit a36da7b9b7e5 (nbd: Increase bs->in_flight during AioContext switch)
WARNING: Block comments use a leading /* on a separate line
#27: FILE: block/nbd-client.c:985:
+    /* The node is still drained, so we know the coroutine has yielded in

WARNING: Block comments use a trailing */ on a separate line
#30: FILE: block/nbd-client.c:988:
+     * coroutine.*/

WARNING: Block comments use a leading /* on a separate line
#45: FILE: block/nbd-client.c:1001:
+    /* Need to wait here for the BH to run because the BH must run while the

WARNING: Block comments use a trailing */ on a separate line
#46: FILE: block/nbd-client.c:1002:
+     * node is still drained. */

total: 0 errors, 4 warnings, 32 lines checked

Patch 16/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
17/71 Checking commit 530936533806 (block: Don't poll in bdrv_set_aio_context())
18/71 Checking commit e92fa43053fe (block: Fix AioContext switch for drained node)
19/71 Checking commit a544d77ae25d (test-bdrv-drain: AioContext switch in drained section)
20/71 Checking commit 3eb02d8d80db (block: Use normal drain for bdrv_set_aio_context())
WARNING: Block comments use a leading /* on a separate line
#22: FILE: block.c:5271:
+/* The caller must own the AioContext lock for the old AioContext of bs, but it

WARNING: Block comments use a trailing */ on a separate line
#24: FILE: block.c:5273:
+ * the same as the current context of bs). */

total: 0 errors, 2 warnings, 32 lines checked

Patch 20/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
21/71 Checking commit 7aca78a67ddb (aio-posix: Assert that aio_poll() is always called in home thread)
22/71 Checking commit 101679f23a28 (block: improve should_update_child)
23/71 Checking commit d0ccc31d4b0d (block: fix bdrv_check_perm for non-tree subgraph)
24/71 Checking commit be9088d03b04 (tests: add test-bdrv-graph-mod)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#34: 
new file mode 100644

total: 0 errors, 1 warnings, 212 lines checked

Patch 24/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
25/71 Checking commit 1f4652d66b23 (qcow2: Assert that L2 table offsets fit in the L1 table)
26/71 Checking commit c5dad9864af9 (block/nvme: Remove QEMU_PACKED from naturally aligned NVMeRegs struct)
27/71 Checking commit b5a7a2b3664e (block: Use bdrv_refresh_filename() to pull)
WARNING: Block comments use a leading /* on a separate line
#269: FILE: blockdev.c:3354:
+        /* This strcmp() is just a shortcut, there is no need to

WARNING: Block comments use a trailing */ on a separate line
#272: FILE: blockdev.c:3357:
+         * return @bs. */

total: 0 errors, 2 warnings, 246 lines checked

Patch 27/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
28/71 Checking commit c683acac55d9 (block: Use children list in bdrv_refresh_filename)
WARNING: Block comments use a leading /* on a separate line
#39: FILE: block.c:5546:
+    /* This BDS's file name may depend on any of its children's file names, so

WARNING: Block comments use a trailing */ on a separate line
#40: FILE: block.c:5547:
+     * refresh those first */

total: 0 errors, 2 warnings, 60 lines checked

Patch 28/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
29/71 Checking commit c501017de0e5 (block: Skip implicit nodes for filename info)
30/71 Checking commit bee568962b08 (block: Add BDS.auto_backing_file)
WARNING: Block comments use a leading /* on a separate line
#86: FILE: block.c:2401:
+            /* If the user specifies options that do not modify the

WARNING: Block comments use a trailing */ on a separate line
#91: FILE: block.c:2406:
+             * schema forces the user to specify everything). */

WARNING: Block comments use a leading /* on a separate line
#230: FILE: include/block/block_int.h:705:
+    /* The backing filename indicated by the image header; if we ever

WARNING: Block comments use a trailing */ on a separate line
#232: FILE: include/block/block_int.h:707:
+     * filename (i.e. after a bdrv_refresh_filename() run). */

total: 0 errors, 4 warnings, 133 lines checked

Patch 30/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
31/71 Checking commit 62bdce13336f (block: Respect backing bs in bdrv_refresh_filename)
WARNING: Block comments use a leading /* on a separate line
#39: FILE: block.c:5543:
+/* Note: This function may return false positives; it may return true

WARNING: Block comments use a trailing */ on a separate line
#41: FILE: block.c:5545:
+ * would result in exactly bs->backing. */

WARNING: Block comments use a leading /* on a separate line
#48: FILE: block.c:5552:
+        /* No backing BDS, so if the image header reports any backing

WARNING: Block comments use a trailing */ on a separate line
#49: FILE: block.c:5553:
+         * file, it must have been suppressed */

WARNING: Block comments use a leading /* on a separate line
#72: FILE: block.c:5604:
+        /* Without I/O, the backing file does not change anything.

WARNING: Block comments use a trailing */ on a separate line
#75: FILE: block.c:5607:
+         * it technically has been. */

total: 0 errors, 6 warnings, 136 lines checked

Patch 31/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
32/71 Checking commit cad0e50e0b5c (iotests.py: Add filter_imgfmt())
33/71 Checking commit b5677819277b (iotests.py: Add node_info())
34/71 Checking commit f487ae593bd8 (iotests: Add test for backing file overrides)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#13: 
new file mode 100755

total: 0 errors, 1 warnings, 333 lines checked

Patch 34/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
35/71 Checking commit 6d6f62311299 (block: Make path_combine() return the path)
WARNING: Block comments use a leading /* on a separate line
#30: FILE: block.c:155:
+/* if filename is absolute, just return its duplicate. Otherwise, build a

total: 0 errors, 1 warnings, 148 lines checked

Patch 35/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
36/71 Checking commit 5d9bc79de657 (block: bdrv_get_full_backing_filename_from_...'s ret. val.)
37/71 Checking commit 685b9b0e6090 (block: bdrv_get_full_backing_filename's ret. val.)
38/71 Checking commit 980761fa981a (block: Add bdrv_make_absolute_filename())
39/71 Checking commit ce5fba1b39af (block: Fix bdrv_find_backing_image())
40/71 Checking commit d0cdd12eeb46 (block: Add bdrv_dirname())
41/71 Checking commit da78f0d6478a (blkverify: Make bdrv_dirname() return NULL)
WARNING: Block comments use a leading /* on a separate line
#29: FILE: block/blkverify.c:318:
+    /* In general, there are two BDSs with different dirnames below this one;

WARNING: Block comments use a trailing */ on a separate line
#31: FILE: block/blkverify.c:320:
+     * chance). Therefore, to be consistent, just always return NULL. */

total: 0 errors, 2 warnings, 22 lines checked

Patch 41/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
42/71 Checking commit 976dead85d94 (quorum: Make bdrv_dirname() return NULL)
WARNING: Block comments use a leading /* on a separate line
#29: FILE: block/quorum.c:1099:
+    /* In general, there are multiple BDSs with different dirnames below this

WARNING: Block comments use a trailing */ on a separate line
#32: FILE: block/quorum.c:1102:
+     * always return NULL. */

total: 0 errors, 2 warnings, 23 lines checked

Patch 42/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
43/71 Checking commit 2405785d83d3 (block/nbd: Make bdrv_dirname() return NULL)
WARNING: Block comments use a leading /* on a separate line
#29: FILE: block/nbd.c:533:
+    /* The generic bdrv_dirname() implementation is able to work out some

WARNING: Block comments use a trailing */ on a separate line
#32: FILE: block/nbd.c:536:
+     * directory names. */

total: 0 errors, 2 warnings, 37 lines checked

Patch 43/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
44/71 Checking commit c2211aa7d0f1 (block/nfs: Implement bdrv_dirname())
45/71 Checking commit 3606fa84935d (block: Use bdrv_dirname() for relative filenames)
46/71 Checking commit 2c2612b1ad10 (iotests: Add quorum case to test 110)
47/71 Checking commit 8ecb70ac6205 (block: Add strong_runtime_opts to BlockDriver)
WARNING: Block comments use a leading /* on a separate line
#763: FILE: include/block/block_int.h:521:
+    /* Pointer to a NULL-terminated array of names of strong options

WARNING: Block comments use a trailing */ on a separate line
#767: FILE: include/block/block_int.h:525:
+     * "filename" and "driver" are always considered strong. */

total: 0 errors, 2 warnings, 599 lines checked

Patch 47/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
48/71 Checking commit 78eaed8ed7ee (block: Add BlockDriver.bdrv_gather_child_options)
49/71 Checking commit 8592be931fbb (block: Generically refresh runtime options)
WARNING: Block comments use a leading /* on a separate line
#106: FILE: block.c:5612:
+        /* While "driver" and "filename" need to be included in a JSON filename,

WARNING: Block comments use a trailing */ on a separate line
#107: FILE: block.c:5613:
+         * their existence does not prohibit generation of a plain filename. */

WARNING: Block comments use a leading /* on a separate line
#130: FILE: block.c:5805:
+        /* Some block drivers may not want to present all of their children's

WARNING: Block comments use a trailing */ on a separate line
#131: FILE: block.c:5806:
+         * options, or name them differently from BdrvChild.name */

total: 0 errors, 4 warnings, 159 lines checked

Patch 49/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
50/71 Checking commit 4e48b99d4455 (block: Purify .bdrv_refresh_filename())
WARNING: Block comments use a leading /* on a separate line
#75: FILE: block.c:5657:
+    bool generate_json_filename; /* Whether our default implementation should

WARNING: Block comments use * on subsequent lines
#76: FILE: block.c:5658:
+    bool generate_json_filename; /* Whether our default implementation should
+                                    fill exact_filename (false) or not (true) */

WARNING: Block comments use a trailing */ on a separate line
#76: FILE: block.c:5658:
+                                    fill exact_filename (false) or not (true) */

WARNING: Block comments use a leading /* on a separate line
#178: FILE: block.c:5724:
+        /* Obsolete information is of no use here, so drop the old file name

WARNING: Block comments use a trailing */ on a separate line
#179: FILE: block.c:5725:
+         * information before refreshing it */

WARNING: Block comments use a leading /* on a separate line
#188: FILE: block.c:5734:
+        /* If no specific options have been given for this BDS, the filename of

WARNING: Block comments use a trailing */ on a separate line
#189: FILE: block.c:5735:
+         * the underlying file should suffice for this one as well */

WARNING: Block comments use a leading /* on a separate line
#241: FILE: block/blkdebug.c:827:
+        /* Real child options are under "image", but "x-image" may

WARNING: Block comments use a trailing */ on a separate line
#242: FILE: block/blkdebug.c:828:
+         * contain a filename */

total: 0 errors, 9 warnings, 528 lines checked

Patch 50/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
51/71 Checking commit 630f7fffac82 (block: Do not copy exact_filename from format file)
52/71 Checking commit f66bd5522628 (block/nvme: Fix bdrv_refresh_filename())
53/71 Checking commit 9db1762a1aab (block/curl: Harmonize option defaults)
54/71 Checking commit ee9187663b75 (block/curl: Implement bdrv_refresh_filename())
WARNING: Block comments use a leading /* on a separate line
#25: FILE: block/curl.c:957:
+    /* "readahead" and "timeout" do not change the guest-visible data,

WARNING: Block comments use a trailing */ on a separate line
#26: FILE: block/curl.c:958:
+     * so ignore them */

total: 0 errors, 2 warnings, 51 lines checked

Patch 54/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
55/71 Checking commit 64f4152927d9 (block/null: Generate filename even with latency-ns)
56/71 Checking commit d580ab483f6f (block: BDS options may lack the "driver" option)
WARNING: Block comments use a leading /* on a separate line
#26: FILE: block.c:5622:
+        /* Drivers created with bdrv_new_open_driver() may not have a

WARNING: Block comments use a trailing */ on a separate line
#27: FILE: block.c:5623:
+         * @driver option.  Add it here. */

total: 0 errors, 2 warnings, 12 lines checked

Patch 56/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
57/71 Checking commit 470731a3b696 (iotests: Test json:{} filenames of internal BDSs)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#13: 
new file mode 100755

total: 0 errors, 1 warnings, 164 lines checked

Patch 57/71 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
58/71 Checking commit c2f383ed9ab1 (iotests: Re-add filename filters)
59/71 Checking commit 10dac5f479c1 (iotests: Fix 237 for Python 2.x)
60/71 Checking commit dd6602faddfb (iotests: Remove superfluous rm from 232)
61/71 Checking commit 96b6f085bb5f (iotests: Fix 232 for LUKS)
62/71 Checking commit d505dbed82ef (iotests: Fix 207 to use QMP filters for qmp_log)
63/71 Checking commit 7889a8c497f7 (iotests.py: Add is_str())
64/71 Checking commit 95eecc971470 (iotests.py: Filter filename in any string value)
65/71 Checking commit 24ce684632b5 (iotests: Filter SSH paths)
66/71 Checking commit 62dbee3637e4 (iotests: Let 045 be run concurrently)
67/71 Checking commit 6beaf9e3abca (iotests.py: s/_/-/g on keys in qmp_log())
68/71 Checking commit c80e628e7160 (qcow2: include LUKS payload overhead in qemu-img measure)
69/71 Checking commit c64025e1f51f (iotests: add LUKS payload overhead to 178 qemu-img measure test)
70/71 Checking commit a05be02caa4b (vmdk: false positive of compat6 with hwversion not set)
71/71 Checking commit 10b32f5be0f3 (iotests: Skip 211 on insufficient memory)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190225152053.15976-1-kwolf@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PULL 00/71] Block layer patches

[Peter Maydell]

On Mon, 25 Feb 2019 at 15:21, Kevin Wolf <kwolf@redhat.com> wrote:
>
> The following changes since commit 59a568b57848b10e8a44518a889323f12ccdd8f4:
>
>   Merge remote-tracking branch 'remotes/kraxel/tags/vga-20190222-pull-request' into staging (2019-02-25 12:49:07 +0000)
>
> are available in the Git repository at:
>
>   git://repo.or.cz/qemu/kevin.git tags/for-upstream
>
> for you to fetch changes up to 1b967e9f348d48788a2ab481d45398b80ce71fa6:
>
>   Merge remote-tracking branch 'mreitz/tags/pull-block-2019-02-25' into queue-block (2019-02-25 15:16:57 +0100)
>
> ----------------------------------------------------------------
> Block layer patches:
>
> - Block graph change fixes (avoid loops, cope with non-tree graphs)
> - bdrv_set_aio_context() related fixes
> - HMP snapshot commands: Use only tag, not the ID to identify snapshots
> - qmeu-img, commit: Error path fixes
> - block/nvme: Build fix for gcc 9
> - MAINTAINERS updates
> - Fix various issues with bdrv_refresh_filename()
> - Fix various iotests
> - Include LUKS overhead in qemu-img measure for qcow2
> - A fix for vmdk's image creation interface
>

Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/4.0
for any user-visible changes.

-- PMM

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Vladimir Sementsov-Ogievskiy]

25.02.2019 18:19, Kevin Wolf wrote:
> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
> needs to be increased while the coroutine is waiting to be scheduled
> in the new AioContext after nbd_client_attach_aio_context().

Hi!

I have some questions, could you explain, please?

"bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
sure that connection_co yielded from nbd_read_eof, yes?

But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
bs->in_flight ?

> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>   block/nbd-client.c | 20 ++++++++++++++++++--
>   1 file changed, 18 insertions(+), 2 deletions(-)
> 
> diff --git a/block/nbd-client.c b/block/nbd-client.c
> index 60f38f0320..bfbaf7ebe9 100644
> --- a/block/nbd-client.c
> +++ b/block/nbd-client.c
> @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
>       qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
>   }
>   
> +static void nbd_client_attach_aio_context_bh(void *opaque)
> +{
> +    BlockDriverState *bs = opaque;
> +    NBDClientSession *client = nbd_get_client_session(bs);
> +
> +    /* The node is still drained, so we know the coroutine has yielded in
> +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
> +     * entered for the first time. Both places are safe for entering the
> +     * coroutine.*/
> +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
> +    bdrv_dec_in_flight(bs);
> +}
> +
>   void nbd_client_attach_aio_context(BlockDriverState *bs,
>                                      AioContext *new_context)
>   {
>       NBDClientSession *client = nbd_get_client_session(bs);
>       qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
>   
> -    /* FIXME Really need a bdrv_inc_in_flight() here */
> -    aio_co_schedule(new_context, client->connection_co);
> +    bdrv_inc_in_flight(bs);
> +
> +    /* Need to wait here for the BH to run because the BH must run while the
> +     * node is still drained. */
> +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
>   }
>   
>   void nbd_client_close(BlockDriverState *bs)
> 


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Kevin Wolf]

Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 25.02.2019 18:19, Kevin Wolf wrote:
> > bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
> > needs to be increased while the coroutine is waiting to be scheduled
> > in the new AioContext after nbd_client_attach_aio_context().
> 
> Hi!
> 
> I have some questions, could you explain, please?
> 
> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
> sure that connection_co yielded from nbd_read_eof, yes?
> 
> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
> bs->in_flight ?

Without incrementing bs->in_flight, nothing would guarantee that
aio_poll() is called and the BH is actually executed before bdrv_drain()
returns.

Kevin

> > 
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> >   block/nbd-client.c | 20 ++++++++++++++++++--
> >   1 file changed, 18 insertions(+), 2 deletions(-)
> > 
> > diff --git a/block/nbd-client.c b/block/nbd-client.c
> > index 60f38f0320..bfbaf7ebe9 100644
> > --- a/block/nbd-client.c
> > +++ b/block/nbd-client.c
> > @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
> >       qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
> >   }
> >   
> > +static void nbd_client_attach_aio_context_bh(void *opaque)
> > +{
> > +    BlockDriverState *bs = opaque;
> > +    NBDClientSession *client = nbd_get_client_session(bs);
> > +
> > +    /* The node is still drained, so we know the coroutine has yielded in
> > +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
> > +     * entered for the first time. Both places are safe for entering the
> > +     * coroutine.*/
> > +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
> > +    bdrv_dec_in_flight(bs);
> > +}
> > +
> >   void nbd_client_attach_aio_context(BlockDriverState *bs,
> >                                      AioContext *new_context)
> >   {
> >       NBDClientSession *client = nbd_get_client_session(bs);
> >       qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
> >   
> > -    /* FIXME Really need a bdrv_inc_in_flight() here */
> > -    aio_co_schedule(new_context, client->connection_co);
> > +    bdrv_inc_in_flight(bs);
> > +
> > +    /* Need to wait here for the BH to run because the BH must run while the
> > +     * node is still drained. */
> > +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
> >   }
> >   
> >   void nbd_client_close(BlockDriverState *bs)
> > 
> 
> 
> -- 
> Best regards,
> Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Kevin Wolf]

Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 25.02.2019 18:19, Kevin Wolf wrote:
> > bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
> > needs to be increased while the coroutine is waiting to be scheduled
> > in the new AioContext after nbd_client_attach_aio_context().
> 
> Hi!
> 
> I have some questions, could you explain, please?
> 
> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
> sure that connection_co yielded from nbd_read_eof, yes?
> 
> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
> bs->in_flight ?

Without incrementing bs->in_flight, nothing would guarantee that
aio_poll() is called and the BH is actually executed before bdrv_drain()
returns.

Kevin

> > 
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> >   block/nbd-client.c | 20 ++++++++++++++++++--
> >   1 file changed, 18 insertions(+), 2 deletions(-)
> > 
> > diff --git a/block/nbd-client.c b/block/nbd-client.c
> > index 60f38f0320..bfbaf7ebe9 100644
> > --- a/block/nbd-client.c
> > +++ b/block/nbd-client.c
> > @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
> >       qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
> >   }
> >   
> > +static void nbd_client_attach_aio_context_bh(void *opaque)
> > +{
> > +    BlockDriverState *bs = opaque;
> > +    NBDClientSession *client = nbd_get_client_session(bs);
> > +
> > +    /* The node is still drained, so we know the coroutine has yielded in
> > +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
> > +     * entered for the first time. Both places are safe for entering the
> > +     * coroutine.*/
> > +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
> > +    bdrv_dec_in_flight(bs);
> > +}
> > +
> >   void nbd_client_attach_aio_context(BlockDriverState *bs,
> >                                      AioContext *new_context)
> >   {
> >       NBDClientSession *client = nbd_get_client_session(bs);
> >       qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
> >   
> > -    /* FIXME Really need a bdrv_inc_in_flight() here */
> > -    aio_co_schedule(new_context, client->connection_co);
> > +    bdrv_inc_in_flight(bs);
> > +
> > +    /* Need to wait here for the BH to run because the BH must run while the
> > +     * node is still drained. */
> > +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
> >   }
> >   
> >   void nbd_client_close(BlockDriverState *bs)
> > 
> 
> 
> -- 
> Best regards,
> Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Vladimir Sementsov-Ogievskiy]

11.04.2019 17:15, Kevin Wolf wrote:
> Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> 25.02.2019 18:19, Kevin Wolf wrote:
>>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
>>> needs to be increased while the coroutine is waiting to be scheduled
>>> in the new AioContext after nbd_client_attach_aio_context().
>>
>> Hi!
>>
>> I have some questions, could you explain, please?
>>
>> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
>> sure that connection_co yielded from nbd_read_eof, yes?
>>
>> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
>> bs->in_flight ?
> 
> Without incrementing bs->in_flight, nothing would guarantee that
> aio_poll() is called and the BH is actually executed before bdrv_drain()
> returns.

Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
drained, as you write in comment?

> 
>>>
>>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>>> ---
>>>    block/nbd-client.c | 20 ++++++++++++++++++--
>>>    1 file changed, 18 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/block/nbd-client.c b/block/nbd-client.c
>>> index 60f38f0320..bfbaf7ebe9 100644
>>> --- a/block/nbd-client.c
>>> +++ b/block/nbd-client.c
>>> @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
>>>        qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
>>>    }
>>>    
>>> +static void nbd_client_attach_aio_context_bh(void *opaque)
>>> +{
>>> +    BlockDriverState *bs = opaque;
>>> +    NBDClientSession *client = nbd_get_client_session(bs);
>>> +
>>> +    /* The node is still drained, so we know the coroutine has yielded in
>>> +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
>>> +     * entered for the first time. Both places are safe for entering the
>>> +     * coroutine.*/
>>> +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
>>> +    bdrv_dec_in_flight(bs);
>>> +}
>>> +
>>>    void nbd_client_attach_aio_context(BlockDriverState *bs,
>>>                                       AioContext *new_context)
>>>    {
>>>        NBDClientSession *client = nbd_get_client_session(bs);
>>>        qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
>>>    
>>> -    /* FIXME Really need a bdrv_inc_in_flight() here */
>>> -    aio_co_schedule(new_context, client->connection_co);
>>> +    bdrv_inc_in_flight(bs);
>>> +
>>> +    /* Need to wait here for the BH to run because the BH must run while the
>>> +     * node is still drained. */
>>> +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
>>>    }
>>>    
>>>    void nbd_client_close(BlockDriverState *bs)
>>>
>>
>>
>> -- 
>> Best regards,
>> Vladimir


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Vladimir Sementsov-Ogievskiy]

11.04.2019 17:15, Kevin Wolf wrote:
> Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> 25.02.2019 18:19, Kevin Wolf wrote:
>>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
>>> needs to be increased while the coroutine is waiting to be scheduled
>>> in the new AioContext after nbd_client_attach_aio_context().
>>
>> Hi!
>>
>> I have some questions, could you explain, please?
>>
>> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
>> sure that connection_co yielded from nbd_read_eof, yes?
>>
>> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
>> bs->in_flight ?
> 
> Without incrementing bs->in_flight, nothing would guarantee that
> aio_poll() is called and the BH is actually executed before bdrv_drain()
> returns.

Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
drained, as you write in comment?

> 
>>>
>>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>>> ---
>>>    block/nbd-client.c | 20 ++++++++++++++++++--
>>>    1 file changed, 18 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/block/nbd-client.c b/block/nbd-client.c
>>> index 60f38f0320..bfbaf7ebe9 100644
>>> --- a/block/nbd-client.c
>>> +++ b/block/nbd-client.c
>>> @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
>>>        qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
>>>    }
>>>    
>>> +static void nbd_client_attach_aio_context_bh(void *opaque)
>>> +{
>>> +    BlockDriverState *bs = opaque;
>>> +    NBDClientSession *client = nbd_get_client_session(bs);
>>> +
>>> +    /* The node is still drained, so we know the coroutine has yielded in
>>> +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
>>> +     * entered for the first time. Both places are safe for entering the
>>> +     * coroutine.*/
>>> +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
>>> +    bdrv_dec_in_flight(bs);
>>> +}
>>> +
>>>    void nbd_client_attach_aio_context(BlockDriverState *bs,
>>>                                       AioContext *new_context)
>>>    {
>>>        NBDClientSession *client = nbd_get_client_session(bs);
>>>        qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
>>>    
>>> -    /* FIXME Really need a bdrv_inc_in_flight() here */
>>> -    aio_co_schedule(new_context, client->connection_co);
>>> +    bdrv_inc_in_flight(bs);
>>> +
>>> +    /* Need to wait here for the BH to run because the BH must run while the
>>> +     * node is still drained. */
>>> +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
>>>    }
>>>    
>>>    void nbd_client_close(BlockDriverState *bs)
>>>
>>
>>
>> -- 
>> Best regards,
>> Vladimir


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Kevin Wolf]

Am 11.04.2019 um 16:48 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 11.04.2019 17:15, Kevin Wolf wrote:
> > Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
> >> 25.02.2019 18:19, Kevin Wolf wrote:
> >>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
> >>> needs to be increased while the coroutine is waiting to be scheduled
> >>> in the new AioContext after nbd_client_attach_aio_context().
> >>
> >> Hi!
> >>
> >> I have some questions, could you explain, please?
> >>
> >> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
> >> sure that connection_co yielded from nbd_read_eof, yes?
> >>
> >> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
> >> bs->in_flight ?
> > 
> > Without incrementing bs->in_flight, nothing would guarantee that
> > aio_poll() is called and the BH is actually executed before bdrv_drain()
> > returns.
> 
> Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
> drained, as you write in comment?

Yes, exactly. But if bs->in_flight == 0, the AIO_WAIT_WHILE() condition
in the drain code could become false, so aio_poll() would not be called
again and drain would return even if the BH is still pending.

Kevin

> > 
> >>>
> >>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> >>> ---
> >>>    block/nbd-client.c | 20 ++++++++++++++++++--
> >>>    1 file changed, 18 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/block/nbd-client.c b/block/nbd-client.c
> >>> index 60f38f0320..bfbaf7ebe9 100644
> >>> --- a/block/nbd-client.c
> >>> +++ b/block/nbd-client.c
> >>> @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
> >>>        qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
> >>>    }
> >>>    
> >>> +static void nbd_client_attach_aio_context_bh(void *opaque)
> >>> +{
> >>> +    BlockDriverState *bs = opaque;
> >>> +    NBDClientSession *client = nbd_get_client_session(bs);
> >>> +
> >>> +    /* The node is still drained, so we know the coroutine has yielded in
> >>> +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
> >>> +     * entered for the first time. Both places are safe for entering the
> >>> +     * coroutine.*/
> >>> +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
> >>> +    bdrv_dec_in_flight(bs);
> >>> +}
> >>> +
> >>>    void nbd_client_attach_aio_context(BlockDriverState *bs,
> >>>                                       AioContext *new_context)
> >>>    {
> >>>        NBDClientSession *client = nbd_get_client_session(bs);
> >>>        qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
> >>>    
> >>> -    /* FIXME Really need a bdrv_inc_in_flight() here */
> >>> -    aio_co_schedule(new_context, client->connection_co);
> >>> +    bdrv_inc_in_flight(bs);
> >>> +
> >>> +    /* Need to wait here for the BH to run because the BH must run while the
> >>> +     * node is still drained. */
> >>> +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
> >>>    }
> >>>    
> >>>    void nbd_client_close(BlockDriverState *bs)
> >>>
> >>
> >>
> >> -- 
> >> Best regards,
> >> Vladimir
> 
> 
> -- 
> Best regards,
> Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Kevin Wolf]

Am 11.04.2019 um 16:48 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 11.04.2019 17:15, Kevin Wolf wrote:
> > Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
> >> 25.02.2019 18:19, Kevin Wolf wrote:
> >>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
> >>> needs to be increased while the coroutine is waiting to be scheduled
> >>> in the new AioContext after nbd_client_attach_aio_context().
> >>
> >> Hi!
> >>
> >> I have some questions, could you explain, please?
> >>
> >> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
> >> sure that connection_co yielded from nbd_read_eof, yes?
> >>
> >> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
> >> bs->in_flight ?
> > 
> > Without incrementing bs->in_flight, nothing would guarantee that
> > aio_poll() is called and the BH is actually executed before bdrv_drain()
> > returns.
> 
> Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
> drained, as you write in comment?

Yes, exactly. But if bs->in_flight == 0, the AIO_WAIT_WHILE() condition
in the drain code could become false, so aio_poll() would not be called
again and drain would return even if the BH is still pending.

Kevin

> > 
> >>>
> >>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> >>> ---
> >>>    block/nbd-client.c | 20 ++++++++++++++++++--
> >>>    1 file changed, 18 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/block/nbd-client.c b/block/nbd-client.c
> >>> index 60f38f0320..bfbaf7ebe9 100644
> >>> --- a/block/nbd-client.c
> >>> +++ b/block/nbd-client.c
> >>> @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
> >>>        qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
> >>>    }
> >>>    
> >>> +static void nbd_client_attach_aio_context_bh(void *opaque)
> >>> +{
> >>> +    BlockDriverState *bs = opaque;
> >>> +    NBDClientSession *client = nbd_get_client_session(bs);
> >>> +
> >>> +    /* The node is still drained, so we know the coroutine has yielded in
> >>> +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
> >>> +     * entered for the first time. Both places are safe for entering the
> >>> +     * coroutine.*/
> >>> +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
> >>> +    bdrv_dec_in_flight(bs);
> >>> +}
> >>> +
> >>>    void nbd_client_attach_aio_context(BlockDriverState *bs,
> >>>                                       AioContext *new_context)
> >>>    {
> >>>        NBDClientSession *client = nbd_get_client_session(bs);
> >>>        qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
> >>>    
> >>> -    /* FIXME Really need a bdrv_inc_in_flight() here */
> >>> -    aio_co_schedule(new_context, client->connection_co);
> >>> +    bdrv_inc_in_flight(bs);
> >>> +
> >>> +    /* Need to wait here for the BH to run because the BH must run while the
> >>> +     * node is still drained. */
> >>> +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
> >>>    }
> >>>    
> >>>    void nbd_client_close(BlockDriverState *bs)
> >>>
> >>
> >>
> >> -- 
> >> Best regards,
> >> Vladimir
> 
> 
> -- 
> Best regards,
> Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Vladimir Sementsov-Ogievskiy]

11.04.2019 19:48, Kevin Wolf wrote:
> Am 11.04.2019 um 16:48 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> 11.04.2019 17:15, Kevin Wolf wrote:
>>> Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>> 25.02.2019 18:19, Kevin Wolf wrote:
>>>>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
>>>>> needs to be increased while the coroutine is waiting to be scheduled
>>>>> in the new AioContext after nbd_client_attach_aio_context().
>>>>
>>>> Hi!
>>>>
>>>> I have some questions, could you explain, please?
>>>>
>>>> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
>>>> sure that connection_co yielded from nbd_read_eof, yes?
>>>>
>>>> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
>>>> bs->in_flight ?
>>>
>>> Without incrementing bs->in_flight, nothing would guarantee that
>>> aio_poll() is called and the BH is actually executed before bdrv_drain()
>>> returns.
>>
>> Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
>> drained, as you write in comment?
> 
> Yes, exactly. But if bs->in_flight == 0, the AIO_WAIT_WHILE() condition
> in the drain code could become false, so aio_poll() would not be called
> again and drain would return even if the BH is still pending.
> 

Ah, oops, sorry my English, I read it like "nothing would prevent". Understand now, thanks.

>>>
>>>>>
>>>>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>>>>> ---
>>>>>     block/nbd-client.c | 20 ++++++++++++++++++--
>>>>>     1 file changed, 18 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/block/nbd-client.c b/block/nbd-client.c
>>>>> index 60f38f0320..bfbaf7ebe9 100644
>>>>> --- a/block/nbd-client.c
>>>>> +++ b/block/nbd-client.c
>>>>> @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
>>>>>         qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
>>>>>     }
>>>>>     
>>>>> +static void nbd_client_attach_aio_context_bh(void *opaque)
>>>>> +{
>>>>> +    BlockDriverState *bs = opaque;
>>>>> +    NBDClientSession *client = nbd_get_client_session(bs);
>>>>> +
>>>>> +    /* The node is still drained, so we know the coroutine has yielded in
>>>>> +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
>>>>> +     * entered for the first time. Both places are safe for entering the
>>>>> +     * coroutine.*/
>>>>> +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
>>>>> +    bdrv_dec_in_flight(bs);
>>>>> +}
>>>>> +
>>>>>     void nbd_client_attach_aio_context(BlockDriverState *bs,
>>>>>                                        AioContext *new_context)
>>>>>     {
>>>>>         NBDClientSession *client = nbd_get_client_session(bs);
>>>>>         qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
>>>>>     
>>>>> -    /* FIXME Really need a bdrv_inc_in_flight() here */
>>>>> -    aio_co_schedule(new_context, client->connection_co);
>>>>> +    bdrv_inc_in_flight(bs);
>>>>> +
>>>>> +    /* Need to wait here for the BH to run because the BH must run while the
>>>>> +     * node is still drained. */
>>>>> +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
>>>>>     }
>>>>>     
>>>>>     void nbd_client_close(BlockDriverState *bs)
>>>>>
>>>>
>>>>
>>>> -- 
>>>> Best regards,
>>>> Vladimir
>>
>>
>> -- 
>> Best regards,
>> Vladimir


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Vladimir Sementsov-Ogievskiy]

11.04.2019 19:48, Kevin Wolf wrote:
> Am 11.04.2019 um 16:48 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> 11.04.2019 17:15, Kevin Wolf wrote:
>>> Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>> 25.02.2019 18:19, Kevin Wolf wrote:
>>>>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
>>>>> needs to be increased while the coroutine is waiting to be scheduled
>>>>> in the new AioContext after nbd_client_attach_aio_context().
>>>>
>>>> Hi!
>>>>
>>>> I have some questions, could you explain, please?
>>>>
>>>> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
>>>> sure that connection_co yielded from nbd_read_eof, yes?
>>>>
>>>> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
>>>> bs->in_flight ?
>>>
>>> Without incrementing bs->in_flight, nothing would guarantee that
>>> aio_poll() is called and the BH is actually executed before bdrv_drain()
>>> returns.
>>
>> Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
>> drained, as you write in comment?
> 
> Yes, exactly. But if bs->in_flight == 0, the AIO_WAIT_WHILE() condition
> in the drain code could become false, so aio_poll() would not be called
> again and drain would return even if the BH is still pending.
> 

Ah, oops, sorry my English, I read it like "nothing would prevent". Understand now, thanks.

>>>
>>>>>
>>>>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>>>>> ---
>>>>>     block/nbd-client.c | 20 ++++++++++++++++++--
>>>>>     1 file changed, 18 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/block/nbd-client.c b/block/nbd-client.c
>>>>> index 60f38f0320..bfbaf7ebe9 100644
>>>>> --- a/block/nbd-client.c
>>>>> +++ b/block/nbd-client.c
>>>>> @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
>>>>>         qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
>>>>>     }
>>>>>     
>>>>> +static void nbd_client_attach_aio_context_bh(void *opaque)
>>>>> +{
>>>>> +    BlockDriverState *bs = opaque;
>>>>> +    NBDClientSession *client = nbd_get_client_session(bs);
>>>>> +
>>>>> +    /* The node is still drained, so we know the coroutine has yielded in
>>>>> +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
>>>>> +     * entered for the first time. Both places are safe for entering the
>>>>> +     * coroutine.*/
>>>>> +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
>>>>> +    bdrv_dec_in_flight(bs);
>>>>> +}
>>>>> +
>>>>>     void nbd_client_attach_aio_context(BlockDriverState *bs,
>>>>>                                        AioContext *new_context)
>>>>>     {
>>>>>         NBDClientSession *client = nbd_get_client_session(bs);
>>>>>         qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
>>>>>     
>>>>> -    /* FIXME Really need a bdrv_inc_in_flight() here */
>>>>> -    aio_co_schedule(new_context, client->connection_co);
>>>>> +    bdrv_inc_in_flight(bs);
>>>>> +
>>>>> +    /* Need to wait here for the BH to run because the BH must run while the
>>>>> +     * node is still drained. */
>>>>> +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
>>>>>     }
>>>>>     
>>>>>     void nbd_client_close(BlockDriverState *bs)
>>>>>
>>>>
>>>>
>>>> -- 
>>>> Best regards,
>>>> Vladimir
>>
>>
>> -- 
>> Best regards,
>> Vladimir


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Vladimir Sementsov-Ogievskiy]

11.04.2019 20:13, Vladimir Sementsov-Ogievskiy wrote:
> 11.04.2019 19:48, Kevin Wolf wrote:
>> Am 11.04.2019 um 16:48 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>> 11.04.2019 17:15, Kevin Wolf wrote:
>>>> Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>>> 25.02.2019 18:19, Kevin Wolf wrote:
>>>>>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
>>>>>> needs to be increased while the coroutine is waiting to be scheduled
>>>>>> in the new AioContext after nbd_client_attach_aio_context().
>>>>>
>>>>> Hi!
>>>>>
>>>>> I have some questions, could you explain, please?
>>>>>
>>>>> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
>>>>> sure that connection_co yielded from nbd_read_eof, yes?
>>>>>
>>>>> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
>>>>> bs->in_flight ?
>>>>
>>>> Without incrementing bs->in_flight, nothing would guarantee that
>>>> aio_poll() is called and the BH is actually executed before bdrv_drain()
>>>> returns.
>>>
>>> Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
>>> drained, as you write in comment?
>>
>> Yes, exactly. But if bs->in_flight == 0, the AIO_WAIT_WHILE() condition
>> in the drain code could become false, so aio_poll() would not be called
>> again and drain would return even if the BH is still pending.
>>
> 
> Ah, oops, sorry my English, I read it like "nothing would prevent". Understand now, thanks.

Or not again.. We will not return to drain code, as we will loop in aio_wait_bh_oneshot,
which will not return until BH handled

> 
>>>>
>>>>>>
>>>>>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>>>>>> ---
>>>>>>     block/nbd-client.c | 20 ++++++++++++++++++--
>>>>>>     1 file changed, 18 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/block/nbd-client.c b/block/nbd-client.c
>>>>>> index 60f38f0320..bfbaf7ebe9 100644
>>>>>> --- a/block/nbd-client.c
>>>>>> +++ b/block/nbd-client.c
>>>>>> @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
>>>>>>         qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
>>>>>>     }
>>>>>> +static void nbd_client_attach_aio_context_bh(void *opaque)
>>>>>> +{
>>>>>> +    BlockDriverState *bs = opaque;
>>>>>> +    NBDClientSession *client = nbd_get_client_session(bs);
>>>>>> +
>>>>>> +    /* The node is still drained, so we know the coroutine has yielded in
>>>>>> +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
>>>>>> +     * entered for the first time. Both places are safe for entering the
>>>>>> +     * coroutine.*/
>>>>>> +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
>>>>>> +    bdrv_dec_in_flight(bs);
>>>>>> +}
>>>>>> +
>>>>>>     void nbd_client_attach_aio_context(BlockDriverState *bs,
>>>>>>                                        AioContext *new_context)
>>>>>>     {
>>>>>>         NBDClientSession *client = nbd_get_client_session(bs);
>>>>>>         qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
>>>>>> -    /* FIXME Really need a bdrv_inc_in_flight() here */
>>>>>> -    aio_co_schedule(new_context, client->connection_co);
>>>>>> +    bdrv_inc_in_flight(bs);
>>>>>> +
>>>>>> +    /* Need to wait here for the BH to run because the BH must run while the
>>>>>> +     * node is still drained. */
>>>>>> +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
>>>>>>     }
>>>>>>     void nbd_client_close(BlockDriverState *bs)
>>>>>>
>>>>>
>>>>>
>>>>> -- 
>>>>> Best regards,
>>>>> Vladimir
>>>
>>>
>>> -- 
>>> Best regards,
>>> Vladimir
> 
> 


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Vladimir Sementsov-Ogievskiy]

11.04.2019 20:13, Vladimir Sementsov-Ogievskiy wrote:
> 11.04.2019 19:48, Kevin Wolf wrote:
>> Am 11.04.2019 um 16:48 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>> 11.04.2019 17:15, Kevin Wolf wrote:
>>>> Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>>> 25.02.2019 18:19, Kevin Wolf wrote:
>>>>>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
>>>>>> needs to be increased while the coroutine is waiting to be scheduled
>>>>>> in the new AioContext after nbd_client_attach_aio_context().
>>>>>
>>>>> Hi!
>>>>>
>>>>> I have some questions, could you explain, please?
>>>>>
>>>>> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
>>>>> sure that connection_co yielded from nbd_read_eof, yes?
>>>>>
>>>>> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
>>>>> bs->in_flight ?
>>>>
>>>> Without incrementing bs->in_flight, nothing would guarantee that
>>>> aio_poll() is called and the BH is actually executed before bdrv_drain()
>>>> returns.
>>>
>>> Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
>>> drained, as you write in comment?
>>
>> Yes, exactly. But if bs->in_flight == 0, the AIO_WAIT_WHILE() condition
>> in the drain code could become false, so aio_poll() would not be called
>> again and drain would return even if the BH is still pending.
>>
> 
> Ah, oops, sorry my English, I read it like "nothing would prevent". Understand now, thanks.

Or not again.. We will not return to drain code, as we will loop in aio_wait_bh_oneshot,
which will not return until BH handled

> 
>>>>
>>>>>>
>>>>>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>>>>>> ---
>>>>>>     block/nbd-client.c | 20 ++++++++++++++++++--
>>>>>>     1 file changed, 18 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/block/nbd-client.c b/block/nbd-client.c
>>>>>> index 60f38f0320..bfbaf7ebe9 100644
>>>>>> --- a/block/nbd-client.c
>>>>>> +++ b/block/nbd-client.c
>>>>>> @@ -977,14 +977,30 @@ void nbd_client_detach_aio_context(BlockDriverState *bs)
>>>>>>         qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc));
>>>>>>     }
>>>>>> +static void nbd_client_attach_aio_context_bh(void *opaque)
>>>>>> +{
>>>>>> +    BlockDriverState *bs = opaque;
>>>>>> +    NBDClientSession *client = nbd_get_client_session(bs);
>>>>>> +
>>>>>> +    /* The node is still drained, so we know the coroutine has yielded in
>>>>>> +     * nbd_read_eof(), the only place where bs->in_flight can reach 0, or it is
>>>>>> +     * entered for the first time. Both places are safe for entering the
>>>>>> +     * coroutine.*/
>>>>>> +    qemu_aio_coroutine_enter(bs->aio_context, client->connection_co);
>>>>>> +    bdrv_dec_in_flight(bs);
>>>>>> +}
>>>>>> +
>>>>>>     void nbd_client_attach_aio_context(BlockDriverState *bs,
>>>>>>                                        AioContext *new_context)
>>>>>>     {
>>>>>>         NBDClientSession *client = nbd_get_client_session(bs);
>>>>>>         qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), new_context);
>>>>>> -    /* FIXME Really need a bdrv_inc_in_flight() here */
>>>>>> -    aio_co_schedule(new_context, client->connection_co);
>>>>>> +    bdrv_inc_in_flight(bs);
>>>>>> +
>>>>>> +    /* Need to wait here for the BH to run because the BH must run while the
>>>>>> +     * node is still drained. */
>>>>>> +    aio_wait_bh_oneshot(new_context, nbd_client_attach_aio_context_bh, bs);
>>>>>>     }
>>>>>>     void nbd_client_close(BlockDriverState *bs)
>>>>>>
>>>>>
>>>>>
>>>>> -- 
>>>>> Best regards,
>>>>> Vladimir
>>>
>>>
>>> -- 
>>> Best regards,
>>> Vladimir
> 
> 


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Kevin Wolf]

Am 11.04.2019 um 19:20 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 11.04.2019 20:13, Vladimir Sementsov-Ogievskiy wrote:
> > 11.04.2019 19:48, Kevin Wolf wrote:
> >> Am 11.04.2019 um 16:48 hat Vladimir Sementsov-Ogievskiy geschrieben:
> >>> 11.04.2019 17:15, Kevin Wolf wrote:
> >>>> Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
> >>>>> 25.02.2019 18:19, Kevin Wolf wrote:
> >>>>>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
> >>>>>> needs to be increased while the coroutine is waiting to be scheduled
> >>>>>> in the new AioContext after nbd_client_attach_aio_context().
> >>>>>
> >>>>> Hi!
> >>>>>
> >>>>> I have some questions, could you explain, please?
> >>>>>
> >>>>> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
> >>>>> sure that connection_co yielded from nbd_read_eof, yes?
> >>>>>
> >>>>> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
> >>>>> bs->in_flight ?
> >>>>
> >>>> Without incrementing bs->in_flight, nothing would guarantee that
> >>>> aio_poll() is called and the BH is actually executed before bdrv_drain()
> >>>> returns.
> >>>
> >>> Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
> >>> drained, as you write in comment?
> >>
> >> Yes, exactly. But if bs->in_flight == 0, the AIO_WAIT_WHILE() condition
> >> in the drain code could become false, so aio_poll() would not be called
> >> again and drain would return even if the BH is still pending.
> >>
> > 
> > Ah, oops, sorry my English, I read it like "nothing would prevent". Understand now, thanks.
> 
> Or not again.. We will not return to drain code, as we will loop in aio_wait_bh_oneshot,
> which will not return until BH handled

Oh, hm, good point, it's not just aio_bh_schedule_oneshot(), but the
waiting version.

I think I added the aio_wait_bh_oneshot() only later to address a review
comment, so maybe increasing bs->in_flight isn't needed any more now.

But it can't hurt at least...

Kevin

Re: [Qemu-devel] [PULL 16/71] nbd: Increase bs->in_flight during AioContext switch

[Kevin Wolf]

Am 11.04.2019 um 19:20 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 11.04.2019 20:13, Vladimir Sementsov-Ogievskiy wrote:
> > 11.04.2019 19:48, Kevin Wolf wrote:
> >> Am 11.04.2019 um 16:48 hat Vladimir Sementsov-Ogievskiy geschrieben:
> >>> 11.04.2019 17:15, Kevin Wolf wrote:
> >>>> Am 11.04.2019 um 15:40 hat Vladimir Sementsov-Ogievskiy geschrieben:
> >>>>> 25.02.2019 18:19, Kevin Wolf wrote:
> >>>>>> bdrv_drain() must not leave connection_co scheduled, so bs->in_flight
> >>>>>> needs to be increased while the coroutine is waiting to be scheduled
> >>>>>> in the new AioContext after nbd_client_attach_aio_context().
> >>>>>
> >>>>> Hi!
> >>>>>
> >>>>> I have some questions, could you explain, please?
> >>>>>
> >>>>> "bdrv_drain() must not leave connection_co scheduled" - it's because we want to be
> >>>>> sure that connection_co yielded from nbd_read_eof, yes?
> >>>>>
> >>>>> But it is guaranteed by aio_wait_bh_oneshot.. Why do we need additioinally inc/dec
> >>>>> bs->in_flight ?
> >>>>
> >>>> Without incrementing bs->in_flight, nothing would guarantee that
> >>>> aio_poll() is called and the BH is actually executed before bdrv_drain()
> >>>> returns.
> >>>
> >>> Don't follow.. Don't we want exactly this, we want BH to be executed while node is still
> >>> drained, as you write in comment?
> >>
> >> Yes, exactly. But if bs->in_flight == 0, the AIO_WAIT_WHILE() condition
> >> in the drain code could become false, so aio_poll() would not be called
> >> again and drain would return even if the BH is still pending.
> >>
> > 
> > Ah, oops, sorry my English, I read it like "nothing would prevent". Understand now, thanks.
> 
> Or not again.. We will not return to drain code, as we will loop in aio_wait_bh_oneshot,
> which will not return until BH handled

Oh, hm, good point, it's not just aio_bh_schedule_oneshot(), but the
waiting version.

I think I added the aio_wait_bh_oneshot() only later to address a review
comment, so maybe increasing bs->in_flight isn't needed any more now.

But it can't hurt at least...

Kevin