[Qemu-devel] [PULL for-2.9 0/3] Block patches

[Qemu-devel] [PULL for-2.9 0/3] Block patches

[Jeff Cody]

The following changes since commit 95b31d709ba343ad237c3630047ee7438bac4065:

  Merge remote-tracking branch 'remotes/awilliam/tags/vfio-updates-20170331.0' into staging (2017-03-31 18:06:13 +0100)

are available in the git repository at:

  git://github.com/codyprime/qemu-kvm-jtc.git tags/block-pull-request

for you to fetch changes up to 34634ca28688652198f77d7001c0f1e204434663:

  block/curl: Check protocol prefix (2017-03-31 15:53:22 -0400)

----------------------------------------------------------------
Block patches for -rc3
----------------------------------------------------------------

Eric Blake (1):
  rbd: Fix regression in legacy key/values containing escaped :

Max Reitz (2):
  qapi/curl: Extend and fix blockdev-add schema
  block/curl: Check protocol prefix

 block/curl.c         |  10 +++++
 block/rbd.c          |  83 +++++++++++++++++++++--------------------
 qapi/block-core.json | 103 ++++++++++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 146 insertions(+), 50 deletions(-)

-- 
2.9.3

[Qemu-devel] [PULL for-2.9 1/3] rbd: Fix regression in legacy key/values containing escaped :

[Jeff Cody]

From: Eric Blake <eblake@redhat.com>

Commit c7cacb3 accidentally broke legacy key-value parsing through
pseudo-filename parsing of -drive file=rbd://..., for any key that
contains an escaped ':'.  Such a key is surprisingly common, thanks
to mon_host specifying a 'host:port' string.  The break happens
because passing things from QDict through QemuOpts back to another
QDict requires that we pack our parsed key/value pairs into a string,
and then reparse that string, but the intermediate string that we
created ("key1=value1:key2=value2") lost the \: escaping that was
present in the original, so that we could no longer see which : were
used as separators vs. those used as part of the original input.

Fix it by collecting the key/value pairs through a QList, and
sending that list on a round trip through a JSON QString (as in
'["key1","value1","key2","value2"]') on its way through QemuOpts,
rather than hand-rolling our own string.  Since the string is only
handled internally, this was faster than creating a full-blown
struct of '[{"key1":"value1"},{"key2":"value2"}]', and safer at
guaranteeing order compared to '{"key1":"value1","key2":"value2"}'.

It would be nicer if we didn't have to round-trip through QemuOpts
in the first place, but that's a much bigger task for later.

Reproducer:
./x86_64-softmmu/qemu-system-x86_64 -nodefaults -nographic -qmp stdio \
-drive 'file=rbd:volumes/volume-ea141b5c-cdb3-4765-910d-e7008b209a70'\
':id=compute:key=AQAVkvxXAAAAABAA9ZxWFYdRmV+DSwKr7BKKXg=='\
':auth_supported=cephx\;none:mon_host=192.168.1.2\:6789'\
',format=raw,if=none,id=drive-virtio-disk0,'\
'serial=ea141b5c-cdb3-4765-910d-e7008b209a70,cache=writeback'

Even without an RBD setup, this serves a test of whether we get
the incorrect parser error of:
qemu-system-x86_64: -drive file=rbd:...cache=writeback: conf option 6789 has no value
or the correct behavior of hanging while trying to connect to
the requested mon_host of 192.168.1.2:6789.

Reported-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20170331152730.12514-1-eblake@redhat.com
Signed-off-by: Jeff Cody <jcody@redhat.com>
---
 block/rbd.c | 83 +++++++++++++++++++++++++++++++------------------------------
 1 file changed, 42 insertions(+), 41 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index 498322b..fbdb131 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -20,6 +20,7 @@
 #include "crypto/secret.h"
 #include "qemu/cutils.h"
 #include "qapi/qmp/qstring.h"
+#include "qapi/qmp/qjson.h"
 
 /*
  * When specifying the image filename use:
@@ -135,18 +136,16 @@ static void qemu_rbd_parse_filename(const char *filename, QDict *options,
                                     Error **errp)
 {
     const char *start;
-    char *p, *buf, *keypairs;
+    char *p, *buf;
+    QList *keypairs = NULL;
     char *found_str;
-    size_t max_keypair_size;
 
     if (!strstart(filename, "rbd:", &start)) {
         error_setg(errp, "File name must start with 'rbd:'");
         return;
     }
 
-    max_keypair_size = strlen(start) + 1;
     buf = g_strdup(start);
-    keypairs = g_malloc0(max_keypair_size);
     p = buf;
 
     found_str = qemu_rbd_next_tok(p, '/', &p);
@@ -194,33 +193,30 @@ static void qemu_rbd_parse_filename(const char *filename, QDict *options,
         } else if (!strcmp(name, "id")) {
             qdict_put(options, "user" , qstring_from_str(value));
         } else {
-            /* FIXME: This is pretty ugly, and not the right way to do this.
-             *        These should be contained in a structure, and then
-             *        passed explicitly as individual key/value pairs to
-             *        rados.  Consider this legacy code that needs to be
-             *        updated. */
-            char *tmp = g_malloc0(max_keypair_size);
-            /* only use a delimiter if it is not the first keypair found */
-            /* These are sets of unknown key/value pairs we'll pass along
-             * to ceph */
-            if (keypairs[0]) {
-                snprintf(tmp, max_keypair_size, ":%s=%s", name, value);
-                pstrcat(keypairs, max_keypair_size, tmp);
-            } else {
-                snprintf(keypairs, max_keypair_size, "%s=%s", name, value);
+            /*
+             * We pass these internally to qemu_rbd_set_keypairs(), so
+             * we can get away with the simpler list of [ "key1",
+             * "value1", "key2", "value2" ] rather than a raw dict
+             * { "key1": "value1", "key2": "value2" } where we can't
+             * guarantee order, or even a more correct but complex
+             * [ { "key1": "value1" }, { "key2": "value2" } ]
+             */
+            if (!keypairs) {
+                keypairs = qlist_new();
             }
-            g_free(tmp);
+            qlist_append(keypairs, qstring_from_str(name));
+            qlist_append(keypairs, qstring_from_str(value));
         }
     }
 
-    if (keypairs[0]) {
-        qdict_put(options, "=keyvalue-pairs", qstring_from_str(keypairs));
+    if (keypairs) {
+        qdict_put(options, "=keyvalue-pairs",
+                  qobject_to_json(QOBJECT(keypairs)));
     }
 
-
 done:
     g_free(buf);
-    g_free(keypairs);
+    QDECREF(keypairs);
     return;
 }
 
@@ -244,36 +240,41 @@ static int qemu_rbd_set_auth(rados_t cluster, const char *secretid,
     return 0;
 }
 
-static int qemu_rbd_set_keypairs(rados_t cluster, const char *keypairs,
+static int qemu_rbd_set_keypairs(rados_t cluster, const char *keypairs_json,
                                  Error **errp)
 {
-    char *p, *buf;
-    char *name;
-    char *value;
+    QList *keypairs;
+    QString *name;
+    QString *value;
+    const char *key;
+    size_t remaining;
     int ret = 0;
 
-    buf = g_strdup(keypairs);
-    p = buf;
+    if (!keypairs_json) {
+        return ret;
+    }
+    keypairs = qobject_to_qlist(qobject_from_json(keypairs_json,
+                                                  &error_abort));
+    remaining = qlist_size(keypairs) / 2;
+    assert(remaining);
 
-    while (p) {
-        name = qemu_rbd_next_tok(p, '=', &p);
-        if (!p) {
-            error_setg(errp, "conf option %s has no value", name);
-            ret = -EINVAL;
-            break;
-        }
+    while (remaining--) {
+        name = qobject_to_qstring(qlist_pop(keypairs));
+        value = qobject_to_qstring(qlist_pop(keypairs));
+        assert(name && value);
+        key = qstring_get_str(name);
 
-        value = qemu_rbd_next_tok(p, ':', &p);
-
-        ret = rados_conf_set(cluster, name, value);
+        ret = rados_conf_set(cluster, key, qstring_get_str(value));
+        QDECREF(name);
+        QDECREF(value);
         if (ret < 0) {
-            error_setg_errno(errp, -ret, "invalid conf option %s", name);
+            error_setg_errno(errp, -ret, "invalid conf option %s", key);
             ret = -EINVAL;
             break;
         }
     }
 
-    g_free(buf);
+    QDECREF(keypairs);
     return ret;
 }
 
-- 
2.9.3

[Qemu-devel] [PULL for-2.9 2/3] qapi/curl: Extend and fix blockdev-add schema

[Jeff Cody]

From: Max Reitz <mreitz@redhat.com>

The curl block driver accepts more options than just "filename"; also,
the URL is actually expected to be passed through the "url" option
instead of "filename".

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20170331120431.1767-2-mreitz@redhat.com
Signed-off-by: Jeff Cody <jcody@redhat.com>
---
 qapi/block-core.json | 103 ++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 94 insertions(+), 9 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 4e8e4e3..8de39d1 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -2737,16 +2737,101 @@
             '*debug': 'int' } }
 
 ##
-# @BlockdevOptionsCurl:
+# @BlockdevOptionsCurlBase:
 #
-# Driver specific block device options for the curl backend.
+# Driver specific block device options shared by all protocols supported by the
+# curl backend.
 #
-# @filename:    path to the image file
+# @url:                     URL of the image file
+#
+# @readahead:               Size of the read-ahead cache; must be a multiple of
+#                           512 (defaults to 256 kB)
+#
+# @timeout:                 Timeout for connections, in seconds (defaults to 5)
+#
+# @username:                Username for authentication (defaults to none)
+#
+# @password-secret:         ID of a QCryptoSecret object providing a password
+#                           for authentication (defaults to no password)
+#
+# @proxy-username:          Username for proxy authentication (defaults to none)
+#
+# @proxy-password-secret:   ID of a QCryptoSecret object providing a password
+#                           for proxy authentication (defaults to no password)
+#
+# Since: 2.9
+##
+{ 'struct': 'BlockdevOptionsCurlBase',
+  'data': { 'url': 'str',
+            '*readahead': 'int',
+            '*timeout': 'int',
+            '*username': 'str',
+            '*password-secret': 'str',
+            '*proxy-username': 'str',
+            '*proxy-password-secret': 'str' } }
+
+##
+# @BlockdevOptionsCurlHttp:
+#
+# Driver specific block device options for HTTP connections over the curl
+# backend.  URLs must start with "http://".
+#
+# @cookie:      List of cookies to set; format is
+#               "name1=content1; name2=content2;" as explained by
+#               CURLOPT_COOKIE(3). Defaults to no cookies.
+#
+# Since: 2.9
+##
+{ 'struct': 'BlockdevOptionsCurlHttp',
+  'base': 'BlockdevOptionsCurlBase',
+  'data': { '*cookie': 'str' } }
+
+##
+# @BlockdevOptionsCurlHttps:
+#
+# Driver specific block device options for HTTPS connections over the curl
+# backend.  URLs must start with "https://".
+#
+# @cookie:      List of cookies to set; format is
+#               "name1=content1; name2=content2;" as explained by
+#               CURLOPT_COOKIE(3). Defaults to no cookies.
+#
+# @sslverify:   Whether to verify the SSL certificate's validity (defaults to
+#               true)
+#
+# Since: 2.9
+##
+{ 'struct': 'BlockdevOptionsCurlHttps',
+  'base': 'BlockdevOptionsCurlBase',
+  'data': { '*cookie': 'str',
+            '*sslverify': 'bool' } }
+
+##
+# @BlockdevOptionsCurlFtp:
+#
+# Driver specific block device options for FTP connections over the curl
+# backend.  URLs must start with "ftp://".
+#
+# Since: 2.9
+##
+{ 'struct': 'BlockdevOptionsCurlFtp',
+  'base': 'BlockdevOptionsCurlBase',
+  'data': { } }
+
+##
+# @BlockdevOptionsCurlFtps:
+#
+# Driver specific block device options for FTPS connections over the curl
+# backend.  URLs must start with "ftps://".
+#
+# @sslverify:   Whether to verify the SSL certificate's validity (defaults to
+#               true)
 #
 # Since: 2.9
 ##
-{ 'struct': 'BlockdevOptionsCurl',
-  'data': { 'filename': 'str' } }
+{ 'struct': 'BlockdevOptionsCurlFtps',
+  'base': 'BlockdevOptionsCurlBase',
+  'data': { '*sslverify': 'bool' } }
 
 ##
 # @BlockdevOptionsNbd:
@@ -2815,13 +2900,13 @@
       'cloop':      'BlockdevOptionsGenericFormat',
       'dmg':        'BlockdevOptionsGenericFormat',
       'file':       'BlockdevOptionsFile',
-      'ftp':        'BlockdevOptionsCurl',
-      'ftps':       'BlockdevOptionsCurl',
+      'ftp':        'BlockdevOptionsCurlFtp',
+      'ftps':       'BlockdevOptionsCurlFtps',
       'gluster':    'BlockdevOptionsGluster',
       'host_cdrom': 'BlockdevOptionsFile',
       'host_device':'BlockdevOptionsFile',
-      'http':       'BlockdevOptionsCurl',
-      'https':      'BlockdevOptionsCurl',
+      'http':       'BlockdevOptionsCurlHttp',
+      'https':      'BlockdevOptionsCurlHttps',
       'iscsi':      'BlockdevOptionsIscsi',
       'luks':       'BlockdevOptionsLUKS',
       'nbd':        'BlockdevOptionsNbd',
-- 
2.9.3

[Qemu-devel] [PULL for-2.9 3/3] block/curl: Check protocol prefix

[Jeff Cody]

From: Max Reitz <mreitz@redhat.com>

If the user has explicitly specified a block driver and thus a protocol,
we have to make sure the URL's protocol prefix matches. Otherwise the
latter will silently override the former which might catch some users by
surprise.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20170331120431.1767-3-mreitz@redhat.com
Signed-off-by: Jeff Cody <jcody@redhat.com>
---
 block/curl.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/block/curl.c b/block/curl.c
index 34dbd33..2708d57 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -659,6 +659,7 @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
     const char *cookie;
     double d;
     const char *secretid;
+    const char *protocol_delimiter;
 
     static int inited = 0;
 
@@ -700,6 +701,15 @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
         goto out_noclean;
     }
 
+    if (!strstart(file, bs->drv->protocol_name, &protocol_delimiter) ||
+        !strstart(protocol_delimiter, "://", NULL))
+    {
+        error_setg(errp, "%s curl driver cannot handle the URL '%s' (does not "
+                   "start with '%s://')", bs->drv->protocol_name, file,
+                   bs->drv->protocol_name);
+        goto out_noclean;
+    }
+
     s->username = g_strdup(qemu_opt_get(opts, CURL_BLOCK_OPT_USERNAME));
     secretid = qemu_opt_get(opts, CURL_BLOCK_OPT_PASSWORD_SECRET);
 
-- 
2.9.3

Re: [Qemu-devel] [PULL for-2.9 0/3] Block patches

[Peter Maydell]

On 1 April 2017 at 02:25, Jeff Cody <jcody@redhat.com> wrote:
> The following changes since commit 95b31d709ba343ad237c3630047ee7438bac4065:
>
>   Merge remote-tracking branch 'remotes/awilliam/tags/vfio-updates-20170331.0' into staging (2017-03-31 18:06:13 +0100)
>
> are available in the git repository at:
>
>   git://github.com/codyprime/qemu-kvm-jtc.git tags/block-pull-request
>
> for you to fetch changes up to 34634ca28688652198f77d7001c0f1e204434663:
>
>   block/curl: Check protocol prefix (2017-03-31 15:53:22 -0400)
>
> ----------------------------------------------------------------
> Block patches for -rc3
> ----------------------------------------------------------------
>
> Eric Blake (1):
>   rbd: Fix regression in legacy key/values containing escaped :
>
> Max Reitz (2):
>   qapi/curl: Extend and fix blockdev-add schema
>   block/curl: Check protocol prefix
>
>  block/curl.c         |  10 +++++
>  block/rbd.c          |  83 +++++++++++++++++++++--------------------
>  qapi/block-core.json | 103 ++++++++++++++++++++++++++++++++++++++++++++++-----
>  3 files changed, 146 insertions(+), 50 deletions(-)

Applied, thanks.

-- PMM

Re: [Qemu-devel] [PULL for-2.9 0/3] Block patches

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1367 bytes --]

On Fri, May 12, 2017 at 10:37:14AM -0400, Stefan Hajnoczi wrote:
> The following changes since commit ecc1f5adeec4e3324d1b695a7c54e3967c526949:
> 
>   maintainers: Add myself as linux-user reviewer (2017-05-11 13:31:11 -0400)
> 
> are available in the git repository at:
> 
>   git://github.com/stefanha/qemu.git tags/block-pull-request
> 
> for you to fetch changes up to 321d1dba8bef9676a77e9399484e3cd8bf2cf16a:
> 
>   aio: add missing aio_notify() to aio_enable_external() (2017-05-12 10:36:46 -0400)
> 
> ----------------------------------------------------------------
> 
> ----------------------------------------------------------------
> 
> Daniel P. Berrange (1):
>   coroutine: remove GThread implementation
> 
> Eric Blake (1):
>   block: Simplify BDRV_BLOCK_RAW recursion
> 
> Stefan Hajnoczi (1):
>   aio: add missing aio_notify() to aio_enable_external()
> 
>  configure                |  19 ++---
>  include/block/aio.h      |  10 ++-
>  block/io.c               |   4 +-
>  util/coroutine-gthread.c | 198 -----------------------------------------------
>  .travis.yml              |   5 +-
>  5 files changed, 16 insertions(+), 220 deletions(-)
>  delete mode 100644 util/coroutine-gthread.c
> 
> -- 
> 2.9.3
> 
> 

Thanks, applied to my master tree:
https://github.com/stefanha/qemu/commits/master

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[Qemu-devel] [PULL for-2.9 0/3] Block patches

[Stefan Hajnoczi]

The following changes since commit ecc1f5adeec4e3324d1b695a7c54e3967c526949:

  maintainers: Add myself as linux-user reviewer (2017-05-11 13:31:11 -0400)

are available in the git repository at:

  git://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 321d1dba8bef9676a77e9399484e3cd8bf2cf16a:

  aio: add missing aio_notify() to aio_enable_external() (2017-05-12 10:36:46 -0400)

----------------------------------------------------------------

----------------------------------------------------------------

Daniel P. Berrange (1):
  coroutine: remove GThread implementation

Eric Blake (1):
  block: Simplify BDRV_BLOCK_RAW recursion

Stefan Hajnoczi (1):
  aio: add missing aio_notify() to aio_enable_external()

 configure                |  19 ++---
 include/block/aio.h      |  10 ++-
 block/io.c               |   4 +-
 util/coroutine-gthread.c | 198 -----------------------------------------------
 .travis.yml              |   5 +-
 5 files changed, 16 insertions(+), 220 deletions(-)
 delete mode 100644 util/coroutine-gthread.c

-- 
2.9.3