[Qemu-devel] [PATCH v5 0/4] fdc: Use separate qdev device for drives

[Qemu-devel] [PATCH v5 0/4] fdc: Use separate qdev device for drives

[Kevin Wolf]

We have been complaining for a long time about how the floppy controller and
floppy drives are combined in a single qdev device and how this makes the
device awkward to work with because it behaves different from all other block
devices.

The latest reason to complain was when I noticed that using qdev device names
in QMP commands (e.g. for media change) doesn't really work when only the
controller is a qdev device, but the drives aren't.

So I decided to have a go at it, and this is the result.

It doesn't actually change any of the inner workings of the floppy controller,
but it wires things up differently on the qdev layer so that a floppy
controller now exposes a bus on which the floppy drives sit. This results in a
structure that is similar to IDE where the actual drive state is still in the
controller and the qdev device basically just contains the qdev properties -
not pretty, but quite workable.

The commit message of patch 3 explains how to use it. In short, there is a
'-device floppy' now and it does what you would expect if you ever used ide-cd.

The other problem is old command lines, especially those using things like
'-global isa-fdc,driveA=...'. In order to keep them working, we need to forward
the property to an internally created floppy drive device. This is a bit like
usb-storage, which we know is ugly, but works well enough in practice. The good
news here is that in contrast to usb-storage, the floppy controller only does
the forwarding for legacy configurations; as soon as you start using '-device
floppy', it doesn't happen any more.

So as you may have expected, this conversion doesn't result in a perfect
device, but I think it's definitely an improvement over the old state. I hope
you like it despite the warts. :-)

v5:
- Apply _filter_qemu to stderr, too [John]
- Rename the bus to floppy-bus [Frederic]
- Use FLOPPY_BUS() instead of DO_UPDATE() [Frederic]

v4:
- John says that his grep is broken and hangs at 100% CPU with my attempt to
  extract the floppy controller from info qtree. Use a simpler sed command
  instead (which, unlike the grep command, doesn't handle arbitrary indentation
  level of the next item, but we know what comes next, so just hardcoding 10
  spaces works, too).

v3:
- Fixed omissons in the conversion sysbus-fdc and the Sun one. Nice, combining
  floppy controllers and weird platforms in a single series. [John]

v2:
- Added patch 4 (qemu-iotests case for floppy config on the command line)
- Patch 2: Create a floppy device only if a BlockBackend exists instead of
  always creating two of them
- Patch 2: Initialise drive->fdctrl even if no drive is attached, it is
  accessed anyway during migration
- Patch 3: Keep 'type' qdev property and FDrive->drive in sync
- Patch 3: Removed if with condition that is always true

Alberto Garcia (2):
  throttle: Correct access to wrong BlockBackendPublic structures
  qemu-iotests: Test I/O in a single drive from a throttling group

Halil Pasic (1):
  block: improve error handling in raw_open

Pino Toscano (1):
  qapi: fix memory leak in bdrv_image_info_specific_dump

 block/qapi.c               |  1 +
 block/raw-posix.c          |  1 +
 block/raw-win32.c          |  1 +
 block/throttle-groups.c    | 27 +++++++++++++++++++++++----
 tests/qemu-iotests/093     | 33 ++++++++++++++++++++++++++++-----
 tests/qemu-iotests/093.out |  4 ++--
 6 files changed, 56 insertions(+), 11 deletions(-)

-- 
1.8.3.1

[Qemu-devel] [PATCH v5 1/4] block: improve error handling in raw_open

[Kevin Wolf]

From: Halil Pasic <pasic@linux.vnet.ibm.com>

Make raw_open for POSIX more consistent in handling errors by setting
the error object also when qemu_open fails. The error object was set
generally set in case of errors, but I guess this case was overlooked.
Do the same for win32.

Signed-off-by: Halil Pasic <pasic@linux.vnet.ibm.com>
Reviewed-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Tested-by: Marc Hartmayer <mhartmay@linux.vnet.ibm.com> (POSIX only)
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/raw-posix.c | 1 +
 block/raw-win32.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/block/raw-posix.c b/block/raw-posix.c
index 166e9d1..f481e57 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -443,6 +443,7 @@ static int raw_open_common(BlockDriverState *bs, QDict *options,
     fd = qemu_open(filename, s->open_flags, 0644);
     if (fd < 0) {
         ret = -errno;
+        error_setg_errno(errp, errno, "Could not open '%s'", filename);
         if (ret == -EROFS) {
             ret = -EACCES;
         }
diff --git a/block/raw-win32.c b/block/raw-win32.c
index 734bb10..800fabd 100644
--- a/block/raw-win32.c
+++ b/block/raw-win32.c
@@ -373,6 +373,7 @@ static int raw_open(BlockDriverState *bs, QDict *options, int flags,
     if (s->hfile == INVALID_HANDLE_VALUE) {
         int err = GetLastError();
 
+        error_setg_win32(errp, err, "Could not open '%s'", filename);
         if (err == ERROR_ACCESS_DENIED) {
             ret = -EACCES;
         } else {
-- 
1.8.3.1

[Qemu-devel] [PATCH v5 2/4] qapi: fix memory leak in bdrv_image_info_specific_dump

[Kevin Wolf]

From: Pino Toscano <ptoscano@redhat.com>

The 'obj' result of the visitor was not properly freed, like done in
other places doing a similar job.

Signed-off-by: Pino Toscano <ptoscano@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/qapi.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/block/qapi.c b/block/qapi.c
index 6f947e3..50d3090 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -698,6 +698,7 @@ void bdrv_image_info_specific_dump(fprintf_function func_fprintf, void *f,
     assert(qobject_type(obj) == QTYPE_QDICT);
     data = qdict_get(qobject_to_qdict(obj), "data");
     dump_qobject(func_fprintf, f, 1, data);
+    qobject_decref(obj);
     visit_free(v);
 }
 
-- 
1.8.3.1

[Qemu-devel] [PATCH v5 3/4] throttle: Correct access to wrong BlockBackendPublic structures

[Kevin Wolf]

From: Alberto Garcia <berto@igalia.com>

In 27ccdd52598290f0f8b58be56e235aff7aebfaf3 the throttling fields were
moved from BlockDriverState to BlockBackend. However in a few cases
the code started using throttling fields from the active BlockBackend
instead of the round-robin token, making the algorithm behave
incorrectly.

This can cause starvation if there's a throttling group with several
drives but only one of them has I/O.

Cc: qemu-stable@nongnu.org
Reported-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 block/throttle-groups.c | 27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)

diff --git a/block/throttle-groups.c b/block/throttle-groups.c
index 59545e2..17b2efb 100644
--- a/block/throttle-groups.c
+++ b/block/throttle-groups.c
@@ -168,6 +168,22 @@ static BlockBackend *throttle_group_next_blk(BlockBackend *blk)
     return blk_by_public(next);
 }
 
+/*
+ * Return whether a BlockBackend has pending requests.
+ *
+ * This assumes that tg->lock is held.
+ *
+ * @blk: the BlockBackend
+ * @is_write:  the type of operation (read/write)
+ * @ret:       whether the BlockBackend has pending requests.
+ */
+static inline bool blk_has_pending_reqs(BlockBackend *blk,
+                                        bool is_write)
+{
+    const BlockBackendPublic *blkp = blk_get_public(blk);
+    return blkp->pending_reqs[is_write];
+}
+
 /* Return the next BlockBackend in the round-robin sequence with pending I/O
  * requests.
  *
@@ -188,7 +204,7 @@ static BlockBackend *next_throttle_token(BlockBackend *blk, bool is_write)
 
     /* get next bs round in round robin style */
     token = throttle_group_next_blk(token);
-    while (token != start && !blkp->pending_reqs[is_write]) {
+    while (token != start && !blk_has_pending_reqs(token, is_write)) {
         token = throttle_group_next_blk(token);
     }
 
@@ -196,10 +212,13 @@ static BlockBackend *next_throttle_token(BlockBackend *blk, bool is_write)
      * then decide the token is the current bs because chances are
      * the current bs get the current request queued.
      */
-    if (token == start && !blkp->pending_reqs[is_write]) {
+    if (token == start && !blk_has_pending_reqs(token, is_write)) {
         token = blk;
     }
 
+    /* Either we return the original BB, or one with pending requests */
+    assert(token == blk || blk_has_pending_reqs(token, is_write));
+
     return token;
 }
 
@@ -257,7 +276,7 @@ static void schedule_next_request(BlockBackend *blk, bool is_write)
 
     /* Check if there's any pending request to schedule next */
     token = next_throttle_token(blk, is_write);
-    if (!blkp->pending_reqs[is_write]) {
+    if (!blk_has_pending_reqs(token, is_write)) {
         return;
     }
 
@@ -271,7 +290,7 @@ static void schedule_next_request(BlockBackend *blk, bool is_write)
             qemu_co_queue_next(&blkp->throttled_reqs[is_write])) {
             token = blk;
         } else {
-            ThrottleTimers *tt = &blkp->throttle_timers;
+            ThrottleTimers *tt = &blk_get_public(token)->throttle_timers;
             int64_t now = qemu_clock_get_ns(tt->clock_type);
             timer_mod(tt->timers[is_write], now + 1);
             tg->any_timer_armed[is_write] = true;
-- 
1.8.3.1

[Qemu-devel] [PATCH v5 4/4] qemu-iotests: Test I/O in a single drive from a throttling group

[Kevin Wolf]

From: Alberto Garcia <berto@igalia.com>

iotest 093 contains a test that creates a throttling group with
several drives and performs I/O in all of them. This patch adds a new
test that creates a similar setup but only performs I/O in one of the
drives at the same time.

This is useful to test that the round robin algorithm is behaving
properly in these scenarios, and is specifically written using the
regression introduced in 27ccdd52598290f0f8b58be56e as an example.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/qemu-iotests/093     | 33 ++++++++++++++++++++++++++++-----
 tests/qemu-iotests/093.out |  4 ++--
 2 files changed, 30 insertions(+), 7 deletions(-)

diff --git a/tests/qemu-iotests/093 b/tests/qemu-iotests/093
index ffcb271..2ed393a 100755
--- a/tests/qemu-iotests/093
+++ b/tests/qemu-iotests/093
@@ -53,7 +53,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
             result = self.vm.qmp("block_set_io_throttle", conv_keys=False, **params)
             self.assert_qmp(result, 'return', {})
 
-    def do_test_throttle(self, ndrives, seconds, params):
+    def do_test_throttle(self, ndrives, seconds, params, first_drive = 0):
         def check_limit(limit, num):
             # IO throttling algorithm is discrete, allow 10% error so the test
             # is more robust
@@ -85,12 +85,14 @@ class ThrottleTestCase(iotests.QMPTestCase):
         # Send I/O requests to all drives
         for i in range(rd_nr):
             for drive in range(0, ndrives):
-                self.vm.hmp_qemu_io("drive%d" % drive, "aio_read %d %d" %
+                idx = first_drive + drive
+                self.vm.hmp_qemu_io("drive%d" % idx, "aio_read %d %d" %
                                     (i * rq_size, rq_size))
 
         for i in range(wr_nr):
             for drive in range(0, ndrives):
-                self.vm.hmp_qemu_io("drive%d" % drive, "aio_write %d %d" %
+                idx = first_drive + drive
+                self.vm.hmp_qemu_io("drive%d" % idx, "aio_write %d %d" %
                                     (i * rq_size, rq_size))
 
         # We'll store the I/O stats for each drive in these arrays
@@ -105,15 +107,17 @@ class ThrottleTestCase(iotests.QMPTestCase):
 
         # Read the stats before advancing the clock
         for i in range(0, ndrives):
+            idx = first_drive + i
             start_rd_bytes[i], start_rd_iops[i], start_wr_bytes[i], \
-                start_wr_iops[i] = self.blockstats('drive%d' % i)
+                start_wr_iops[i] = self.blockstats('drive%d' % idx)
 
         self.vm.qtest("clock_step %d" % ns)
 
         # Read the stats after advancing the clock
         for i in range(0, ndrives):
+            idx = first_drive + i
             end_rd_bytes[i], end_rd_iops[i], end_wr_bytes[i], \
-                end_wr_iops[i] = self.blockstats('drive%d' % i)
+                end_wr_iops[i] = self.blockstats('drive%d' % idx)
 
         # Check that the I/O is within the limits and evenly distributed
         for i in range(0, ndrives):
@@ -129,6 +133,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
             self.assertTrue(check_limit(params['iops_rd'], rd_iops))
             self.assertTrue(check_limit(params['iops_wr'], wr_iops))
 
+    # Connect N drives to a VM and test I/O in all of them
     def test_all(self):
         params = {"bps": 4096,
                   "bps_rd": 4096,
@@ -146,6 +151,24 @@ class ThrottleTestCase(iotests.QMPTestCase):
                 self.configure_throttle(ndrives, limits)
                 self.do_test_throttle(ndrives, 5, limits)
 
+    # Connect N drives to a VM and test I/O in just one of them a time
+    def test_one(self):
+        params = {"bps": 4096,
+                  "bps_rd": 4096,
+                  "bps_wr": 4096,
+                  "iops": 10,
+                  "iops_rd": 10,
+                  "iops_wr": 10,
+                 }
+        # Repeat the test for each one of the drives
+        for drive in range(0, self.max_drives):
+            # Pick each out of all possible params and test
+            for tk in params:
+                limits = dict([(k, 0) for k in params])
+                limits[tk] = params[tk] * self.max_drives
+                self.configure_throttle(self.max_drives, limits)
+                self.do_test_throttle(1, 5, limits, drive)
+
     def test_burst(self):
         params = {"bps": 4096,
                   "bps_rd": 4096,
diff --git a/tests/qemu-iotests/093.out b/tests/qemu-iotests/093.out
index 914e373..2f7d390 100644
--- a/tests/qemu-iotests/093.out
+++ b/tests/qemu-iotests/093.out
@@ -1,5 +1,5 @@
-.....
+.......
 ----------------------------------------------------------------------
-Ran 5 tests
+Ran 7 tests
 
 OK
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH v5 0/4] fdc: Use separate qdev device for drives

[John Snow]

On 10/24/2016 07:37 AM, Kevin Wolf wrote:
> We have been complaining for a long time about how the floppy controller and
> floppy drives are combined in a single qdev device and how this makes the
> device awkward to work with because it behaves different from all other block
> devices.
>
> The latest reason to complain was when I noticed that using qdev device names
> in QMP commands (e.g. for media change) doesn't really work when only the
> controller is a qdev device, but the drives aren't.
>
> So I decided to have a go at it, and this is the result.
>
> It doesn't actually change any of the inner workings of the floppy controller,
> but it wires things up differently on the qdev layer so that a floppy
> controller now exposes a bus on which the floppy drives sit. This results in a
> structure that is similar to IDE where the actual drive state is still in the
> controller and the qdev device basically just contains the qdev properties -
> not pretty, but quite workable.
>
> The commit message of patch 3 explains how to use it. In short, there is a
> '-device floppy' now and it does what you would expect if you ever used ide-cd.
>
> The other problem is old command lines, especially those using things like
> '-global isa-fdc,driveA=...'. In order to keep them working, we need to forward
> the property to an internally created floppy drive device. This is a bit like
> usb-storage, which we know is ugly, but works well enough in practice. The good
> news here is that in contrast to usb-storage, the floppy controller only does
> the forwarding for legacy configurations; as soon as you start using '-device
> floppy', it doesn't happen any more.
>
> So as you may have expected, this conversion doesn't result in a perfect
> device, but I think it's definitely an improvement over the old state. I hope
> you like it despite the warts. :-)
>
> v5:
> - Apply _filter_qemu to stderr, too [John]
> - Rename the bus to floppy-bus [Frederic]
> - Use FLOPPY_BUS() instead of DO_UPDATE() [Frederic]
>
> v4:
> - John says that his grep is broken and hangs at 100% CPU with my attempt to
>   extract the floppy controller from info qtree. Use a simpler sed command
>   instead (which, unlike the grep command, doesn't handle arbitrary indentation
>   level of the next item, but we know what comes next, so just hardcoding 10
>   spaces works, too).
>
> v3:
> - Fixed omissons in the conversion sysbus-fdc and the Sun one. Nice, combining
>   floppy controllers and weird platforms in a single series. [John]
>
> v2:
> - Added patch 4 (qemu-iotests case for floppy config on the command line)
> - Patch 2: Create a floppy device only if a BlockBackend exists instead of
>   always creating two of them
> - Patch 2: Initialise drive->fdctrl even if no drive is attached, it is
>   accessed anyway during migration
> - Patch 3: Keep 'type' qdev property and FDrive->drive in sync
> - Patch 3: Removed if with condition that is always true
>
> Alberto Garcia (2):
>   throttle: Correct access to wrong BlockBackendPublic structures
>   qemu-iotests: Test I/O in a single drive from a throttling group
>
> Halil Pasic (1):
>   block: improve error handling in raw_open
>
> Pino Toscano (1):
>   qapi: fix memory leak in bdrv_image_info_specific_dump
>

uhhh

http://i.imgur.com/60YQvmg.png

>  block/qapi.c               |  1 +
>  block/raw-posix.c          |  1 +
>  block/raw-win32.c          |  1 +
>  block/throttle-groups.c    | 27 +++++++++++++++++++++++----
>  tests/qemu-iotests/093     | 33 ++++++++++++++++++++++++++++-----
>  tests/qemu-iotests/093.out |  4 ++--
>  6 files changed, 56 insertions(+), 11 deletions(-)
>

Re: [Qemu-devel] [PATCH v5 0/4] fdc: Use separate qdev device for drives

[Kevin Wolf]

Am 24.10.2016 um 20:44 hat John Snow geschrieben:
> On 10/24/2016 07:37 AM, Kevin Wolf wrote:
> >We have been complaining for a long time about how the floppy controller and
> >floppy drives are combined in a single qdev device and how this makes the
> >device awkward to work with because it behaves different from all other block
> >devices.
>
> [...]
>
> >Alberto Garcia (2):
> >  throttle: Correct access to wrong BlockBackendPublic structures
> >  qemu-iotests: Test I/O in a single drive from a throttling group
> >
> >Halil Pasic (1):
> >  block: improve error handling in raw_open
> >
> >Pino Toscano (1):
> >  qapi: fix memory leak in bdrv_image_info_specific_dump
> >
> 
> uhhh
> 
> http://i.imgur.com/60YQvmg.png

wtf did I do there? :-/

I'll send a new version... But if it continues like this, we'll miss the
freeze...

Kevin