qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH 00/16] aio: first part of aio_context_acquire/release pushdown
@ 2016-01-15 15:12 Paolo Bonzini
  2016-01-15 15:12 ` [Qemu-devel] [PATCH 01/16] aio: introduce aio_context_in_iothread Paolo Bonzini
                   ` (17 more replies)
  0 siblings, 18 replies; 27+ messages in thread
From: Paolo Bonzini @ 2016-01-15 15:12 UTC (permalink / raw)
  To: qemu-devel; +Cc: stefanha

This is the infrastructure part of the aio_context_acquire/release pushdown,
which in turn is the first step towards a real multiqueue block layer in
QEMU.  The next step is to touch all the drivers and move calls to the
aio_context_acquire/release functions from aio-*.c to the drivers.  This
will be done in a separate patch series.

While the inserted lines are a lot, more than half of it are in documentation
and formal models of the code, as well as in the implementation of the new
"lockcnt" synchronization primitive.  The code is also very heavily commented.

The first four patches are new, as the issue they fix was found after posting
the previous patch.  Everything else is more or less the same as before.

Paolo

Paolo Bonzini (16):
  aio: introduce aio_context_in_iothread
  aio: do not really acquire/release the main AIO context
  aio: introduce aio_poll_internal
  aio: only call aio_poll_internal from iothread
  iothread: release AioContext around aio_poll
  qemu-thread: introduce QemuRecMutex
  aio: convert from RFifoLock to QemuRecMutex
  aio: rename bh_lock to list_lock
  qemu-thread: introduce QemuLockCnt
  aio: make ctx->list_lock a QemuLockCnt, subsuming ctx->walking_bh
  qemu-thread: optimize QemuLockCnt with futexes on Linux
  aio: tweak walking in dispatch phase
  aio-posix: remove walking_handlers, protecting AioHandler list with list_lock
  aio-win32: remove walking_handlers, protecting AioHandler list with list_lock
  aio: document locking
  aio: push aio_context_acquire/release down to dispatching

 aio-posix.c                     |  86 +++++----
 aio-win32.c                     | 106 ++++++-----
 async.c                         | 277 +++++++++++++++++++++++----
 block/io.c                      |  14 +-
 docs/aio_poll_drain.promela     | 210 +++++++++++++++++++++
 docs/aio_poll_drain_bug.promela | 158 ++++++++++++++++
 docs/aio_poll_sync_io.promela   |  88 +++++++++
 docs/lockcnt.txt                | 342 ++++++++++++++++++++++++++++++++++
 docs/multiple-iothreads.txt     |  48 ++---
 include/block/aio.h             |  69 ++++---
 include/qemu/futex.h            |  36 ++++
 include/qemu/rfifolock.h        |  54 ------
 include/qemu/thread-posix.h     |   6 +
 include/qemu/thread-win32.h     |  10 +
 include/qemu/thread.h           |  23 +++
 iothread.c                      |  20 +-
 stubs/iothread-lock.c           |   5 +
 tests/.gitignore                |   1 -
 tests/Makefile                  |   2 -
 tests/test-aio.c                |  19 +-
 tests/test-rfifolock.c          |  91 ---------
 trace-events                    |  10 +
 util/Makefile.objs              |   2 +-
 util/lockcnt.c                  | 404 ++++++++++++++++++++++++++++++++++++++++
 util/qemu-thread-posix.c        |  38 ++--
 util/qemu-thread-win32.c        |  25 +++
 util/rfifolock.c                |  78 --------
 27 files changed, 1779 insertions(+), 443 deletions(-)
 create mode 100644 docs/aio_poll_drain.promela
 create mode 100644 docs/aio_poll_drain_bug.promela
 create mode 100644 docs/aio_poll_sync_io.promela
 create mode 100644 docs/lockcnt.txt
 create mode 100644 include/qemu/futex.h
 delete mode 100644 include/qemu/rfifolock.h
 delete mode 100644 tests/test-rfifolock.c
 create mode 100644 util/lockcnt.c
 delete mode 100644 util/rfifolock.c

-- 
2.5.0

^ permalink raw reply	[flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v2 00/16] aio: first part of aio_context_acquire/release pushdown
@ 2016-02-08 16:14 Paolo Bonzini
  2016-02-08 16:14 ` [Qemu-devel] [PATCH 01/16] aio: introduce aio_context_in_iothread Paolo Bonzini
  0 siblings, 1 reply; 27+ messages in thread
From: Paolo Bonzini @ 2016-02-08 16:14 UTC (permalink / raw)
  To: qemu-devel; +Cc: stefanha

This is the infrastructure part of the aio_context_acquire/release pushdown,
which in turn is the first step towards a real multiqueue block layer in
QEMU.  The next step is to touch all the drivers and move calls to the
aio_context_acquire/release functions from aio-*.c to the drivers.  This
will be done in a separate patch series, which I plan to post before soft
freeze.

While the inserted lines are a lot, more than half of it are in documentation
and formal models of the code, as well as in the implementation of the new
"lockcnt" synchronization primitive.  The code is also very heavily commented.

The first four patches are new, as the issue they fix was found after posting
the previous patch.  Everything else is more or less the same as before.

Paolo

v1->v2: Update documentation [Stefan]
        Remove g_usleep from testcase [Stefan]

Paolo Bonzini (16):
  aio: introduce aio_context_in_iothread
  aio: do not really acquire/release the main AIO context
  aio: introduce aio_poll_internal
  aio: only call aio_poll_internal from iothread
  iothread: release AioContext around aio_poll
  qemu-thread: introduce QemuRecMutex
  aio: convert from RFifoLock to QemuRecMutex
  aio: rename bh_lock to list_lock
  qemu-thread: introduce QemuLockCnt
  aio: make ctx->list_lock a QemuLockCnt, subsuming ctx->walking_bh
  qemu-thread: optimize QemuLockCnt with futexes on Linux
  aio: tweak walking in dispatch phase
  aio-posix: remove walking_handlers, protecting AioHandler list with list_lock
  aio-win32: remove walking_handlers, protecting AioHandler list with list_lock
  aio: document locking
  aio: push aio_context_acquire/release down to dispatching

 aio-posix.c                     |  86 +++++----
 aio-win32.c                     | 106 ++++++-----
 async.c                         | 279 +++++++++++++++++++++++----
 block/io.c                      |  14 +-
 docs/aio_poll_drain.promela     | 210 +++++++++++++++++++++
 docs/aio_poll_drain_bug.promela | 158 ++++++++++++++++
 docs/aio_poll_sync_io.promela   |  88 +++++++++
 docs/lockcnt.txt                | 342 ++++++++++++++++++++++++++++++++++
 docs/multiple-iothreads.txt     |  63 ++++---
 include/block/aio.h             |  69 ++++---
 include/qemu/futex.h            |  36 ++++
 include/qemu/rfifolock.h        |  54 ------
 include/qemu/thread-posix.h     |   6 +
 include/qemu/thread-win32.h     |  10 +
 include/qemu/thread.h           |  23 +++
 iothread.c                      |  20 +-
 stubs/iothread-lock.c           |   5 +
 tests/.gitignore                |   1 -
 tests/Makefile                  |   2 -
 tests/test-aio.c                |  22 ++-
 tests/test-rfifolock.c          |  91 ---------
 trace-events                    |  10 +
 util/Makefile.objs              |   2 +-
 util/lockcnt.c                  | 404 ++++++++++++++++++++++++++++++++++++++++
 util/qemu-thread-posix.c        |  38 ++--
 util/qemu-thread-win32.c        |  25 +++
 util/rfifolock.c                |  78 --------
 27 files changed, 1792 insertions(+), 450 deletions(-)
 create mode 100644 docs/aio_poll_drain.promela
 create mode 100644 docs/aio_poll_drain_bug.promela
 create mode 100644 docs/aio_poll_sync_io.promela
 create mode 100644 docs/lockcnt.txt
 create mode 100644 include/qemu/futex.h
 delete mode 100644 include/qemu/rfifolock.h
 delete mode 100644 tests/test-rfifolock.c
 create mode 100644 util/lockcnt.c
 delete mode 100644 util/rfifolock.c

-- 
2.5.0

^ permalink raw reply	[flat|nested] 27+ messages in thread
* [Qemu-devel] [PATCH v3 00/16] aio: first part of aio_context_acquire/release pushdown
@ 2016-02-09 11:45 Paolo Bonzini
  2016-02-09 11:45 ` [Qemu-devel] [PATCH 01/16] aio: introduce aio_context_in_iothread Paolo Bonzini
  0 siblings, 1 reply; 27+ messages in thread
From: Paolo Bonzini @ 2016-02-09 11:45 UTC (permalink / raw)
  To: qemu-devel; +Cc: stefanha

This is the infrastructure part of the aio_context_acquire/release pushdown,
which in turn is the first step towards a real multiqueue block layer in
QEMU.  The next step is to touch all the drivers and move calls to the
aio_context_acquire/release functions from aio-*.c to the drivers.  This
will be done in a separate patch series, which I plan to post before soft
freeze.

While the inserted lines are a lot, more than half of it are in documentation
and formal models of the code, as well as in the implementation of the new
"lockcnt" synchronization primitive.  The code is also very heavily commented.

The first four patches are new, as the issue they fix was found after posting
the previous patch.  Everything else is more or less the same as before.

Paolo

v1->v2: Update documentation [Stefan]
        Remove g_usleep from testcase [Stefan]

v2->v3: Fix broken sentence [Eric]
        Use osdep.h [Eric]
        (v2->v3 diff after diffstat)

Paolo Bonzini (16):
  aio: introduce aio_context_in_iothread
  aio: do not really acquire/release the main AIO context
  aio: introduce aio_poll_internal
  aio: only call aio_poll_internal from iothread
  iothread: release AioContext around aio_poll
  qemu-thread: introduce QemuRecMutex
  aio: convert from RFifoLock to QemuRecMutex
  aio: rename bh_lock to list_lock
  qemu-thread: introduce QemuLockCnt
  aio: make ctx->list_lock a QemuLockCnt, subsuming ctx->walking_bh
  qemu-thread: optimize QemuLockCnt with futexes on Linux
  aio: tweak walking in dispatch phase
  aio-posix: remove walking_handlers, protecting AioHandler list with
    list_lock
  aio-win32: remove walking_handlers, protecting AioHandler list with
    list_lock
  aio: document locking
  aio: push aio_context_acquire/release down to dispatching

 aio-posix.c                     |  86 +++++----
 aio-win32.c                     | 106 ++++++-----
 async.c                         | 278 ++++++++++++++++++++++++----
 block/io.c                      |  14 +-
 docs/aio_poll_drain.promela     | 210 +++++++++++++++++++++
 docs/aio_poll_drain_bug.promela | 158 ++++++++++++++++
 docs/aio_poll_sync_io.promela   |  88 +++++++++
 docs/lockcnt.txt                | 342 ++++++++++++++++++++++++++++++++++
 docs/multiple-iothreads.txt     |  63 ++++---
 include/block/aio.h             |  69 ++++---
 include/qemu/futex.h            |  36 ++++
 include/qemu/rfifolock.h        |  54 ------
 include/qemu/thread-posix.h     |   6 +
 include/qemu/thread-win32.h     |  10 +
 include/qemu/thread.h           |  23 +++
 iothread.c                      |  20 +-
 stubs/iothread-lock.c           |   5 +
 tests/.gitignore                |   1 -
 tests/Makefile                  |   2 -
 tests/test-aio.c                |  22 ++-
 tests/test-rfifolock.c          |  91 ---------
 trace-events                    |  10 +
 util/Makefile.objs              |   2 +-
 util/lockcnt.c                  | 395 ++++++++++++++++++++++++++++++++++++++++
 util/qemu-thread-posix.c        |  38 ++--
 util/qemu-thread-win32.c        |  25 +++
 util/rfifolock.c                |  78 --------
 27 files changed, 1782 insertions(+), 450 deletions(-)
 create mode 100644 docs/aio_poll_drain.promela
 create mode 100644 docs/aio_poll_drain_bug.promela
 create mode 100644 docs/aio_poll_sync_io.promela
 create mode 100644 docs/lockcnt.txt
 create mode 100644 include/qemu/futex.h
 delete mode 100644 include/qemu/rfifolock.h
 delete mode 100644 tests/test-rfifolock.c
 create mode 100644 util/lockcnt.c
 delete mode 100644 util/rfifolock.c

-- 
2.5.0

v2->v3:

diff --git a/async.c b/async.c
index 9eab833..03a8e69 100644
--- a/async.c
+++ b/async.c
@@ -322,11 +322,10 @@ void aio_notify_accept(AioContext *ctx)
  * only, this only works when the calling thread holds the big QEMU lock.
  *
  * Because aio_poll is used in a loop, spurious wakeups are okay.
- * Therefore, the I/O thread calls qemu_event_set very liberally
- * (it helps that qemu_event_set is cheap on an already-set event).
- * generally used in a loop, it's okay to have spurious wakeups.
- * Similarly it is okay to return true when no progress was made
- * (as long as this doesn't happen forever, or you get livelock).
+ * Therefore, the I/O thread calls qemu_event_set very liberally;
+ * it helps that qemu_event_set is cheap on an already-set event.
+ * Similarly it is okay to return true when no progress was made,
+ * as long as this doesn't happen forever (or you get livelock).
  *
  * The important thing is that you need to report progress from
  * aio_poll(ctx, false) correctly.  This is complicated and the
diff --git a/util/lockcnt.c b/util/lockcnt.c
index 56eb29e..71e8f8f 100644
--- a/util/lockcnt.c
+++ b/util/lockcnt.c
@@ -6,16 +6,7 @@
  * Author:
  *   Paolo Bonzini <pbonzini@redhat.com>
  */
-#include <stdlib.h>
-#include <stdio.h>
-#include <errno.h>
-#include <time.h>
-#include <signal.h>
-#include <stdint.h>
-#include <string.h>
-#include <limits.h>
-#include <unistd.h>
-#include <sys/time.h>
+#include "qemu/osdep.h"
 #include "qemu/thread.h"
 #include "qemu/atomic.h"
 #include "trace.h"

^ permalink raw reply related	[flat|nested] 27+ messages in thread

end of thread, other threads:[~2016-02-09 11:46 UTC | newest]

Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-15 15:12 [Qemu-devel] [PATCH 00/16] aio: first part of aio_context_acquire/release pushdown Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 01/16] aio: introduce aio_context_in_iothread Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 02/16] aio: do not really acquire/release the main AIO context Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 03/16] aio: introduce aio_poll_internal Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 04/16] aio: only call aio_poll_internal from iothread Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 05/16] iothread: release AioContext around aio_poll Paolo Bonzini
2016-02-02 14:52   ` Stefan Hajnoczi
2016-02-02 15:01     ` Paolo Bonzini
2016-02-03  9:34       ` Stefan Hajnoczi
2016-02-03  9:52         ` Paolo Bonzini
2016-02-05 17:51           ` Stefan Hajnoczi
2016-02-08 15:34             ` Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 06/16] qemu-thread: introduce QemuRecMutex Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 07/16] aio: convert from RFifoLock to QemuRecMutex Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 08/16] aio: rename bh_lock to list_lock Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 09/16] qemu-thread: introduce QemuLockCnt Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 10/16] aio: make ctx->list_lock a QemuLockCnt, subsuming ctx->walking_bh Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 11/16] qemu-thread: optimize QemuLockCnt with futexes on Linux Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 12/16] aio: tweak walking in dispatch phase Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 13/16] aio-posix: remove walking_handlers, protecting AioHandler list with list_lock Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 14/16] aio-win32: " Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 15/16] aio: document locking Paolo Bonzini
2016-01-15 15:12 ` [Qemu-devel] [PATCH 16/16] aio: push aio_context_acquire/release down to dispatching Paolo Bonzini
2016-02-02  9:41 ` [Qemu-devel] [PATCH 00/16] aio: first part of aio_context_acquire/release pushdown Paolo Bonzini
2016-02-02 17:26 ` Stefan Hajnoczi
2016-02-08 16:14 [Qemu-devel] [PATCH v2 " Paolo Bonzini
2016-02-08 16:14 ` [Qemu-devel] [PATCH 01/16] aio: introduce aio_context_in_iothread Paolo Bonzini
2016-02-09 11:45 [Qemu-devel] [PATCH v3 00/16] aio: first part of aio_context_acquire/release pushdown Paolo Bonzini
2016-02-09 11:45 ` [Qemu-devel] [PATCH 01/16] aio: introduce aio_context_in_iothread Paolo Bonzini

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).