* [PATCH v10 00/21] job: replace AioContext lock with job_mutex
@ 2022-07-25 7:38 Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 01/21] job.c: make job_mutex and job_lock/unlock() public Emanuele Giuseppe Esposito
` (20 more replies)
0 siblings, 21 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito
In this series, we want to remove the AioContext lock and instead
use the already existent job_mutex to protect the job structures
and list. This is part of the work to get rid of AioContext lock
usage in favour of smaller granularity locks.
In order to simplify reviewer's job, job lock/unlock functions and
macros are added as empty prototypes (nop) in patch 1.
They are converted to use the actual job mutex only in the last
patch. In this way we can freely create locking sections
without worrying about deadlocks with the aiocontext lock.
Patch 2 defines what fields in the job structure need protection.
Patches 3-6 are in preparation to the job locks, moving functions
from global to static and introducing helpers.
Patch 7-9 introduce the (nop) job lock into the job API and
its users, and patches 10-13 categorize respectively locked and
unlocked functions in the job API.
Patches 14-17 take care of protecting job->aio_context, and
finally patch 18 makes the prototypes in patch 1 use the
job_mutex and removes all aiocontext lock at the same time.
Tested this series by running unit tests, qemu-iotests and qtests
(x86_64).
---
v10:
* protect job->status in unit tests
* patch 11: change commit description and avoid using lock guard for a single
function call
* move patch 19 before patch 15
v9:
* merge patch 6 and 7 to 5.
* additional "taken with job lock/unlock" added and propagated in callers
* protect iostatus field of BlockJobs
* move all blockjob patches torward the end of the serie
v8:
* reorganize patch ordering according with Vladimir proposal
* minor nitpicks
v7:
* s/temporary/temporarly
* double identical locking comment to the same function
* patch 2: add "Protected by AioContext lock" to better categorize fields in
job.h
* use same comment style in all function headers ("Just like {funct}, but
called between job_lock and job_unlock")
v6:
* patch 4 and 6 squashed with patch 19 (enable job lock and
reduce/remove AioContext lock)
* patch 19: job_unref_locked read the aiocontext inside the
job lock.
v5:
* just restructured patches a little bit better, as there were
functions used before they were defined.
* rebased on kwolf/block branch and API split serie
v4:
* move "protected by job_mutex" from patch 2 to 15, where the job_mutex is
actually added.
* s/aio_co_enter/aio_co_schedule in job.c, and adjust tests accordingly.
* remove job_get_aio_context, add job_set_aio_context. Use "fake rwlock"
to protect job->aiocontext.
* get rid of useless getters method, namely:
job_get_status
job_get_pause_count
job_get_paused
job_get_busy
They are all used only by tests, and such getter is pretty useless.
Replace with job_lock(); assert(); job_unlock();
* use job lock macros instead of job lock/unlock in unit tests.
* convert also blockjob functions to have _locked
* put the job_lock/unlock patches before the _locked ones
* replace aio_co_enter in job.c and detect change of context
v3:
* add "_locked" suffix to the functions called under job_mutex lock
* rename _job_lock in real_job_lock
* job_mutex is now public, and drivers like monitor use it directly
* introduce and protect job_get_aio_context
* remove mirror-specific APIs and just use WITH_JOB_GUARD
* more extensive use of WITH_JOB_GUARD and JOB_LOCK_GUARD
RFC v2:
* use JOB_LOCK_GUARD and WITH_JOB_LOCK_GUARD
* mu(u)ltiple typos in commit messages
* job API split patches are sent separately in another series
* use of empty job_{lock/unlock} and JOB_LOCK_GUARD/WITH_JOB_LOCK_GUARD
to avoid deadlocks and simplify the reviewer job
* move patch 11 (block_job_query: remove atomic read) as last
Emanuele Giuseppe Esposito (20):
job.c: make job_mutex and job_lock/unlock() public
job.h: categorize fields in struct Job
job.c: API functions not used outside should be static
aio-wait.h: introduce AIO_WAIT_WHILE_UNLOCKED
job.c: add job_lock/unlock while keeping job.h intact
job: move and update comments from blockjob.c
blockjob: introduce block_job _locked() APIs
jobs: add job lock in find_* functions
jobs: use job locks also in the unit tests
block/mirror.c: use of job helpers in drivers to avoid TOC/TOU
jobs: group together API calls under the same job lock
commit and mirror: create new nodes using bdrv_get_aio_context, and
not the job aiocontext
jobs: protect job.aio_context with BQL and job_mutex
blockjob.h: categorize fields in struct BlockJob
blockjob: rename notifier callbacks as _locked
blockjob: protect iostatus field in BlockJob struct
job.c: enable job lock/unlock and remove Aiocontext locks
block_job_query: remove atomic read
blockjob: remove unused functions
job: remove unused functions
Paolo Bonzini (1):
job: detect change of aiocontext within job coroutine
block.c | 17 +-
block/commit.c | 4 +-
block/mirror.c | 21 +-
block/replication.c | 6 +-
blockdev.c | 129 +++---
blockjob.c | 132 ++++---
include/block/aio-wait.h | 17 +-
include/block/blockjob.h | 44 ++-
include/qemu/job.h | 267 +++++++++----
job-qmp.c | 87 ++--
job.c | 656 +++++++++++++++++++------------
monitor/qmp-cmds.c | 7 +-
qemu-img.c | 37 +-
tests/unit/test-bdrv-drain.c | 80 ++--
tests/unit/test-block-iothread.c | 8 +-
tests/unit/test-blockjob-txn.c | 24 +-
tests/unit/test-blockjob.c | 129 +++---
17 files changed, 1026 insertions(+), 639 deletions(-)
--
2.31.1
^ permalink raw reply [flat|nested] 71+ messages in thread
* [PATCH v10 01/21] job.c: make job_mutex and job_lock/unlock() public
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 02/21] job.h: categorize fields in struct Job Emanuele Giuseppe Esposito
` (19 subsequent siblings)
20 siblings, 0 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
job mutex will be used to protect the job struct elements and list,
replacing AioContext locks.
Right now use a shared lock for all jobs, in order to keep things
simple. Once the AioContext lock is gone, we can introduce per-job
locks.
To simplify the switch from aiocontext to job lock, introduce
*nop* lock/unlock functions and macros.
We want to always call job_lock/unlock outside the AioContext locks,
and not vice-versa, otherwise we might get a deadlock. This is not
straightforward to do, and that's why we start with nop functions.
Once everything is protected by job_lock/unlock, we can change the nop into
an actual mutex and remove the aiocontext lock.
Since job_mutex is already being used, add static
real_job_{lock/unlock} for the existing usage.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
---
include/qemu/job.h | 24 ++++++++++++++++++++++++
job.c | 35 +++++++++++++++++++++++------------
2 files changed, 47 insertions(+), 12 deletions(-)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index c105b31076..d1192ffd61 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -303,6 +303,30 @@ typedef enum JobCreateFlags {
JOB_MANUAL_DISMISS = 0x04,
} JobCreateFlags;
+extern QemuMutex job_mutex;
+
+#define JOB_LOCK_GUARD() /* QEMU_LOCK_GUARD(&job_mutex) */
+
+#define WITH_JOB_LOCK_GUARD() /* WITH_QEMU_LOCK_GUARD(&job_mutex) */
+
+/**
+ * job_lock:
+ *
+ * Take the mutex protecting the list of jobs and their status.
+ * Most functions called by the monitor need to call job_lock
+ * and job_unlock manually. On the other hand, function called
+ * by the block jobs themselves and by the block layer will take the
+ * lock for you.
+ */
+void job_lock(void);
+
+/**
+ * job_unlock:
+ *
+ * Release the mutex protecting the list of jobs and their status.
+ */
+void job_unlock(void);
+
/**
* Allocate and return a new job transaction. Jobs can be added to the
* transaction using job_txn_add_job().
diff --git a/job.c b/job.c
index 075c6f3a20..2b4ffca9d4 100644
--- a/job.c
+++ b/job.c
@@ -32,6 +32,12 @@
#include "trace/trace-root.h"
#include "qapi/qapi-events-job.h"
+/*
+ * job_mutex protects the jobs list, but also makes the
+ * struct job fields thread-safe.
+ */
+QemuMutex job_mutex;
+
static QLIST_HEAD(, Job) jobs = QLIST_HEAD_INITIALIZER(jobs);
/* Job State Transition Table */
@@ -74,17 +80,22 @@ struct JobTxn {
int refcnt;
};
-/* Right now, this mutex is only needed to synchronize accesses to job->busy
- * and job->sleep_timer, such as concurrent calls to job_do_yield and
- * job_enter. */
-static QemuMutex job_mutex;
+void job_lock(void)
+{
+ /* nop */
+}
+
+void job_unlock(void)
+{
+ /* nop */
+}
-static void job_lock(void)
+static void real_job_lock(void)
{
qemu_mutex_lock(&job_mutex);
}
-static void job_unlock(void)
+static void real_job_unlock(void)
{
qemu_mutex_unlock(&job_mutex);
}
@@ -450,21 +461,21 @@ void job_enter_cond(Job *job, bool(*fn)(Job *job))
return;
}
- job_lock();
+ real_job_lock();
if (job->busy) {
- job_unlock();
+ real_job_unlock();
return;
}
if (fn && !fn(job)) {
- job_unlock();
+ real_job_unlock();
return;
}
assert(!job->deferred_to_main_loop);
timer_del(&job->sleep_timer);
job->busy = true;
- job_unlock();
+ real_job_unlock();
aio_co_enter(job->aio_context, job->co);
}
@@ -481,13 +492,13 @@ void job_enter(Job *job)
* called explicitly. */
static void coroutine_fn job_do_yield(Job *job, uint64_t ns)
{
- job_lock();
+ real_job_lock();
if (ns != -1) {
timer_mod(&job->sleep_timer, ns);
}
job->busy = false;
job_event_idle(job);
- job_unlock();
+ real_job_unlock();
qemu_coroutine_yield();
/* Set by job_enter_cond() before re-entering the coroutine. */
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 02/21] job.h: categorize fields in struct Job
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 01/21] job.c: make job_mutex and job_lock/unlock() public Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-29 12:30 ` Kevin Wolf
2022-08-16 18:28 ` Stefan Hajnoczi
2022-07-25 7:38 ` [PATCH v10 03/21] job.c: API functions not used outside should be static Emanuele Giuseppe Esposito
` (18 subsequent siblings)
20 siblings, 2 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
Categorize the fields in struct Job to understand which ones
need to be protected by the job mutex and which don't.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
---
include/qemu/job.h | 61 +++++++++++++++++++++++++++-------------------
1 file changed, 36 insertions(+), 25 deletions(-)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index d1192ffd61..876e13d549 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -40,27 +40,52 @@ typedef struct JobTxn JobTxn;
* Long-running operation.
*/
typedef struct Job {
+
+ /* Fields set at initialization (job_create), and never modified */
+
/** The ID of the job. May be NULL for internal jobs. */
char *id;
- /** The type of this job. */
+ /**
+ * The type of this job.
+ * All callbacks are called with job_mutex *not* held.
+ */
const JobDriver *driver;
- /** Reference count of the block job */
- int refcnt;
-
- /** Current state; See @JobStatus for details. */
- JobStatus status;
-
- /** AioContext to run the job coroutine in */
- AioContext *aio_context;
-
/**
* The coroutine that executes the job. If not NULL, it is reentered when
* busy is false and the job is cancelled.
+ * Initialized in job_start()
*/
Coroutine *co;
+ /** True if this job should automatically finalize itself */
+ bool auto_finalize;
+
+ /** True if this job should automatically dismiss itself */
+ bool auto_dismiss;
+
+ /** The completion function that will be called when the job completes. */
+ BlockCompletionFunc *cb;
+
+ /** The opaque value that is passed to the completion function. */
+ void *opaque;
+
+ /* ProgressMeter API is thread-safe */
+ ProgressMeter progress;
+
+
+ /** Protected by AioContext lock */
+
+ /** AioContext to run the job coroutine in */
+ AioContext *aio_context;
+
+ /** Reference count of the block job */
+ int refcnt;
+
+ /** Current state; See @JobStatus for details. */
+ JobStatus status;
+
/**
* Timer that is used by @job_sleep_ns. Accessed under job_mutex (in
* job.c).
@@ -112,14 +137,6 @@ typedef struct Job {
/** Set to true when the job has deferred work to the main loop. */
bool deferred_to_main_loop;
- /** True if this job should automatically finalize itself */
- bool auto_finalize;
-
- /** True if this job should automatically dismiss itself */
- bool auto_dismiss;
-
- ProgressMeter progress;
-
/**
* Return code from @run and/or @prepare callback(s).
* Not final until the job has reached the CONCLUDED status.
@@ -134,12 +151,6 @@ typedef struct Job {
*/
Error *err;
- /** The completion function that will be called when the job completes. */
- BlockCompletionFunc *cb;
-
- /** The opaque value that is passed to the completion function. */
- void *opaque;
-
/** Notifiers called when a cancelled job is finalised */
NotifierList on_finalize_cancelled;
@@ -167,6 +178,7 @@ typedef struct Job {
/**
* Callbacks and other information about a Job driver.
+ * All callbacks are invoked with job_mutex *not* held.
*/
struct JobDriver {
@@ -472,7 +484,6 @@ void job_yield(Job *job);
*/
void coroutine_fn job_sleep_ns(Job *job, int64_t ns);
-
/** Returns the JobType of a given Job. */
JobType job_type(const Job *job);
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 03/21] job.c: API functions not used outside should be static
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 01/21] job.c: make job_mutex and job_lock/unlock() public Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 02/21] job.h: categorize fields in struct Job Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-29 12:30 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 04/21] aio-wait.h: introduce AIO_WAIT_WHILE_UNLOCKED Emanuele Giuseppe Esposito
` (17 subsequent siblings)
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
job_event_* functions can all be static, as they are not used
outside job.c.
Same applies for job_txn_add_job().
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
---
include/qemu/job.h | 18 ------------------
job.c | 22 +++++++++++++++++++---
2 files changed, 19 insertions(+), 21 deletions(-)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index 876e13d549..4b64eb15f7 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -358,18 +358,6 @@ JobTxn *job_txn_new(void);
*/
void job_txn_unref(JobTxn *txn);
-/**
- * @txn: The transaction (may be NULL)
- * @job: Job to add to the transaction
- *
- * Add @job to the transaction. The @job must not already be in a transaction.
- * The caller must call either job_txn_unref() or job_completed() to release
- * the reference that is automatically grabbed here.
- *
- * If @txn is NULL, the function does nothing.
- */
-void job_txn_add_job(JobTxn *txn, Job *job);
-
/**
* Create a new long-running job and return it.
*
@@ -431,12 +419,6 @@ void job_progress_set_remaining(Job *job, uint64_t remaining);
*/
void job_progress_increase_remaining(Job *job, uint64_t delta);
-/** To be called when a cancelled job is finalised. */
-void job_event_cancelled(Job *job);
-
-/** To be called when a successfully completed job is finalised. */
-void job_event_completed(Job *job);
-
/**
* Conditionally enter the job coroutine if the job is ready to run, not
* already busy and fn() returns true. fn() is called while under the job_lock
diff --git a/job.c b/job.c
index 2b4ffca9d4..cafd597ba4 100644
--- a/job.c
+++ b/job.c
@@ -125,7 +125,17 @@ void job_txn_unref(JobTxn *txn)
}
}
-void job_txn_add_job(JobTxn *txn, Job *job)
+/**
+ * @txn: The transaction (may be NULL)
+ * @job: Job to add to the transaction
+ *
+ * Add @job to the transaction. The @job must not already be in a transaction.
+ * The caller must call either job_txn_unref() or job_completed() to release
+ * the reference that is automatically grabbed here.
+ *
+ * If @txn is NULL, the function does nothing.
+ */
+static void job_txn_add_job(JobTxn *txn, Job *job)
{
if (!txn) {
return;
@@ -427,12 +437,18 @@ void job_progress_increase_remaining(Job *job, uint64_t delta)
progress_increase_remaining(&job->progress, delta);
}
-void job_event_cancelled(Job *job)
+/**
+ * To be called when a cancelled job is finalised.
+ */
+static void job_event_cancelled(Job *job)
{
notifier_list_notify(&job->on_finalize_cancelled, job);
}
-void job_event_completed(Job *job)
+/**
+ * To be called when a successfully completed job is finalised.
+ */
+static void job_event_completed(Job *job)
{
notifier_list_notify(&job->on_finalize_completed, job);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 04/21] aio-wait.h: introduce AIO_WAIT_WHILE_UNLOCKED
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (2 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 03/21] job.c: API functions not used outside should be static Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-29 12:33 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact Emanuele Giuseppe Esposito
` (16 subsequent siblings)
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
Same as AIO_WAIT_WHILE macro, but if we are in the Main loop
do not release and then acquire ctx_ 's aiocontext.
Once all Aiocontext locks go away, this macro will replace
AIO_WAIT_WHILE.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
---
include/block/aio-wait.h | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/include/block/aio-wait.h b/include/block/aio-wait.h
index 54840f8622..a61f82c617 100644
--- a/include/block/aio-wait.h
+++ b/include/block/aio-wait.h
@@ -59,10 +59,13 @@ typedef struct {
extern AioWait global_aio_wait;
/**
- * AIO_WAIT_WHILE:
+ * _AIO_WAIT_WHILE:
* @ctx: the aio context, or NULL if multiple aio contexts (for which the
* caller does not hold a lock) are involved in the polling condition.
* @cond: wait while this conditional expression is true
+ * @unlock: whether to unlock and then lock again @ctx. This apples
+ * only when waiting for another AioContext from the main loop.
+ * Otherwise it's ignored.
*
* Wait while a condition is true. Use this to implement synchronous
* operations that require event loop activity.
@@ -75,7 +78,7 @@ extern AioWait global_aio_wait;
* wait on conditions between two IOThreads since that could lead to deadlock,
* go via the main loop instead.
*/
-#define AIO_WAIT_WHILE(ctx, cond) ({ \
+#define _AIO_WAIT_WHILE(ctx, cond, unlock) ({ \
bool waited_ = false; \
AioWait *wait_ = &global_aio_wait; \
AioContext *ctx_ = (ctx); \
@@ -92,11 +95,11 @@ extern AioWait global_aio_wait;
assert(qemu_get_current_aio_context() == \
qemu_get_aio_context()); \
while ((cond)) { \
- if (ctx_) { \
+ if (unlock && ctx_) { \
aio_context_release(ctx_); \
} \
aio_poll(qemu_get_aio_context(), true); \
- if (ctx_) { \
+ if (unlock && ctx_) { \
aio_context_acquire(ctx_); \
} \
waited_ = true; \
@@ -105,6 +108,12 @@ extern AioWait global_aio_wait;
qatomic_dec(&wait_->num_waiters); \
waited_; })
+#define AIO_WAIT_WHILE(ctx, cond) \
+ _AIO_WAIT_WHILE(ctx, cond, true)
+
+#define AIO_WAIT_WHILE_UNLOCKED(ctx, cond) \
+ _AIO_WAIT_WHILE(ctx, cond, false)
+
/**
* aio_wait_kick:
* Wake up the main thread if it is waiting on AIO_WAIT_WHILE(). During
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (3 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 04/21] aio-wait.h: introduce AIO_WAIT_WHILE_UNLOCKED Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-27 10:45 ` Vladimir Sementsov-Ogievskiy
` (2 more replies)
2022-07-25 7:38 ` [PATCH v10 06/21] job: move and update comments from blockjob.c Emanuele Giuseppe Esposito
` (15 subsequent siblings)
20 siblings, 3 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
With "intact" we mean that all job.h functions implicitly
take the lock. Therefore API callers are unmodified.
This means that:
- many static functions that will be always called with job lock held
become _locked, and call _locked functions
- all public functions take the lock internally if needed, and call _locked
functions
- all public functions called internally by other functions in job.c will have a
_locked counterpart (sometimes public), to avoid deadlocks (job lock already taken).
These functions are not used for now.
- some public functions called only from exernal files (not job.c) do not
have _locked() counterpart and take the lock inside. Others won't need
the lock at all because use fields only set at initialization and
never modified.
job_{lock/unlock} is independent from real_job_{lock/unlock}.
Note: at this stage, job_{lock/unlock} and job lock guard macros
are *nop*
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
include/qemu/job.h | 138 ++++++++++-
job.c | 600 +++++++++++++++++++++++++++++++--------------
2 files changed, 553 insertions(+), 185 deletions(-)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index 4b64eb15f7..5709e8d4a8 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -358,8 +358,15 @@ JobTxn *job_txn_new(void);
*/
void job_txn_unref(JobTxn *txn);
+/*
+ * Same as job_txn_unref(), but called with job lock held.
+ * Might release the lock temporarily.
+ */
+void job_txn_unref_locked(JobTxn *txn);
+
/**
* Create a new long-running job and return it.
+ * Called with job_mutex *not* held.
*
* @job_id: The id of the newly-created job, or %NULL for internal jobs
* @driver: The class object for the newly-created job.
@@ -380,17 +387,25 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
*/
void job_ref(Job *job);
+/* Same as job_ref(), but called with job lock held. */
+void job_ref_locked(Job *job);
+
/**
* Release a reference that was previously acquired with job_ref() or
* job_create(). If it's the last reference to the object, it will be freed.
*/
void job_unref(Job *job);
+/* Same as job_unref(), but called with job lock held. */
+void job_unref_locked(Job *job);
+
/**
* @job: The job that has made progress
* @done: How much progress the job made since the last call
*
* Updates the progress counter of the job.
+ *
+ * May be called with mutex held or not held.
*/
void job_progress_update(Job *job, uint64_t done);
@@ -401,6 +416,8 @@ void job_progress_update(Job *job, uint64_t done);
*
* Sets the expected end value of the progress counter of a job so that a
* completion percentage can be calculated when the progress is updated.
+ *
+ * May be called with mutex held or not held.
*/
void job_progress_set_remaining(Job *job, uint64_t remaining);
@@ -416,6 +433,8 @@ void job_progress_set_remaining(Job *job, uint64_t remaining);
* length before, and job_progress_update() afterwards.
* (So the operation acts as a parenthesis in regards to the main job
* operation running in background.)
+ *
+ * May be called with mutex held or not held.
*/
void job_progress_increase_remaining(Job *job, uint64_t delta);
@@ -426,11 +445,19 @@ void job_progress_increase_remaining(Job *job, uint64_t delta);
*/
void job_enter_cond(Job *job, bool(*fn)(Job *job));
+/*
+ * Same as job_enter_cond(), but called with job lock held.
+ * Might release the lock temporarily.
+ */
+void job_enter_cond_locked(Job *job, bool(*fn)(Job *job));
+
/**
* @job: A job that has not yet been started.
*
* Begins execution of a job.
* Takes ownership of one reference to the job object.
+ *
+ * Called with job_mutex *not* held.
*/
void job_start(Job *job);
@@ -438,6 +465,7 @@ void job_start(Job *job);
* @job: The job to enter.
*
* Continue the specified job by entering the coroutine.
+ * Called with job_mutex *not* held.
*/
void job_enter(Job *job);
@@ -446,6 +474,8 @@ void job_enter(Job *job);
*
* Pause now if job_pause() has been called. Jobs that perform lots of I/O
* must call this between requests so that the job can be paused.
+ *
+ * Called with job_mutex *not* held.
*/
void coroutine_fn job_pause_point(Job *job);
@@ -453,6 +483,7 @@ void coroutine_fn job_pause_point(Job *job);
* @job: The job that calls the function.
*
* Yield the job coroutine.
+ * Called with job_mutex *not* held.
*/
void job_yield(Job *job);
@@ -463,6 +494,8 @@ void job_yield(Job *job);
* Put the job to sleep (assuming that it wasn't canceled) for @ns
* %QEMU_CLOCK_REALTIME nanoseconds. Canceling the job will immediately
* interrupt the wait.
+ *
+ * Called with job_mutex *not* held.
*/
void coroutine_fn job_sleep_ns(Job *job, int64_t ns);
@@ -475,21 +508,40 @@ const char *job_type_str(const Job *job);
/** Returns true if the job should not be visible to the management layer. */
bool job_is_internal(Job *job);
-/** Returns whether the job is being cancelled. */
+/**
+ * Returns whether the job is being cancelled.
+ * Called with job_mutex *not* held.
+ */
bool job_is_cancelled(Job *job);
+/* Same as job_is_cancelled(), but called with job lock held. */
+bool job_is_cancelled_locked(Job *job);
+
/**
* Returns whether the job is scheduled for cancellation (at an
* indefinite point).
+ * Called with job_mutex *not* held.
*/
bool job_cancel_requested(Job *job);
-/** Returns whether the job is in a completed state. */
+/**
+ * Returns whether the job is in a completed state.
+ * Called with job_mutex *not* held.
+ */
bool job_is_completed(Job *job);
-/** Returns whether the job is ready to be completed. */
+/* Same as job_is_completed(), but called with job lock held. */
+bool job_is_completed_locked(Job *job);
+
+/**
+ * Returns whether the job is ready to be completed.
+ * Called with job_mutex *not* held.
+ */
bool job_is_ready(Job *job);
+/* Same as job_is_ready(), but called with job lock held. */
+bool job_is_ready_locked(Job *job);
+
/**
* Request @job to pause at the next pause point. Must be paired with
* job_resume(). If the job is supposed to be resumed by user action, call
@@ -497,24 +549,45 @@ bool job_is_ready(Job *job);
*/
void job_pause(Job *job);
+/* Same as job_pause(), but called with job lock held. */
+void job_pause_locked(Job *job);
+
/** Resumes a @job paused with job_pause. */
void job_resume(Job *job);
+/*
+ * Same as job_resume(), but called with job lock held.
+ * Might release the lock temporarily.
+ */
+void job_resume_locked(Job *job);
+
/**
* Asynchronously pause the specified @job.
* Do not allow a resume until a matching call to job_user_resume.
*/
void job_user_pause(Job *job, Error **errp);
+/* Same as job_user_pause(), but called with job lock held. */
+void job_user_pause_locked(Job *job, Error **errp);
+
/** Returns true if the job is user-paused. */
bool job_user_paused(Job *job);
+/* Same as job_user_paused(), but called with job lock held. */
+bool job_user_paused_locked(Job *job);
+
/**
* Resume the specified @job.
* Must be paired with a preceding job_user_pause.
*/
void job_user_resume(Job *job, Error **errp);
+/*
+ * Same as job_user_resume(), but called with job lock held.
+ * Might release the lock temporarily.
+ */
+void job_user_resume_locked(Job *job, Error **errp);
+
/**
* Get the next element from the list of block jobs after @job, or the
* first one if @job is %NULL.
@@ -523,6 +596,9 @@ void job_user_resume(Job *job, Error **errp);
*/
Job *job_next(Job *job);
+/* Same as job_next(), but called with job lock held. */
+Job *job_next_locked(Job *job);
+
/**
* Get the job identified by @id (which must not be %NULL).
*
@@ -530,6 +606,9 @@ Job *job_next(Job *job);
*/
Job *job_get(const char *id);
+/* Same as job_get(), but called with job lock held. */
+Job *job_get_locked(const char *id);
+
/**
* Check whether the verb @verb can be applied to @job in its current state.
* Returns 0 if the verb can be applied; otherwise errp is set and -EPERM
@@ -537,27 +616,48 @@ Job *job_get(const char *id);
*/
int job_apply_verb(Job *job, JobVerb verb, Error **errp);
-/** The @job could not be started, free it. */
+/* Same as job_apply_verb, but called with job lock held. */
+int job_apply_verb_locked(Job *job, JobVerb verb, Error **errp);
+
+/**
+ * The @job could not be started, free it.
+ * Called with job_mutex *not* held.
+ */
void job_early_fail(Job *job);
-/** Moves the @job from RUNNING to READY */
+/**
+ * Moves the @job from RUNNING to READY.
+ * Called with job_mutex *not* held.
+ */
void job_transition_to_ready(Job *job);
/** Asynchronously complete the specified @job. */
void job_complete(Job *job, Error **errp);
+/*
+ * Same as job_complete(), but called with job lock held.
+ * Might release the lock temporarily.
+ */
+void job_complete_locked(Job *job, Error **errp);
+
/**
* Asynchronously cancel the specified @job. If @force is true, the job should
* be cancelled immediately without waiting for a consistent state.
*/
void job_cancel(Job *job, bool force);
+/* Same as job_cancel(), but called with job lock held. */
+void job_cancel_locked(Job *job, bool force);
+
/**
* Cancels the specified job like job_cancel(), but may refuse to do so if the
* operation isn't meaningful in the current state of the job.
*/
void job_user_cancel(Job *job, bool force, Error **errp);
+/* Same as job_user_cancel(), but called with job lock held. */
+void job_user_cancel_locked(Job *job, bool force, Error **errp);
+
/**
* Synchronously cancel the @job. The completion callback is called
* before the function returns. If @force is false, the job may
@@ -571,7 +671,14 @@ void job_user_cancel(Job *job, bool force, Error **errp);
*/
int job_cancel_sync(Job *job, bool force);
-/** Synchronously force-cancels all jobs using job_cancel_sync(). */
+/* Same as job_cancel_sync, but called with job lock held. */
+int job_cancel_sync_locked(Job *job, bool force);
+
+/**
+ * Synchronously force-cancels all jobs using job_cancel_sync_locked().
+ *
+ * Called with job_lock *not* held.
+ */
void job_cancel_sync_all(void);
/**
@@ -590,6 +697,9 @@ void job_cancel_sync_all(void);
*/
int job_complete_sync(Job *job, Error **errp);
+/* Same as job_complete_sync, but called with job lock held. */
+int job_complete_sync_locked(Job *job, Error **errp);
+
/**
* For a @job that has finished its work and is pending awaiting explicit
* acknowledgement to commit its work, this will commit that work.
@@ -600,12 +710,18 @@ int job_complete_sync(Job *job, Error **errp);
*/
void job_finalize(Job *job, Error **errp);
+/* Same as job_finalize(), but called with job lock held. */
+void job_finalize_locked(Job *job, Error **errp);
+
/**
* Remove the concluded @job from the query list and resets the passed pointer
* to %NULL. Returns an error if the job is not actually concluded.
*/
void job_dismiss(Job **job, Error **errp);
+/* Same as job_dismiss(), but called with job lock held. */
+void job_dismiss_locked(Job **job, Error **errp);
+
/**
* Synchronously finishes the given @job. If @finish is given, it is called to
* trigger completion or cancellation of the job.
@@ -615,6 +731,14 @@ void job_dismiss(Job **job, Error **errp);
*
* Callers must hold the AioContext lock of job->aio_context.
*/
-int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp);
+int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
+ Error **errp);
+
+/*
+ * Same as job_finish_sync(), but called with job lock held.
+ * Might release the lock temporarily.
+ */
+int job_finish_sync_locked(Job *job, void (*finish)(Job *, Error **errp),
+ Error **errp);
#endif
diff --git a/job.c b/job.c
index cafd597ba4..ae25db97ac 100644
--- a/job.c
+++ b/job.c
@@ -38,6 +38,7 @@
*/
QemuMutex job_mutex;
+/* Protected by job_mutex */
static QLIST_HEAD(, Job) jobs = QLIST_HEAD_INITIALIZER(jobs);
/* Job State Transition Table */
@@ -113,18 +114,25 @@ JobTxn *job_txn_new(void)
return txn;
}
-static void job_txn_ref(JobTxn *txn)
+/* Called with job_mutex held. */
+static void job_txn_ref_locked(JobTxn *txn)
{
txn->refcnt++;
}
-void job_txn_unref(JobTxn *txn)
+void job_txn_unref_locked(JobTxn *txn)
{
if (txn && --txn->refcnt == 0) {
g_free(txn);
}
}
+void job_txn_unref(JobTxn *txn)
+{
+ JOB_LOCK_GUARD();
+ job_txn_unref_locked(txn);
+}
+
/**
* @txn: The transaction (may be NULL)
* @job: Job to add to the transaction
@@ -134,8 +142,10 @@ void job_txn_unref(JobTxn *txn)
* the reference that is automatically grabbed here.
*
* If @txn is NULL, the function does nothing.
+ *
+ * Called with job_mutex held.
*/
-static void job_txn_add_job(JobTxn *txn, Job *job)
+static void job_txn_add_job_locked(JobTxn *txn, Job *job)
{
if (!txn) {
return;
@@ -145,19 +155,21 @@ static void job_txn_add_job(JobTxn *txn, Job *job)
job->txn = txn;
QLIST_INSERT_HEAD(&txn->jobs, job, txn_list);
- job_txn_ref(txn);
+ job_txn_ref_locked(txn);
}
-static void job_txn_del_job(Job *job)
+/* Called with job_mutex held. */
+static void job_txn_del_job_locked(Job *job)
{
if (job->txn) {
QLIST_REMOVE(job, txn_list);
- job_txn_unref(job->txn);
+ job_txn_unref_locked(job->txn);
job->txn = NULL;
}
}
-static int job_txn_apply(Job *job, int fn(Job *))
+/* Called with job_mutex held, but releases it temporarily. */
+static int job_txn_apply_locked(Job *job, int fn(Job *))
{
AioContext *inner_ctx;
Job *other_job, *next;
@@ -170,7 +182,7 @@ static int job_txn_apply(Job *job, int fn(Job *))
* we need to release it here to avoid holding the lock twice - which would
* break AIO_WAIT_WHILE from within fn.
*/
- job_ref(job);
+ job_ref_locked(job);
aio_context_release(job->aio_context);
QLIST_FOREACH_SAFE(other_job, &txn->jobs, txn_list, next) {
@@ -188,7 +200,7 @@ static int job_txn_apply(Job *job, int fn(Job *))
* can't use a local variable to cache it.
*/
aio_context_acquire(job->aio_context);
- job_unref(job);
+ job_unref_locked(job);
return rc;
}
@@ -197,7 +209,8 @@ bool job_is_internal(Job *job)
return (job->id == NULL);
}
-static void job_state_transition(Job *job, JobStatus s1)
+/* Called with job_mutex held. */
+static void job_state_transition_locked(Job *job, JobStatus s1)
{
JobStatus s0 = job->status;
assert(s1 >= 0 && s1 < JOB_STATUS__MAX);
@@ -212,7 +225,7 @@ static void job_state_transition(Job *job, JobStatus s1)
}
}
-int job_apply_verb(Job *job, JobVerb verb, Error **errp)
+int job_apply_verb_locked(Job *job, JobVerb verb, Error **errp)
{
JobStatus s0 = job->status;
assert(verb >= 0 && verb < JOB_VERB__MAX);
@@ -226,6 +239,12 @@ int job_apply_verb(Job *job, JobVerb verb, Error **errp)
return -EPERM;
}
+int job_apply_verb(Job *job, JobVerb verb, Error **errp)
+{
+ JOB_LOCK_GUARD();
+ return job_apply_verb_locked(job, verb, errp);
+}
+
JobType job_type(const Job *job)
{
return job->driver->job_type;
@@ -236,19 +255,32 @@ const char *job_type_str(const Job *job)
return JobType_str(job_type(job));
}
-bool job_is_cancelled(Job *job)
+bool job_is_cancelled_locked(Job *job)
{
/* force_cancel may be true only if cancelled is true, too */
assert(job->cancelled || !job->force_cancel);
return job->force_cancel;
}
-bool job_cancel_requested(Job *job)
+bool job_is_cancelled(Job *job)
+{
+ JOB_LOCK_GUARD();
+ return job_is_cancelled_locked(job);
+}
+
+/* Called with job_mutex held. */
+static bool job_cancel_requested_locked(Job *job)
{
return job->cancelled;
}
-bool job_is_ready(Job *job)
+bool job_cancel_requested(Job *job)
+{
+ JOB_LOCK_GUARD();
+ return job_cancel_requested_locked(job);
+}
+
+bool job_is_ready_locked(Job *job)
{
switch (job->status) {
case JOB_STATUS_UNDEFINED:
@@ -270,7 +302,13 @@ bool job_is_ready(Job *job)
return false;
}
-bool job_is_completed(Job *job)
+bool job_is_ready(Job *job)
+{
+ JOB_LOCK_GUARD();
+ return job_is_ready_locked(job);
+}
+
+bool job_is_completed_locked(Job *job)
{
switch (job->status) {
case JOB_STATUS_UNDEFINED:
@@ -292,17 +330,24 @@ bool job_is_completed(Job *job)
return false;
}
-static bool job_started(Job *job)
+bool job_is_completed(Job *job)
+{
+ JOB_LOCK_GUARD();
+ return job_is_completed_locked(job);
+}
+
+static bool job_started_locked(Job *job)
{
return job->co;
}
-static bool job_should_pause(Job *job)
+/* Called with job_mutex held. */
+static bool job_should_pause_locked(Job *job)
{
return job->pause_count > 0;
}
-Job *job_next(Job *job)
+Job *job_next_locked(Job *job)
{
if (!job) {
return QLIST_FIRST(&jobs);
@@ -310,7 +355,13 @@ Job *job_next(Job *job)
return QLIST_NEXT(job, job_list);
}
-Job *job_get(const char *id)
+Job *job_next(Job *job)
+{
+ JOB_LOCK_GUARD();
+ return job_next_locked(job);
+}
+
+Job *job_get_locked(const char *id)
{
Job *job;
@@ -323,6 +374,13 @@ Job *job_get(const char *id)
return NULL;
}
+Job *job_get(const char *id)
+{
+ JOB_LOCK_GUARD();
+ return job_get_locked(id);
+}
+
+/* Called with job_mutex *not* held. */
static void job_sleep_timer_cb(void *opaque)
{
Job *job = opaque;
@@ -336,6 +394,8 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
{
Job *job;
+ JOB_LOCK_GUARD();
+
if (job_id) {
if (flags & JOB_INTERNAL) {
error_setg(errp, "Cannot specify job ID for internal job");
@@ -345,7 +405,7 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
error_setg(errp, "Invalid job ID '%s'", job_id);
return NULL;
}
- if (job_get(job_id)) {
+ if (job_get_locked(job_id)) {
error_setg(errp, "Job ID '%s' already in use", job_id);
return NULL;
}
@@ -375,7 +435,7 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
notifier_list_init(&job->on_ready);
notifier_list_init(&job->on_idle);
- job_state_transition(job, JOB_STATUS_CREATED);
+ job_state_transition_locked(job, JOB_STATUS_CREATED);
aio_timer_init(qemu_get_aio_context(), &job->sleep_timer,
QEMU_CLOCK_REALTIME, SCALE_NS,
job_sleep_timer_cb, job);
@@ -386,21 +446,27 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
* consolidating the job management logic */
if (!txn) {
txn = job_txn_new();
- job_txn_add_job(txn, job);
- job_txn_unref(txn);
+ job_txn_add_job_locked(txn, job);
+ job_txn_unref_locked(txn);
} else {
- job_txn_add_job(txn, job);
+ job_txn_add_job_locked(txn, job);
}
return job;
}
-void job_ref(Job *job)
+void job_ref_locked(Job *job)
{
++job->refcnt;
}
-void job_unref(Job *job)
+void job_ref(Job *job)
+{
+ JOB_LOCK_GUARD();
+ job_ref_locked(job);
+}
+
+void job_unref_locked(Job *job)
{
GLOBAL_STATE_CODE();
@@ -410,7 +476,9 @@ void job_unref(Job *job)
assert(!job->txn);
if (job->driver->free) {
+ job_unlock();
job->driver->free(job);
+ job_lock();
}
QLIST_REMOVE(job, job_list);
@@ -422,6 +490,12 @@ void job_unref(Job *job)
}
}
+void job_unref(Job *job)
+{
+ JOB_LOCK_GUARD();
+ job_unref_locked(job);
+}
+
void job_progress_update(Job *job, uint64_t done)
{
progress_work_done(&job->progress, done);
@@ -439,38 +513,43 @@ void job_progress_increase_remaining(Job *job, uint64_t delta)
/**
* To be called when a cancelled job is finalised.
+ * Called with job_mutex held.
*/
-static void job_event_cancelled(Job *job)
+static void job_event_cancelled_locked(Job *job)
{
notifier_list_notify(&job->on_finalize_cancelled, job);
}
/**
* To be called when a successfully completed job is finalised.
+ * Called with job_mutex held.
*/
-static void job_event_completed(Job *job)
+static void job_event_completed_locked(Job *job)
{
notifier_list_notify(&job->on_finalize_completed, job);
}
-static void job_event_pending(Job *job)
+/* Called with job_mutex held. */
+static void job_event_pending_locked(Job *job)
{
notifier_list_notify(&job->on_pending, job);
}
-static void job_event_ready(Job *job)
+/* Called with job_mutex held. */
+static void job_event_ready_locked(Job *job)
{
notifier_list_notify(&job->on_ready, job);
}
-static void job_event_idle(Job *job)
+/* Called with job_mutex held. */
+static void job_event_idle_locked(Job *job)
{
notifier_list_notify(&job->on_idle, job);
}
-void job_enter_cond(Job *job, bool(*fn)(Job *job))
+void job_enter_cond_locked(Job *job, bool(*fn)(Job *job))
{
- if (!job_started(job)) {
+ if (!job_started_locked(job)) {
return;
}
if (job->deferred_to_main_loop) {
@@ -495,9 +574,16 @@ void job_enter_cond(Job *job, bool(*fn)(Job *job))
aio_co_enter(job->aio_context, job->co);
}
+void job_enter_cond(Job *job, bool(*fn)(Job *job))
+{
+ JOB_LOCK_GUARD();
+ job_enter_cond_locked(job, fn);
+}
+
void job_enter(Job *job)
{
- job_enter_cond(job, NULL);
+ JOB_LOCK_GUARD();
+ job_enter_cond_locked(job, NULL);
}
/* Yield, and schedule a timer to reenter the coroutine after @ns nanoseconds.
@@ -505,100 +591,129 @@ void job_enter(Job *job)
* is allowed and cancels the timer.
*
* If @ns is (uint64_t) -1, no timer is scheduled and job_enter() must be
- * called explicitly. */
-static void coroutine_fn job_do_yield(Job *job, uint64_t ns)
+ * called explicitly.
+ *
+ * Called with job_mutex held, but releases it temporarily.
+ */
+static void coroutine_fn job_do_yield_locked(Job *job, uint64_t ns)
{
real_job_lock();
if (ns != -1) {
timer_mod(&job->sleep_timer, ns);
}
job->busy = false;
- job_event_idle(job);
+ job_event_idle_locked(job);
real_job_unlock();
+ job_unlock();
qemu_coroutine_yield();
+ job_lock();
/* Set by job_enter_cond() before re-entering the coroutine. */
assert(job->busy);
}
-void coroutine_fn job_pause_point(Job *job)
+/* Called with job_mutex held, but releases it temporarily. */
+static void coroutine_fn job_pause_point_locked(Job *job)
{
- assert(job && job_started(job));
+ assert(job && job_started_locked(job));
- if (!job_should_pause(job)) {
+ if (!job_should_pause_locked(job)) {
return;
}
- if (job_is_cancelled(job)) {
+ if (job_is_cancelled_locked(job)) {
return;
}
if (job->driver->pause) {
+ job_unlock();
job->driver->pause(job);
+ job_lock();
}
- if (job_should_pause(job) && !job_is_cancelled(job)) {
+ if (job_should_pause_locked(job) && !job_is_cancelled_locked(job)) {
JobStatus status = job->status;
- job_state_transition(job, status == JOB_STATUS_READY
- ? JOB_STATUS_STANDBY
- : JOB_STATUS_PAUSED);
+ job_state_transition_locked(job, status == JOB_STATUS_READY
+ ? JOB_STATUS_STANDBY
+ : JOB_STATUS_PAUSED);
job->paused = true;
- job_do_yield(job, -1);
+ job_do_yield_locked(job, -1);
job->paused = false;
- job_state_transition(job, status);
+ job_state_transition_locked(job, status);
}
if (job->driver->resume) {
+ job_unlock();
job->driver->resume(job);
+ job_lock();
}
}
-void job_yield(Job *job)
+void coroutine_fn job_pause_point(Job *job)
+{
+ JOB_LOCK_GUARD();
+ job_pause_point_locked(job);
+}
+
+void job_yield_locked(Job *job)
{
assert(job->busy);
/* Check cancellation *before* setting busy = false, too! */
- if (job_is_cancelled(job)) {
+ if (job_is_cancelled_locked(job)) {
return;
}
- if (!job_should_pause(job)) {
- job_do_yield(job, -1);
+ if (!job_should_pause_locked(job)) {
+ job_do_yield_locked(job, -1);
}
- job_pause_point(job);
+ job_pause_point_locked(job);
+}
+
+void job_yield(Job *job)
+{
+ JOB_LOCK_GUARD();
+ job_yield_locked(job);
}
void coroutine_fn job_sleep_ns(Job *job, int64_t ns)
{
+ JOB_LOCK_GUARD();
assert(job->busy);
/* Check cancellation *before* setting busy = false, too! */
- if (job_is_cancelled(job)) {
+ if (job_is_cancelled_locked(job)) {
return;
}
- if (!job_should_pause(job)) {
- job_do_yield(job, qemu_clock_get_ns(QEMU_CLOCK_REALTIME) + ns);
+ if (!job_should_pause_locked(job)) {
+ job_do_yield_locked(job, qemu_clock_get_ns(QEMU_CLOCK_REALTIME) + ns);
}
- job_pause_point(job);
+ job_pause_point_locked(job);
}
-/* Assumes the block_job_mutex is held */
-static bool job_timer_not_pending(Job *job)
+/* Assumes the job_mutex is held */
+static bool job_timer_not_pending_locked(Job *job)
{
return !timer_pending(&job->sleep_timer);
}
-void job_pause(Job *job)
+void job_pause_locked(Job *job)
{
job->pause_count++;
if (!job->paused) {
- job_enter(job);
+ job_enter_cond_locked(job, NULL);
}
}
-void job_resume(Job *job)
+void job_pause(Job *job)
+{
+ JOB_LOCK_GUARD();
+ job_pause_locked(job);
+}
+
+void job_resume_locked(Job *job)
{
assert(job->pause_count > 0);
job->pause_count--;
@@ -607,12 +722,18 @@ void job_resume(Job *job)
}
/* kick only if no timer is pending */
- job_enter_cond(job, job_timer_not_pending);
+ job_enter_cond_locked(job, job_timer_not_pending_locked);
}
-void job_user_pause(Job *job, Error **errp)
+void job_resume(Job *job)
{
- if (job_apply_verb(job, JOB_VERB_PAUSE, errp)) {
+ JOB_LOCK_GUARD();
+ job_resume_locked(job);
+}
+
+void job_user_pause_locked(Job *job, Error **errp)
+{
+ if (job_apply_verb_locked(job, JOB_VERB_PAUSE, errp)) {
return;
}
if (job->user_paused) {
@@ -620,15 +741,27 @@ void job_user_pause(Job *job, Error **errp)
return;
}
job->user_paused = true;
- job_pause(job);
+ job_pause_locked(job);
}
-bool job_user_paused(Job *job)
+void job_user_pause(Job *job, Error **errp)
+{
+ JOB_LOCK_GUARD();
+ job_user_pause_locked(job, errp);
+}
+
+bool job_user_paused_locked(Job *job)
{
return job->user_paused;
}
-void job_user_resume(Job *job, Error **errp)
+bool job_user_paused(Job *job)
+{
+ JOB_LOCK_GUARD();
+ return job_user_paused_locked(job);
+}
+
+void job_user_resume_locked(Job *job, Error **errp)
{
assert(job);
GLOBAL_STATE_CODE();
@@ -636,66 +769,84 @@ void job_user_resume(Job *job, Error **errp)
error_setg(errp, "Can't resume a job that was not paused");
return;
}
- if (job_apply_verb(job, JOB_VERB_RESUME, errp)) {
+ if (job_apply_verb_locked(job, JOB_VERB_RESUME, errp)) {
return;
}
if (job->driver->user_resume) {
+ job_unlock();
job->driver->user_resume(job);
+ job_lock();
}
job->user_paused = false;
- job_resume(job);
+ job_resume_locked(job);
}
-static void job_do_dismiss(Job *job)
+void job_user_resume(Job *job, Error **errp)
+{
+ JOB_LOCK_GUARD();
+ job_user_resume_locked(job, errp);
+}
+
+/* Called with job_mutex held, but releases it temporarily. */
+static void job_do_dismiss_locked(Job *job)
{
assert(job);
job->busy = false;
job->paused = false;
job->deferred_to_main_loop = true;
- job_txn_del_job(job);
+ job_txn_del_job_locked(job);
- job_state_transition(job, JOB_STATUS_NULL);
- job_unref(job);
+ job_state_transition_locked(job, JOB_STATUS_NULL);
+ job_unref_locked(job);
}
-void job_dismiss(Job **jobptr, Error **errp)
+void job_dismiss_locked(Job **jobptr, Error **errp)
{
Job *job = *jobptr;
/* similarly to _complete, this is QMP-interface only. */
assert(job->id);
- if (job_apply_verb(job, JOB_VERB_DISMISS, errp)) {
+ if (job_apply_verb_locked(job, JOB_VERB_DISMISS, errp)) {
return;
}
- job_do_dismiss(job);
+ job_do_dismiss_locked(job);
*jobptr = NULL;
}
+void job_dismiss(Job **jobptr, Error **errp)
+{
+ JOB_LOCK_GUARD();
+ job_dismiss_locked(jobptr, errp);
+}
+
void job_early_fail(Job *job)
{
+ JOB_LOCK_GUARD();
assert(job->status == JOB_STATUS_CREATED);
- job_do_dismiss(job);
+ job_do_dismiss_locked(job);
}
-static void job_conclude(Job *job)
+/* Called with job_mutex held. */
+static void job_conclude_locked(Job *job)
{
- job_state_transition(job, JOB_STATUS_CONCLUDED);
- if (job->auto_dismiss || !job_started(job)) {
- job_do_dismiss(job);
+ job_state_transition_locked(job, JOB_STATUS_CONCLUDED);
+ if (job->auto_dismiss || !job_started_locked(job)) {
+ job_do_dismiss_locked(job);
}
}
-static void job_update_rc(Job *job)
+/* Called with job_mutex held. */
+static void job_update_rc_locked(Job *job)
{
- if (!job->ret && job_is_cancelled(job)) {
+ if (!job->ret && job_is_cancelled_locked(job)) {
job->ret = -ECANCELED;
}
if (job->ret) {
if (!job->err) {
error_setg(&job->err, "%s", strerror(-job->ret));
}
- job_state_transition(job, JOB_STATUS_ABORTING);
+ job_state_transition_locked(job, JOB_STATUS_ABORTING);
}
}
@@ -725,12 +876,17 @@ static void job_clean(Job *job)
}
}
-static int job_finalize_single(Job *job)
+/* Called with job_mutex held, but releases it temporarily */
+static int job_finalize_single_locked(Job *job)
{
- assert(job_is_completed(job));
+ int job_ret;
+
+ assert(job_is_completed_locked(job));
/* Ensure abort is called for late-transactional failures */
- job_update_rc(job);
+ job_update_rc_locked(job);
+
+ job_unlock();
if (!job->ret) {
job_commit(job);
@@ -739,29 +895,37 @@ static int job_finalize_single(Job *job)
}
job_clean(job);
+ job_lock();
+
if (job->cb) {
- job->cb(job->opaque, job->ret);
+ job_ret = job->ret;
+ job_unlock();
+ job->cb(job->opaque, job_ret);
+ job_lock();
}
/* Emit events only if we actually started */
- if (job_started(job)) {
- if (job_is_cancelled(job)) {
- job_event_cancelled(job);
+ if (job_started_locked(job)) {
+ if (job_is_cancelled_locked(job)) {
+ job_event_cancelled_locked(job);
} else {
- job_event_completed(job);
+ job_event_completed_locked(job);
}
}
- job_txn_del_job(job);
- job_conclude(job);
+ job_txn_del_job_locked(job);
+ job_conclude_locked(job);
return 0;
}
-static void job_cancel_async(Job *job, bool force)
+/* Called with job_mutex held, but releases it temporarily */
+static void job_cancel_async_locked(Job *job, bool force)
{
GLOBAL_STATE_CODE();
if (job->driver->cancel) {
+ job_unlock();
force = job->driver->cancel(job, force);
+ job_lock();
} else {
/* No .cancel() means the job will behave as if force-cancelled */
force = true;
@@ -770,7 +934,9 @@ static void job_cancel_async(Job *job, bool force)
if (job->user_paused) {
/* Do not call job_enter here, the caller will handle it. */
if (job->driver->user_resume) {
+ job_unlock();
job->driver->user_resume(job);
+ job_lock();
}
job->user_paused = false;
assert(job->pause_count > 0);
@@ -791,7 +957,8 @@ static void job_cancel_async(Job *job, bool force)
}
}
-static void job_completed_txn_abort(Job *job)
+/* Called with job_mutex held, but releases it temporarily. */
+static void job_completed_txn_abort_locked(Job *job)
{
AioContext *ctx;
JobTxn *txn = job->txn;
@@ -804,7 +971,7 @@ static void job_completed_txn_abort(Job *job)
return;
}
txn->aborting = true;
- job_txn_ref(txn);
+ job_txn_ref_locked(txn);
/*
* We can only hold the single job's AioContext lock while calling
@@ -812,7 +979,7 @@ static void job_completed_txn_abort(Job *job)
* calls of AIO_WAIT_WHILE(), which could deadlock otherwise.
* Note that the job's AioContext may change when it is finalized.
*/
- job_ref(job);
+ job_ref_locked(job);
aio_context_release(job->aio_context);
/* Other jobs are effectively cancelled by us, set the status for
@@ -827,7 +994,7 @@ static void job_completed_txn_abort(Job *job)
* Therefore, pass force=true to terminate all other jobs as quickly
* as possible.
*/
- job_cancel_async(other_job, true);
+ job_cancel_async_locked(other_job, true);
aio_context_release(ctx);
}
}
@@ -839,11 +1006,11 @@ static void job_completed_txn_abort(Job *job)
*/
ctx = other_job->aio_context;
aio_context_acquire(ctx);
- if (!job_is_completed(other_job)) {
- assert(job_cancel_requested(other_job));
- job_finish_sync(other_job, NULL, NULL);
+ if (!job_is_completed_locked(other_job)) {
+ assert(job_cancel_requested_locked(other_job));
+ job_finish_sync_locked(other_job, NULL, NULL);
}
- job_finalize_single(other_job);
+ job_finalize_single_locked(other_job);
aio_context_release(ctx);
}
@@ -852,110 +1019,129 @@ static void job_completed_txn_abort(Job *job)
* even if the job went away during job_finalize_single().
*/
aio_context_acquire(job->aio_context);
- job_unref(job);
+ job_unref_locked(job);
- job_txn_unref(txn);
+ job_txn_unref_locked(txn);
}
-static int job_prepare(Job *job)
+/* Called with job_mutex held, but releases it temporarily */
+static int job_prepare_locked(Job *job)
{
GLOBAL_STATE_CODE();
if (job->ret == 0 && job->driver->prepare) {
+ job_unlock();
job->ret = job->driver->prepare(job);
- job_update_rc(job);
+ job_lock();
+ job_update_rc_locked(job);
}
return job->ret;
}
-static int job_needs_finalize(Job *job)
+/* Called with job_mutex held */
+static int job_needs_finalize_locked(Job *job)
{
return !job->auto_finalize;
}
-static void job_do_finalize(Job *job)
+/* Called with job_mutex held */
+static void job_do_finalize_locked(Job *job)
{
int rc;
assert(job && job->txn);
/* prepare the transaction to complete */
- rc = job_txn_apply(job, job_prepare);
+ rc = job_txn_apply_locked(job, job_prepare_locked);
if (rc) {
- job_completed_txn_abort(job);
+ job_completed_txn_abort_locked(job);
} else {
- job_txn_apply(job, job_finalize_single);
+ job_txn_apply_locked(job, job_finalize_single_locked);
}
}
-void job_finalize(Job *job, Error **errp)
+void job_finalize_locked(Job *job, Error **errp)
{
assert(job && job->id);
- if (job_apply_verb(job, JOB_VERB_FINALIZE, errp)) {
+ if (job_apply_verb_locked(job, JOB_VERB_FINALIZE, errp)) {
return;
}
- job_do_finalize(job);
+ job_do_finalize_locked(job);
}
-static int job_transition_to_pending(Job *job)
+void job_finalize(Job *job, Error **errp)
{
- job_state_transition(job, JOB_STATUS_PENDING);
+ JOB_LOCK_GUARD();
+ job_finalize_locked(job, errp);
+}
+
+/* Called with job_mutex held. */
+static int job_transition_to_pending_locked(Job *job)
+{
+ job_state_transition_locked(job, JOB_STATUS_PENDING);
if (!job->auto_finalize) {
- job_event_pending(job);
+ job_event_pending_locked(job);
}
return 0;
}
void job_transition_to_ready(Job *job)
{
- job_state_transition(job, JOB_STATUS_READY);
- job_event_ready(job);
+ JOB_LOCK_GUARD();
+ job_state_transition_locked(job, JOB_STATUS_READY);
+ job_event_ready_locked(job);
}
-static void job_completed_txn_success(Job *job)
+/* Called with job_mutex held. */
+static void job_completed_txn_success_locked(Job *job)
{
JobTxn *txn = job->txn;
Job *other_job;
- job_state_transition(job, JOB_STATUS_WAITING);
+ job_state_transition_locked(job, JOB_STATUS_WAITING);
/*
* Successful completion, see if there are other running jobs in this
* txn.
*/
QLIST_FOREACH(other_job, &txn->jobs, txn_list) {
- if (!job_is_completed(other_job)) {
+ if (!job_is_completed_locked(other_job)) {
return;
}
assert(other_job->ret == 0);
}
- job_txn_apply(job, job_transition_to_pending);
+ job_txn_apply_locked(job, job_transition_to_pending_locked);
/* If no jobs need manual finalization, automatically do so */
- if (job_txn_apply(job, job_needs_finalize) == 0) {
- job_do_finalize(job);
+ if (job_txn_apply_locked(job, job_needs_finalize_locked) == 0) {
+ job_do_finalize_locked(job);
}
}
-static void job_completed(Job *job)
+/* Called with job_mutex held. */
+static void job_completed_locked(Job *job)
{
- assert(job && job->txn && !job_is_completed(job));
+ assert(job && job->txn && !job_is_completed_locked(job));
- job_update_rc(job);
+ job_update_rc_locked(job);
trace_job_completed(job, job->ret);
if (job->ret) {
- job_completed_txn_abort(job);
+ job_completed_txn_abort_locked(job);
} else {
- job_completed_txn_success(job);
+ job_completed_txn_success_locked(job);
}
}
-/** Useful only as a type shim for aio_bh_schedule_oneshot. */
+/**
+ * Useful only as a type shim for aio_bh_schedule_oneshot.
+ * Called with job_mutex *not* held.
+ */
static void job_exit(void *opaque)
{
Job *job = (Job *)opaque;
AioContext *ctx;
+ JOB_LOCK_GUARD();
- job_ref(job);
+ job_ref_locked(job);
aio_context_acquire(job->aio_context);
/* This is a lie, we're not quiescent, but still doing the completion
@@ -963,9 +1149,9 @@ static void job_exit(void *opaque)
* drain block nodes, and if .drained_poll still returned true, we would
* deadlock. */
job->busy = false;
- job_event_idle(job);
+ job_event_idle_locked(job);
- job_completed(job);
+ job_completed_locked(job);
/*
* Note that calling job_completed can move the job to a different
@@ -974,7 +1160,7 @@ static void job_exit(void *opaque)
* the job underneath us.
*/
ctx = job->aio_context;
- job_unref(job);
+ job_unref_locked(job);
aio_context_release(ctx);
}
@@ -985,37 +1171,47 @@ static void job_exit(void *opaque)
static void coroutine_fn job_co_entry(void *opaque)
{
Job *job = opaque;
+ int ret;
assert(job && job->driver && job->driver->run);
- assert(job->aio_context == qemu_get_current_aio_context());
- job_pause_point(job);
- job->ret = job->driver->run(job, &job->err);
- job->deferred_to_main_loop = true;
- job->busy = true;
+ WITH_JOB_LOCK_GUARD() {
+ assert(job->aio_context == qemu_get_current_aio_context());
+ job_pause_point_locked(job);
+ }
+ ret = job->driver->run(job, &job->err);
+ WITH_JOB_LOCK_GUARD() {
+ job->ret = ret;
+ job->deferred_to_main_loop = true;
+ job->busy = true;
+ }
aio_bh_schedule_oneshot(qemu_get_aio_context(), job_exit, job);
}
void job_start(Job *job)
{
- assert(job && !job_started(job) && job->paused &&
- job->driver && job->driver->run);
- job->co = qemu_coroutine_create(job_co_entry, job);
- job->pause_count--;
- job->busy = true;
- job->paused = false;
- job_state_transition(job, JOB_STATUS_RUNNING);
+ assert(qemu_in_main_thread());
+
+ WITH_JOB_LOCK_GUARD() {
+ assert(job && !job_started_locked(job) && job->paused &&
+ job->driver && job->driver->run);
+ job->co = qemu_coroutine_create(job_co_entry, job);
+ job->pause_count--;
+ job->busy = true;
+ job->paused = false;
+ job_state_transition_locked(job, JOB_STATUS_RUNNING);
+ }
aio_co_enter(job->aio_context, job->co);
}
-void job_cancel(Job *job, bool force)
+void job_cancel_locked(Job *job, bool force)
{
if (job->status == JOB_STATUS_CONCLUDED) {
- job_do_dismiss(job);
+ job_do_dismiss_locked(job);
return;
}
- job_cancel_async(job, force);
- if (!job_started(job)) {
- job_completed(job);
+ job_cancel_async_locked(job, force);
+ if (!job_started_locked(job)) {
+ job_completed_locked(job);
} else if (job->deferred_to_main_loop) {
/*
* job_cancel_async() ignores soft-cancel requests for jobs
@@ -1027,102 +1223,150 @@ void job_cancel(Job *job, bool force)
* choose to call job_is_cancelled() to show that we invoke
* job_completed_txn_abort() only for force-cancelled jobs.)
*/
- if (job_is_cancelled(job)) {
- job_completed_txn_abort(job);
+ if (job_is_cancelled_locked(job)) {
+ job_completed_txn_abort_locked(job);
}
} else {
- job_enter(job);
+ job_enter_cond_locked(job, NULL);
}
}
-void job_user_cancel(Job *job, bool force, Error **errp)
+void job_cancel(Job *job, bool force)
{
- if (job_apply_verb(job, JOB_VERB_CANCEL, errp)) {
+ JOB_LOCK_GUARD();
+ job_cancel_locked(job, force);
+}
+
+void job_user_cancel_locked(Job *job, bool force, Error **errp)
+{
+ if (job_apply_verb_locked(job, JOB_VERB_CANCEL, errp)) {
return;
}
- job_cancel(job, force);
+ job_cancel_locked(job, force);
+}
+
+void job_user_cancel(Job *job, bool force, Error **errp)
+{
+ JOB_LOCK_GUARD();
+ job_user_cancel_locked(job, force, errp);
}
/* A wrapper around job_cancel() taking an Error ** parameter so it may be
* used with job_finish_sync() without the need for (rather nasty) function
- * pointer casts there. */
-static void job_cancel_err(Job *job, Error **errp)
+ * pointer casts there.
+ *
+ * Called with job_mutex held.
+ */
+static void job_cancel_err_locked(Job *job, Error **errp)
{
- job_cancel(job, false);
+ job_cancel_locked(job, false);
}
/**
* Same as job_cancel_err(), but force-cancel.
+ * Called with job_mutex held.
*/
-static void job_force_cancel_err(Job *job, Error **errp)
+static void job_force_cancel_err_locked(Job *job, Error **errp)
{
- job_cancel(job, true);
+ job_cancel_locked(job, true);
}
-int job_cancel_sync(Job *job, bool force)
+int job_cancel_sync_locked(Job *job, bool force)
{
if (force) {
- return job_finish_sync(job, &job_force_cancel_err, NULL);
+ return job_finish_sync_locked(job, &job_force_cancel_err_locked, NULL);
} else {
- return job_finish_sync(job, &job_cancel_err, NULL);
+ return job_finish_sync_locked(job, &job_cancel_err_locked, NULL);
}
}
+int job_cancel_sync(Job *job, bool force)
+{
+ JOB_LOCK_GUARD();
+ return job_cancel_sync_locked(job, force);
+}
+
void job_cancel_sync_all(void)
{
Job *job;
AioContext *aio_context;
+ JOB_LOCK_GUARD();
- while ((job = job_next(NULL))) {
+ while ((job = job_next_locked(NULL))) {
aio_context = job->aio_context;
aio_context_acquire(aio_context);
- job_cancel_sync(job, true);
+ job_cancel_sync_locked(job, true);
aio_context_release(aio_context);
}
}
+int job_complete_sync_locked(Job *job, Error **errp)
+{
+ return job_finish_sync_locked(job, job_complete_locked, errp);
+}
+
int job_complete_sync(Job *job, Error **errp)
{
- return job_finish_sync(job, job_complete, errp);
+ JOB_LOCK_GUARD();
+ return job_complete_sync_locked(job, errp);
}
-void job_complete(Job *job, Error **errp)
+void job_complete_locked(Job *job, Error **errp)
{
/* Should not be reachable via external interface for internal jobs */
assert(job->id);
GLOBAL_STATE_CODE();
- if (job_apply_verb(job, JOB_VERB_COMPLETE, errp)) {
+ if (job_apply_verb_locked(job, JOB_VERB_COMPLETE, errp)) {
return;
}
- if (job_cancel_requested(job) || !job->driver->complete) {
+ if (job_cancel_requested_locked(job) || !job->driver->complete) {
error_setg(errp, "The active block job '%s' cannot be completed",
job->id);
return;
}
+ job_unlock();
job->driver->complete(job, errp);
+ job_lock();
}
-int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp)
+void job_complete(Job *job, Error **errp)
+{
+ JOB_LOCK_GUARD();
+ job_complete_locked(job, errp);
+}
+
+int job_finish_sync_locked(Job *job,
+ void (*finish)(Job *, Error **errp),
+ Error **errp)
{
Error *local_err = NULL;
int ret;
- job_ref(job);
+ job_ref_locked(job);
if (finish) {
finish(job, &local_err);
}
if (local_err) {
error_propagate(errp, local_err);
- job_unref(job);
+ job_unref_locked(job);
return -EBUSY;
}
+ job_unlock();
AIO_WAIT_WHILE(job->aio_context,
(job_enter(job), !job_is_completed(job)));
+ job_lock();
- ret = (job_is_cancelled(job) && job->ret == 0) ? -ECANCELED : job->ret;
- job_unref(job);
+ ret = (job_is_cancelled_locked(job) && job->ret == 0)
+ ? -ECANCELED : job->ret;
+ job_unref_locked(job);
return ret;
}
+
+int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp)
+{
+ JOB_LOCK_GUARD();
+ return job_finish_sync_locked(job, finish, errp);
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 06/21] job: move and update comments from blockjob.c
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (4 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-03 15:47 ` Kevin Wolf
2022-08-16 18:32 ` Stefan Hajnoczi
2022-07-25 7:38 ` [PATCH v10 07/21] blockjob: introduce block_job _locked() APIs Emanuele Giuseppe Esposito
` (14 subsequent siblings)
20 siblings, 2 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
This comment applies more on job, it was left in blockjob as in the past
the whole job logic was implemented there.
Note: at this stage, job_{lock/unlock} and job lock guard macros
are *nop*.
No functional change intended.
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
blockjob.c | 20 --------------------
job.c | 14 ++++++++++++++
2 files changed, 14 insertions(+), 20 deletions(-)
diff --git a/blockjob.c b/blockjob.c
index 4868453d74..7da59a1f1c 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -36,21 +36,6 @@
#include "qemu/main-loop.h"
#include "qemu/timer.h"
-/*
- * The block job API is composed of two categories of functions.
- *
- * The first includes functions used by the monitor. The monitor is
- * peculiar in that it accesses the block job list with block_job_get, and
- * therefore needs consistency across block_job_get and the actual operation
- * (e.g. block_job_set_speed). The consistency is achieved with
- * aio_context_acquire/release. These functions are declared in blockjob.h.
- *
- * The second includes functions used by the block job drivers and sometimes
- * by the core block layer. These do not care about locking, because the
- * whole coroutine runs under the AioContext lock, and are declared in
- * blockjob_int.h.
- */
-
static bool is_block_job(Job *job)
{
return job_type(job) == JOB_TYPE_BACKUP ||
@@ -433,11 +418,6 @@ static void block_job_event_ready(Notifier *n, void *opaque)
}
-/*
- * API for block job drivers and the block layer. These functions are
- * declared in blockjob_int.h.
- */
-
void *block_job_create(const char *job_id, const BlockJobDriver *driver,
JobTxn *txn, BlockDriverState *bs, uint64_t perm,
uint64_t shared_perm, int64_t speed, int flags,
diff --git a/job.c b/job.c
index ae25db97ac..ebaa4e585b 100644
--- a/job.c
+++ b/job.c
@@ -32,6 +32,20 @@
#include "trace/trace-root.h"
#include "qapi/qapi-events-job.h"
+/*
+ * The job API is composed of two categories of functions.
+ *
+ * The first includes functions used by the monitor. The monitor is
+ * peculiar in that it accesses the block job list with job_get, and
+ * therefore needs consistency across job_get and the actual operation
+ * (e.g. job_user_cancel). To achieve this consistency, the caller
+ * calls job_lock/job_unlock itself around the whole operation.
+ *
+ *
+ * The second includes functions used by the block job drivers and sometimes
+ * by the core block layer. These delegate the locking to the callee instead.
+ */
+
/*
* job_mutex protects the jobs list, but also makes the
* struct job fields thread-safe.
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 07/21] blockjob: introduce block_job _locked() APIs
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (5 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 06/21] job: move and update comments from blockjob.c Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-03 15:52 ` Kevin Wolf
2022-08-16 18:33 ` Stefan Hajnoczi
2022-07-25 7:38 ` [PATCH v10 08/21] jobs: add job lock in find_* functions Emanuele Giuseppe Esposito
` (13 subsequent siblings)
20 siblings, 2 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
Just as done with job.h, create _locked() functions in blockjob.h
These functions will be later useful when caller has already taken
the lock. All blockjob _locked functions call job _locked functions.
Note: at this stage, job_{lock/unlock} and job lock guard macros
are *nop*.
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
blockjob.c | 52 ++++++++++++++++++++++++++++++++--------
include/block/blockjob.h | 18 ++++++++++++++
2 files changed, 60 insertions(+), 10 deletions(-)
diff --git a/blockjob.c b/blockjob.c
index 7da59a1f1c..0d59aba439 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -44,21 +44,27 @@ static bool is_block_job(Job *job)
job_type(job) == JOB_TYPE_STREAM;
}
-BlockJob *block_job_next(BlockJob *bjob)
+BlockJob *block_job_next_locked(BlockJob *bjob)
{
Job *job = bjob ? &bjob->job : NULL;
GLOBAL_STATE_CODE();
do {
- job = job_next(job);
+ job = job_next_locked(job);
} while (job && !is_block_job(job));
return job ? container_of(job, BlockJob, job) : NULL;
}
-BlockJob *block_job_get(const char *id)
+BlockJob *block_job_next(BlockJob *bjob)
{
- Job *job = job_get(id);
+ JOB_LOCK_GUARD();
+ return block_job_next_locked(bjob);
+}
+
+BlockJob *block_job_get_locked(const char *id)
+{
+ Job *job = job_get_locked(id);
GLOBAL_STATE_CODE();
if (job && is_block_job(job)) {
@@ -68,6 +74,12 @@ BlockJob *block_job_get(const char *id)
}
}
+BlockJob *block_job_get(const char *id)
+{
+ JOB_LOCK_GUARD();
+ return block_job_get_locked(id);
+}
+
void block_job_free(Job *job)
{
BlockJob *bjob = container_of(job, BlockJob, job);
@@ -256,14 +268,14 @@ static bool job_timer_pending(Job *job)
return timer_pending(&job->sleep_timer);
}
-bool block_job_set_speed(BlockJob *job, int64_t speed, Error **errp)
+bool block_job_set_speed_locked(BlockJob *job, int64_t speed, Error **errp)
{
const BlockJobDriver *drv = block_job_driver(job);
int64_t old_speed = job->speed;
GLOBAL_STATE_CODE();
- if (job_apply_verb(&job->job, JOB_VERB_SET_SPEED, errp) < 0) {
+ if (job_apply_verb_locked(&job->job, JOB_VERB_SET_SPEED, errp) < 0) {
return false;
}
if (speed < 0) {
@@ -277,7 +289,9 @@ bool block_job_set_speed(BlockJob *job, int64_t speed, Error **errp)
job->speed = speed;
if (drv->set_speed) {
+ job_unlock();
drv->set_speed(job, speed);
+ job_lock();
}
if (speed && speed <= old_speed) {
@@ -285,18 +299,24 @@ bool block_job_set_speed(BlockJob *job, int64_t speed, Error **errp)
}
/* kick only if a timer is pending */
- job_enter_cond(&job->job, job_timer_pending);
+ job_enter_cond_locked(&job->job, job_timer_pending);
return true;
}
+bool block_job_set_speed(BlockJob *job, int64_t speed, Error **errp)
+{
+ JOB_LOCK_GUARD();
+ return block_job_set_speed_locked(job, speed, errp);
+}
+
int64_t block_job_ratelimit_get_delay(BlockJob *job, uint64_t n)
{
IO_CODE();
return ratelimit_calculate_delay(&job->limit, n);
}
-BlockJobInfo *block_job_query(BlockJob *job, Error **errp)
+BlockJobInfo *block_job_query_locked(BlockJob *job, Error **errp)
{
BlockJobInfo *info;
uint64_t progress_current, progress_total;
@@ -320,7 +340,7 @@ BlockJobInfo *block_job_query(BlockJob *job, Error **errp)
info->len = progress_total;
info->speed = job->speed;
info->io_status = job->iostatus;
- info->ready = job_is_ready(&job->job),
+ info->ready = job_is_ready_locked(&job->job),
info->status = job->job.status;
info->auto_finalize = job->job.auto_finalize;
info->auto_dismiss = job->job.auto_dismiss;
@@ -333,6 +353,12 @@ BlockJobInfo *block_job_query(BlockJob *job, Error **errp)
return info;
}
+BlockJobInfo *block_job_query(BlockJob *job, Error **errp)
+{
+ JOB_LOCK_GUARD();
+ return block_job_query_locked(job, errp);
+}
+
static void block_job_iostatus_set_err(BlockJob *job, int error)
{
if (job->iostatus == BLOCK_DEVICE_IO_STATUS_OK) {
@@ -478,7 +504,7 @@ fail:
return NULL;
}
-void block_job_iostatus_reset(BlockJob *job)
+void block_job_iostatus_reset_locked(BlockJob *job)
{
GLOBAL_STATE_CODE();
if (job->iostatus == BLOCK_DEVICE_IO_STATUS_OK) {
@@ -488,6 +514,12 @@ void block_job_iostatus_reset(BlockJob *job)
job->iostatus = BLOCK_DEVICE_IO_STATUS_OK;
}
+void block_job_iostatus_reset(BlockJob *job)
+{
+ JOB_LOCK_GUARD();
+ block_job_iostatus_reset_locked(job);
+}
+
void block_job_user_resume(Job *job)
{
BlockJob *bjob = container_of(job, BlockJob, job);
diff --git a/include/block/blockjob.h b/include/block/blockjob.h
index 6525e16fd5..8b65d3949d 100644
--- a/include/block/blockjob.h
+++ b/include/block/blockjob.h
@@ -92,6 +92,9 @@ typedef struct BlockJob {
*/
BlockJob *block_job_next(BlockJob *job);
+/* Same as block_job_next(), but called with job lock held. */
+BlockJob *block_job_next_locked(BlockJob *job);
+
/**
* block_job_get:
* @id: The id of the block job.
@@ -102,6 +105,9 @@ BlockJob *block_job_next(BlockJob *job);
*/
BlockJob *block_job_get(const char *id);
+/* Same as block_job_get(), but called with job lock held. */
+BlockJob *block_job_get_locked(const char *id);
+
/**
* block_job_add_bdrv:
* @job: A block job
@@ -145,6 +151,12 @@ bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs);
*/
bool block_job_set_speed(BlockJob *job, int64_t speed, Error **errp);
+/*
+ * Same as block_job_set_speed(), but called with job lock held.
+ * Might release the lock temporarily.
+ */
+bool block_job_set_speed_locked(BlockJob *job, int64_t speed, Error **errp);
+
/**
* block_job_query:
* @job: The job to get information about.
@@ -153,6 +165,9 @@ bool block_job_set_speed(BlockJob *job, int64_t speed, Error **errp);
*/
BlockJobInfo *block_job_query(BlockJob *job, Error **errp);
+/* Same as block_job_query(), but called with job lock held. */
+BlockJobInfo *block_job_query_locked(BlockJob *job, Error **errp);
+
/**
* block_job_iostatus_reset:
* @job: The job whose I/O status should be reset.
@@ -162,6 +177,9 @@ BlockJobInfo *block_job_query(BlockJob *job, Error **errp);
*/
void block_job_iostatus_reset(BlockJob *job);
+/* Same as block_job_iostatus_reset(), but called with job lock held. */
+void block_job_iostatus_reset_locked(BlockJob *job);
+
/*
* block_job_get_aio_context:
*
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 08/21] jobs: add job lock in find_* functions
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (6 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 07/21] blockjob: introduce block_job _locked() APIs Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-04 11:47 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 09/21] jobs: use job locks also in the unit tests Emanuele Giuseppe Esposito
` (12 subsequent siblings)
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
Both blockdev.c and job-qmp.c have TOC/TOU conditions, because
they first search for the job and then perform an action on it.
Therefore, we need to do the search + action under the same
job mutex critical section.
Note: at this stage, job_{lock/unlock} and job lock guard macros
are *nop*.
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
blockdev.c | 67 +++++++++++++++++++++++++++++++++++++-----------------
job-qmp.c | 57 ++++++++++++++++++++++++++++++++--------------
2 files changed, 86 insertions(+), 38 deletions(-)
diff --git a/blockdev.c b/blockdev.c
index 9230888e34..71f793c4ab 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3302,9 +3302,13 @@ out:
aio_context_release(aio_context);
}
-/* Get a block job using its ID and acquire its AioContext */
-static BlockJob *find_block_job(const char *id, AioContext **aio_context,
- Error **errp)
+/*
+ * Get a block job using its ID and acquire its AioContext.
+ * Called with job_mutex held.
+ */
+static BlockJob *find_block_job_locked(const char *id,
+ AioContext **aio_context,
+ Error **errp)
{
BlockJob *job;
@@ -3312,7 +3316,7 @@ static BlockJob *find_block_job(const char *id, AioContext **aio_context,
*aio_context = NULL;
- job = block_job_get(id);
+ job = block_job_get_locked(id);
if (!job) {
error_set(errp, ERROR_CLASS_DEVICE_NOT_ACTIVE,
@@ -3329,13 +3333,16 @@ static BlockJob *find_block_job(const char *id, AioContext **aio_context,
void qmp_block_job_set_speed(const char *device, int64_t speed, Error **errp)
{
AioContext *aio_context;
- BlockJob *job = find_block_job(device, &aio_context, errp);
+ BlockJob *job;
+
+ JOB_LOCK_GUARD();
+ job = find_block_job_locked(device, &aio_context, errp);
if (!job) {
return;
}
- block_job_set_speed(job, speed, errp);
+ block_job_set_speed_locked(job, speed, errp);
aio_context_release(aio_context);
}
@@ -3343,7 +3350,10 @@ void qmp_block_job_cancel(const char *device,
bool has_force, bool force, Error **errp)
{
AioContext *aio_context;
- BlockJob *job = find_block_job(device, &aio_context, errp);
+ BlockJob *job;
+
+ JOB_LOCK_GUARD();
+ job = find_block_job_locked(device, &aio_context, errp);
if (!job) {
return;
@@ -3353,14 +3363,14 @@ void qmp_block_job_cancel(const char *device,
force = false;
}
- if (job_user_paused(&job->job) && !force) {
+ if (job_user_paused_locked(&job->job) && !force) {
error_setg(errp, "The block job for device '%s' is currently paused",
device);
goto out;
}
trace_qmp_block_job_cancel(job);
- job_user_cancel(&job->job, force, errp);
+ job_user_cancel_locked(&job->job, force, errp);
out:
aio_context_release(aio_context);
}
@@ -3368,57 +3378,69 @@ out:
void qmp_block_job_pause(const char *device, Error **errp)
{
AioContext *aio_context;
- BlockJob *job = find_block_job(device, &aio_context, errp);
+ BlockJob *job;
+
+ JOB_LOCK_GUARD();
+ job = find_block_job_locked(device, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_block_job_pause(job);
- job_user_pause(&job->job, errp);
+ job_user_pause_locked(&job->job, errp);
aio_context_release(aio_context);
}
void qmp_block_job_resume(const char *device, Error **errp)
{
AioContext *aio_context;
- BlockJob *job = find_block_job(device, &aio_context, errp);
+ BlockJob *job;
+
+ JOB_LOCK_GUARD();
+ job = find_block_job_locked(device, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_block_job_resume(job);
- job_user_resume(&job->job, errp);
+ job_user_resume_locked(&job->job, errp);
aio_context_release(aio_context);
}
void qmp_block_job_complete(const char *device, Error **errp)
{
AioContext *aio_context;
- BlockJob *job = find_block_job(device, &aio_context, errp);
+ BlockJob *job;
+
+ JOB_LOCK_GUARD();
+ job = find_block_job_locked(device, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_block_job_complete(job);
- job_complete(&job->job, errp);
+ job_complete_locked(&job->job, errp);
aio_context_release(aio_context);
}
void qmp_block_job_finalize(const char *id, Error **errp)
{
AioContext *aio_context;
- BlockJob *job = find_block_job(id, &aio_context, errp);
+ BlockJob *job;
+
+ JOB_LOCK_GUARD();
+ job = find_block_job_locked(id, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_block_job_finalize(job);
- job_ref(&job->job);
- job_finalize(&job->job, errp);
+ job_ref_locked(&job->job);
+ job_finalize_locked(&job->job, errp);
/*
* Job's context might have changed via job_finalize (and job_txn_apply
@@ -3426,23 +3448,26 @@ void qmp_block_job_finalize(const char *id, Error **errp)
* one.
*/
aio_context = block_job_get_aio_context(job);
- job_unref(&job->job);
+ job_unref_locked(&job->job);
aio_context_release(aio_context);
}
void qmp_block_job_dismiss(const char *id, Error **errp)
{
AioContext *aio_context;
- BlockJob *bjob = find_block_job(id, &aio_context, errp);
+ BlockJob *bjob;
Job *job;
+ JOB_LOCK_GUARD();
+ bjob = find_block_job_locked(id, &aio_context, errp);
+
if (!bjob) {
return;
}
trace_qmp_block_job_dismiss(bjob);
job = &bjob->job;
- job_dismiss(&job, errp);
+ job_dismiss_locked(&job, errp);
aio_context_release(aio_context);
}
diff --git a/job-qmp.c b/job-qmp.c
index 829a28aa70..ac11a6c23c 100644
--- a/job-qmp.c
+++ b/job-qmp.c
@@ -29,14 +29,19 @@
#include "qapi/error.h"
#include "trace/trace-root.h"
-/* Get a job using its ID and acquire its AioContext */
-static Job *find_job(const char *id, AioContext **aio_context, Error **errp)
+/*
+ * Get a block job using its ID and acquire its AioContext.
+ * Called with job_mutex held.
+ */
+static Job *find_job_locked(const char *id,
+ AioContext **aio_context,
+ Error **errp)
{
Job *job;
*aio_context = NULL;
- job = job_get(id);
+ job = job_get_locked(id);
if (!job) {
error_setg(errp, "Job not found");
return NULL;
@@ -51,71 +56,86 @@ static Job *find_job(const char *id, AioContext **aio_context, Error **errp)
void qmp_job_cancel(const char *id, Error **errp)
{
AioContext *aio_context;
- Job *job = find_job(id, &aio_context, errp);
+ Job *job;
+
+ JOB_LOCK_GUARD();
+ job = find_job_locked(id, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_job_cancel(job);
- job_user_cancel(job, true, errp);
+ job_user_cancel_locked(job, true, errp);
aio_context_release(aio_context);
}
void qmp_job_pause(const char *id, Error **errp)
{
AioContext *aio_context;
- Job *job = find_job(id, &aio_context, errp);
+ Job *job;
+
+ JOB_LOCK_GUARD();
+ job = find_job_locked(id, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_job_pause(job);
- job_user_pause(job, errp);
+ job_user_pause_locked(job, errp);
aio_context_release(aio_context);
}
void qmp_job_resume(const char *id, Error **errp)
{
AioContext *aio_context;
- Job *job = find_job(id, &aio_context, errp);
+ Job *job;
+
+ JOB_LOCK_GUARD();
+ job = find_job_locked(id, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_job_resume(job);
- job_user_resume(job, errp);
+ job_user_resume_locked(job, errp);
aio_context_release(aio_context);
}
void qmp_job_complete(const char *id, Error **errp)
{
AioContext *aio_context;
- Job *job = find_job(id, &aio_context, errp);
+ Job *job;
+
+ JOB_LOCK_GUARD();
+ job = find_job_locked(id, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_job_complete(job);
- job_complete(job, errp);
+ job_complete_locked(job, errp);
aio_context_release(aio_context);
}
void qmp_job_finalize(const char *id, Error **errp)
{
AioContext *aio_context;
- Job *job = find_job(id, &aio_context, errp);
+ Job *job;
+
+ JOB_LOCK_GUARD();
+ job = find_job_locked(id, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_job_finalize(job);
- job_ref(job);
- job_finalize(job, errp);
+ job_ref_locked(job);
+ job_finalize_locked(job, errp);
/*
* Job's context might have changed via job_finalize (and job_txn_apply
@@ -123,21 +143,24 @@ void qmp_job_finalize(const char *id, Error **errp)
* one.
*/
aio_context = job->aio_context;
- job_unref(job);
+ job_unref_locked(job);
aio_context_release(aio_context);
}
void qmp_job_dismiss(const char *id, Error **errp)
{
AioContext *aio_context;
- Job *job = find_job(id, &aio_context, errp);
+ Job *job;
+
+ JOB_LOCK_GUARD();
+ job = find_job_locked(id, &aio_context, errp);
if (!job) {
return;
}
trace_qmp_job_dismiss(job);
- job_dismiss(&job, errp);
+ job_dismiss_locked(&job, errp);
aio_context_release(aio_context);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 09/21] jobs: use job locks also in the unit tests
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (7 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 08/21] jobs: add job lock in find_* functions Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-27 14:29 ` Vladimir Sementsov-Ogievskiy
2022-08-04 11:56 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 10/21] block/mirror.c: use of job helpers in drivers to avoid TOC/TOU Emanuele Giuseppe Esposito
` (11 subsequent siblings)
20 siblings, 2 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito
Add missing job synchronization in the unit tests, with
explicit locks.
We are deliberately using _locked functions wrapped by a guard
instead of a normal call because the normal call will be removed
in future, as the only usage is limited to the tests.
In other words, if a function like job_pause() is/will be only used
in tests to avoid:
WITH_JOB_LOCK_GUARD(){
job_pause_locked();
}
then it is not worth keeping job_pause(), and just use the guard.
Note: at this stage, job_{lock/unlock} and job lock guard macros
are *nop*.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
tests/unit/test-bdrv-drain.c | 76 +++++++++++++-------
tests/unit/test-block-iothread.c | 8 ++-
tests/unit/test-blockjob-txn.c | 24 ++++---
tests/unit/test-blockjob.c | 116 +++++++++++++++++++------------
4 files changed, 141 insertions(+), 83 deletions(-)
diff --git a/tests/unit/test-bdrv-drain.c b/tests/unit/test-bdrv-drain.c
index 36be84ae55..0db056ea63 100644
--- a/tests/unit/test-bdrv-drain.c
+++ b/tests/unit/test-bdrv-drain.c
@@ -943,61 +943,83 @@ static void test_blockjob_common_drain_node(enum drain_type drain_type,
}
}
- g_assert_cmpint(job->job.pause_count, ==, 0);
- g_assert_false(job->job.paused);
- g_assert_true(tjob->running);
- g_assert_true(job->job.busy); /* We're in qemu_co_sleep_ns() */
+ WITH_JOB_LOCK_GUARD() {
+ g_assert_cmpint(job->job.pause_count, ==, 0);
+ g_assert_false(job->job.paused);
+ g_assert_true(tjob->running);
+ g_assert_true(job->job.busy); /* We're in qemu_co_sleep_ns() */
+ }
do_drain_begin_unlocked(drain_type, drain_bs);
- if (drain_type == BDRV_DRAIN_ALL) {
- /* bdrv_drain_all() drains both src and target */
- g_assert_cmpint(job->job.pause_count, ==, 2);
- } else {
- g_assert_cmpint(job->job.pause_count, ==, 1);
+ WITH_JOB_LOCK_GUARD() {
+ if (drain_type == BDRV_DRAIN_ALL) {
+ /* bdrv_drain_all() drains both src and target */
+ g_assert_cmpint(job->job.pause_count, ==, 2);
+ } else {
+ g_assert_cmpint(job->job.pause_count, ==, 1);
+ }
+ g_assert_true(job->job.paused);
+ g_assert_false(job->job.busy); /* The job is paused */
}
- g_assert_true(job->job.paused);
- g_assert_false(job->job.busy); /* The job is paused */
do_drain_end_unlocked(drain_type, drain_bs);
if (use_iothread) {
- /* paused is reset in the I/O thread, wait for it */
+ /*
+ * Here we are waiting for the paused status to change,
+ * so don't bother protecting the read every time.
+ *
+ * paused is reset in the I/O thread, wait for it
+ */
while (job->job.paused) {
aio_poll(qemu_get_aio_context(), false);
}
}
- g_assert_cmpint(job->job.pause_count, ==, 0);
- g_assert_false(job->job.paused);
- g_assert_true(job->job.busy); /* We're in qemu_co_sleep_ns() */
+ WITH_JOB_LOCK_GUARD() {
+ g_assert_cmpint(job->job.pause_count, ==, 0);
+ g_assert_false(job->job.paused);
+ g_assert_true(job->job.busy); /* We're in qemu_co_sleep_ns() */
+ }
do_drain_begin_unlocked(drain_type, target);
- if (drain_type == BDRV_DRAIN_ALL) {
- /* bdrv_drain_all() drains both src and target */
- g_assert_cmpint(job->job.pause_count, ==, 2);
- } else {
- g_assert_cmpint(job->job.pause_count, ==, 1);
+ WITH_JOB_LOCK_GUARD() {
+ if (drain_type == BDRV_DRAIN_ALL) {
+ /* bdrv_drain_all() drains both src and target */
+ g_assert_cmpint(job->job.pause_count, ==, 2);
+ } else {
+ g_assert_cmpint(job->job.pause_count, ==, 1);
+ }
+ g_assert_true(job->job.paused);
+ g_assert_false(job->job.busy); /* The job is paused */
}
- g_assert_true(job->job.paused);
- g_assert_false(job->job.busy); /* The job is paused */
do_drain_end_unlocked(drain_type, target);
if (use_iothread) {
- /* paused is reset in the I/O thread, wait for it */
+ /*
+ * Here we are waiting for the paused status to change,
+ * so don't bother protecting the read every time.
+ *
+ * paused is reset in the I/O thread, wait for it
+ */
while (job->job.paused) {
aio_poll(qemu_get_aio_context(), false);
}
}
- g_assert_cmpint(job->job.pause_count, ==, 0);
- g_assert_false(job->job.paused);
- g_assert_true(job->job.busy); /* We're in qemu_co_sleep_ns() */
+ WITH_JOB_LOCK_GUARD() {
+ g_assert_cmpint(job->job.pause_count, ==, 0);
+ g_assert_false(job->job.paused);
+ g_assert_true(job->job.busy); /* We're in qemu_co_sleep_ns() */
+ }
aio_context_acquire(ctx);
- ret = job_complete_sync(&job->job, &error_abort);
+ WITH_JOB_LOCK_GUARD() {
+ ret = job_complete_sync_locked(&job->job, &error_abort);
+ }
g_assert_cmpint(ret, ==, (result == TEST_JOB_SUCCESS ? 0 : -EIO));
if (use_iothread) {
diff --git a/tests/unit/test-block-iothread.c b/tests/unit/test-block-iothread.c
index 94718c9319..89e7f0fffb 100644
--- a/tests/unit/test-block-iothread.c
+++ b/tests/unit/test-block-iothread.c
@@ -456,7 +456,9 @@ static void test_attach_blockjob(void)
}
aio_context_acquire(ctx);
- job_complete_sync(&tjob->common.job, &error_abort);
+ WITH_JOB_LOCK_GUARD() {
+ job_complete_sync_locked(&tjob->common.job, &error_abort);
+ }
blk_set_aio_context(blk, qemu_get_aio_context(), &error_abort);
aio_context_release(ctx);
@@ -630,7 +632,9 @@ static void test_propagate_mirror(void)
BLOCKDEV_ON_ERROR_REPORT, BLOCKDEV_ON_ERROR_REPORT,
false, "filter_node", MIRROR_COPY_MODE_BACKGROUND,
&error_abort);
- job = job_get("job0");
+ WITH_JOB_LOCK_GUARD() {
+ job = job_get_locked("job0");
+ }
filter = bdrv_find_node("filter_node");
/* Change the AioContext of src */
diff --git a/tests/unit/test-blockjob-txn.c b/tests/unit/test-blockjob-txn.c
index c69028b450..d3b0bb24be 100644
--- a/tests/unit/test-blockjob-txn.c
+++ b/tests/unit/test-blockjob-txn.c
@@ -116,8 +116,10 @@ static void test_single_job(int expected)
job = test_block_job_start(1, true, expected, &result, txn);
job_start(&job->job);
- if (expected == -ECANCELED) {
- job_cancel(&job->job, false);
+ WITH_JOB_LOCK_GUARD() {
+ if (expected == -ECANCELED) {
+ job_cancel_locked(&job->job, false);
+ }
}
while (result == -EINPROGRESS) {
@@ -160,13 +162,15 @@ static void test_pair_jobs(int expected1, int expected2)
/* Release our reference now to trigger as many nice
* use-after-free bugs as possible.
*/
- job_txn_unref(txn);
+ WITH_JOB_LOCK_GUARD() {
+ job_txn_unref_locked(txn);
- if (expected1 == -ECANCELED) {
- job_cancel(&job1->job, false);
- }
- if (expected2 == -ECANCELED) {
- job_cancel(&job2->job, false);
+ if (expected1 == -ECANCELED) {
+ job_cancel_locked(&job1->job, false);
+ }
+ if (expected2 == -ECANCELED) {
+ job_cancel_locked(&job2->job, false);
+ }
}
while (result1 == -EINPROGRESS || result2 == -EINPROGRESS) {
@@ -219,7 +223,9 @@ static void test_pair_jobs_fail_cancel_race(void)
job_start(&job1->job);
job_start(&job2->job);
- job_cancel(&job1->job, false);
+ WITH_JOB_LOCK_GUARD() {
+ job_cancel_locked(&job1->job, false);
+ }
/* Now make job2 finish before the main loop kicks jobs. This simulates
* the race between a pending kick and another job completing.
diff --git a/tests/unit/test-blockjob.c b/tests/unit/test-blockjob.c
index 4c9e1bf1e5..b0cd06c529 100644
--- a/tests/unit/test-blockjob.c
+++ b/tests/unit/test-blockjob.c
@@ -211,8 +211,11 @@ static CancelJob *create_common(Job **pjob)
bjob = mk_job(blk, "Steve", &test_cancel_driver, true,
JOB_MANUAL_FINALIZE | JOB_MANUAL_DISMISS);
job = &bjob->job;
- job_ref(job);
- assert(job->status == JOB_STATUS_CREATED);
+ WITH_JOB_LOCK_GUARD() {
+ job_ref_locked(job);
+ assert(job->status == JOB_STATUS_CREATED);
+ }
+
s = container_of(bjob, CancelJob, common);
s->blk = blk;
@@ -231,12 +234,14 @@ static void cancel_common(CancelJob *s)
aio_context_acquire(ctx);
job_cancel_sync(&job->job, true);
- if (sts != JOB_STATUS_CREATED && sts != JOB_STATUS_CONCLUDED) {
- Job *dummy = &job->job;
- job_dismiss(&dummy, &error_abort);
+ WITH_JOB_LOCK_GUARD() {
+ if (sts != JOB_STATUS_CREATED && sts != JOB_STATUS_CONCLUDED) {
+ Job *dummy = &job->job;
+ job_dismiss_locked(&dummy, &error_abort);
+ }
+ assert(job->job.status == JOB_STATUS_NULL);
+ job_unref_locked(&job->job);
}
- assert(job->job.status == JOB_STATUS_NULL);
- job_unref(&job->job);
destroy_blk(blk);
aio_context_release(ctx);
@@ -251,6 +256,13 @@ static void test_cancel_created(void)
cancel_common(s);
}
+static void assert_job_status_is(Job *job, int status)
+{
+ WITH_JOB_LOCK_GUARD() {
+ assert(job->status == status);
+ }
+}
+
static void test_cancel_running(void)
{
Job *job;
@@ -259,7 +271,7 @@ static void test_cancel_running(void)
s = create_common(&job);
job_start(job);
- assert(job->status == JOB_STATUS_RUNNING);
+ assert_job_status_is(job, JOB_STATUS_RUNNING);
cancel_common(s);
}
@@ -272,11 +284,12 @@ static void test_cancel_paused(void)
s = create_common(&job);
job_start(job);
- assert(job->status == JOB_STATUS_RUNNING);
-
- job_user_pause(job, &error_abort);
+ WITH_JOB_LOCK_GUARD() {
+ assert(job->status == JOB_STATUS_RUNNING);
+ job_user_pause_locked(job, &error_abort);
+ }
job_enter(job);
- assert(job->status == JOB_STATUS_PAUSED);
+ assert_job_status_is(job, JOB_STATUS_PAUSED);
cancel_common(s);
}
@@ -289,11 +302,11 @@ static void test_cancel_ready(void)
s = create_common(&job);
job_start(job);
- assert(job->status == JOB_STATUS_RUNNING);
+ assert_job_status_is(job, JOB_STATUS_RUNNING);
s->should_converge = true;
job_enter(job);
- assert(job->status == JOB_STATUS_READY);
+ assert_job_status_is(job, JOB_STATUS_READY);
cancel_common(s);
}
@@ -306,15 +319,16 @@ static void test_cancel_standby(void)
s = create_common(&job);
job_start(job);
- assert(job->status == JOB_STATUS_RUNNING);
+ assert_job_status_is(job, JOB_STATUS_RUNNING);
s->should_converge = true;
job_enter(job);
- assert(job->status == JOB_STATUS_READY);
-
- job_user_pause(job, &error_abort);
+ WITH_JOB_LOCK_GUARD() {
+ assert(job->status == JOB_STATUS_READY);
+ job_user_pause_locked(job, &error_abort);
+ }
job_enter(job);
- assert(job->status == JOB_STATUS_STANDBY);
+ assert_job_status_is(job, JOB_STATUS_STANDBY);
cancel_common(s);
}
@@ -327,20 +341,21 @@ static void test_cancel_pending(void)
s = create_common(&job);
job_start(job);
- assert(job->status == JOB_STATUS_RUNNING);
+ assert_job_status_is(job, JOB_STATUS_RUNNING);
s->should_converge = true;
job_enter(job);
- assert(job->status == JOB_STATUS_READY);
-
- job_complete(job, &error_abort);
+ WITH_JOB_LOCK_GUARD() {
+ assert(job->status == JOB_STATUS_READY);
+ job_complete_locked(job, &error_abort);
+ }
job_enter(job);
while (!job->deferred_to_main_loop) {
aio_poll(qemu_get_aio_context(), true);
}
- assert(job->status == JOB_STATUS_READY);
+ assert_job_status_is(job, JOB_STATUS_READY);
aio_poll(qemu_get_aio_context(), true);
- assert(job->status == JOB_STATUS_PENDING);
+ assert_job_status_is(job, JOB_STATUS_PENDING);
cancel_common(s);
}
@@ -353,25 +368,28 @@ static void test_cancel_concluded(void)
s = create_common(&job);
job_start(job);
- assert(job->status == JOB_STATUS_RUNNING);
+ assert_job_status_is(job, JOB_STATUS_RUNNING);
s->should_converge = true;
job_enter(job);
- assert(job->status == JOB_STATUS_READY);
-
- job_complete(job, &error_abort);
+ WITH_JOB_LOCK_GUARD() {
+ assert(job->status == JOB_STATUS_READY);
+ job_complete_locked(job, &error_abort);
+ }
job_enter(job);
while (!job->deferred_to_main_loop) {
aio_poll(qemu_get_aio_context(), true);
}
- assert(job->status == JOB_STATUS_READY);
+ assert_job_status_is(job, JOB_STATUS_READY);
aio_poll(qemu_get_aio_context(), true);
- assert(job->status == JOB_STATUS_PENDING);
+ assert_job_status_is(job, JOB_STATUS_PENDING);
aio_context_acquire(job->aio_context);
- job_finalize(job, &error_abort);
+ WITH_JOB_LOCK_GUARD() {
+ job_finalize_locked(job, &error_abort);
+ }
aio_context_release(job->aio_context);
- assert(job->status == JOB_STATUS_CONCLUDED);
+ assert_job_status_is(job, JOB_STATUS_CONCLUDED);
cancel_common(s);
}
@@ -459,36 +477,44 @@ static void test_complete_in_standby(void)
bjob = mk_job(blk, "job", &test_yielding_driver, true,
JOB_MANUAL_FINALIZE | JOB_MANUAL_DISMISS);
job = &bjob->job;
- assert(job->status == JOB_STATUS_CREATED);
+ /* Job did not start, so status is safe to read*/
+ assert_job_status_is(job, JOB_STATUS_CREATED);
/* Wait for the job to become READY */
job_start(job);
aio_context_acquire(ctx);
+ /*
+ * Here we are waiting for the status to change, so don't bother
+ * protecting the read every time.
+ */
AIO_WAIT_WHILE(ctx, job->status != JOB_STATUS_READY);
aio_context_release(ctx);
/* Begin the drained section, pausing the job */
bdrv_drain_all_begin();
- assert(job->status == JOB_STATUS_STANDBY);
+ assert_job_status_is(job, JOB_STATUS_STANDBY);
+
/* Lock the IO thread to prevent the job from being run */
aio_context_acquire(ctx);
/* This will schedule the job to resume it */
bdrv_drain_all_end();
- /* But the job cannot run, so it will remain on standby */
- assert(job->status == JOB_STATUS_STANDBY);
+ WITH_JOB_LOCK_GUARD() {
+ /* But the job cannot run, so it will remain on standby */
+ assert(job->status == JOB_STATUS_STANDBY);
- /* Even though the job is on standby, this should work */
- job_complete(job, &error_abort);
+ /* Even though the job is on standby, this should work */
+ job_complete_locked(job, &error_abort);
- /* The test is done now, clean up. */
- job_finish_sync(job, NULL, &error_abort);
- assert(job->status == JOB_STATUS_PENDING);
+ /* The test is done now, clean up. */
+ job_finish_sync_locked(job, NULL, &error_abort);
+ assert(job->status == JOB_STATUS_PENDING);
- job_finalize(job, &error_abort);
- assert(job->status == JOB_STATUS_CONCLUDED);
+ job_finalize_locked(job, &error_abort);
+ assert(job->status == JOB_STATUS_CONCLUDED);
- job_dismiss(&job, &error_abort);
+ job_dismiss_locked(&job, &error_abort);
+ }
destroy_blk(blk);
aio_context_release(ctx);
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 10/21] block/mirror.c: use of job helpers in drivers to avoid TOC/TOU
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (8 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 09/21] jobs: use job locks also in the unit tests Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-04 16:35 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 11/21] jobs: group together API calls under the same job lock Emanuele Giuseppe Esposito
` (10 subsequent siblings)
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
Once job lock is used and aiocontext is removed, mirror has
to perform job operations under the same critical section,
using the helpers prepared in previous commit.
Note: at this stage, job_{lock/unlock} and job lock guard macros
are *nop*.
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
block/mirror.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/block/mirror.c b/block/mirror.c
index d8ecb9efa2..b38676e19d 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -654,9 +654,13 @@ static int mirror_exit_common(Job *job)
BlockDriverState *target_bs;
BlockDriverState *mirror_top_bs;
Error *local_err = NULL;
- bool abort = job->ret < 0;
+ bool abort;
int ret = 0;
+ WITH_JOB_LOCK_GUARD() {
+ abort = job->ret < 0;
+ }
+
if (s->prepared) {
return 0;
}
@@ -1152,8 +1156,10 @@ static void mirror_complete(Job *job, Error **errp)
s->should_complete = true;
/* If the job is paused, it will be re-entered when it is resumed */
- if (!job->paused) {
- job_enter(job);
+ WITH_JOB_LOCK_GUARD() {
+ if (!job->paused) {
+ job_enter_cond_locked(job, NULL);
+ }
}
}
@@ -1173,8 +1179,11 @@ static bool mirror_drained_poll(BlockJob *job)
* from one of our own drain sections, to avoid a deadlock waiting for
* ourselves.
*/
- if (!s->common.job.paused && !job_is_cancelled(&job->job) && !s->in_drain) {
- return true;
+ WITH_JOB_LOCK_GUARD() {
+ if (!s->common.job.paused && !job_is_cancelled_locked(&job->job)
+ && !s->in_drain) {
+ return true;
+ }
}
return !!s->in_flight;
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 11/21] jobs: group together API calls under the same job lock
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (9 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 10/21] block/mirror.c: use of job helpers in drivers to avoid TOC/TOU Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-27 14:50 ` Vladimir Sementsov-Ogievskiy
2022-08-04 17:10 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 12/21] commit and mirror: create new nodes using bdrv_get_aio_context, and not the job aiocontext Emanuele Giuseppe Esposito
` (9 subsequent siblings)
20 siblings, 2 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito
Now that the API offers also _locked() functions, take advantage
of it and give also the caller control to take the lock and call
_locked functions.
This makes sense especially when we have for loops, because it
makes no sense to have:
for(job = job_next(); ...)
where each job_next() takes the lock internally.
Instead we want
JOB_LOCK_GUARD();
for(job = job_next_locked(); ...)
In addition, protect also direct field accesses, by either creating a
new critical section or widening the existing ones.
Note: at this stage, job_{lock/unlock} and job lock guard macros
are *nop*.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
block.c | 17 ++++++++++-------
blockdev.c | 12 +++++++++---
blockjob.c | 35 ++++++++++++++++++++++-------------
job-qmp.c | 4 +++-
job.c | 7 +++++--
monitor/qmp-cmds.c | 7 +++++--
qemu-img.c | 37 +++++++++++++++++++++----------------
7 files changed, 75 insertions(+), 44 deletions(-)
diff --git a/block.c b/block.c
index 2c00dddd80..7559965dbc 100644
--- a/block.c
+++ b/block.c
@@ -4978,8 +4978,8 @@ static void bdrv_close(BlockDriverState *bs)
void bdrv_close_all(void)
{
- assert(job_next(NULL) == NULL);
GLOBAL_STATE_CODE();
+ assert(job_next(NULL) == NULL);
/* Drop references from requests still in flight, such as canceled block
* jobs whose AIO context has not been polled yet */
@@ -6165,13 +6165,16 @@ XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp)
}
}
- for (job = block_job_next(NULL); job; job = block_job_next(job)) {
- GSList *el;
+ WITH_JOB_LOCK_GUARD() {
+ for (job = block_job_next_locked(NULL); job;
+ job = block_job_next_locked(job)) {
+ GSList *el;
- xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
- job->job.id);
- for (el = job->nodes; el; el = el->next) {
- xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
+ xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
+ job->job.id);
+ for (el = job->nodes; el; el = el->next) {
+ xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
+ }
}
}
diff --git a/blockdev.c b/blockdev.c
index 71f793c4ab..5b79093155 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -150,12 +150,15 @@ void blockdev_mark_auto_del(BlockBackend *blk)
return;
}
- for (job = block_job_next(NULL); job; job = block_job_next(job)) {
+ JOB_LOCK_GUARD();
+
+ for (job = block_job_next_locked(NULL); job;
+ job = block_job_next_locked(job)) {
if (block_job_has_bdrv(job, blk_bs(blk))) {
AioContext *aio_context = job->job.aio_context;
aio_context_acquire(aio_context);
- job_cancel(&job->job, false);
+ job_cancel_locked(&job->job, false);
aio_context_release(aio_context);
}
@@ -3745,7 +3748,10 @@ BlockJobInfoList *qmp_query_block_jobs(Error **errp)
BlockJobInfoList *head = NULL, **tail = &head;
BlockJob *job;
- for (job = block_job_next(NULL); job; job = block_job_next(job)) {
+ JOB_LOCK_GUARD();
+
+ for (job = block_job_next_locked(NULL); job;
+ job = block_job_next_locked(job)) {
BlockJobInfo *value;
AioContext *aio_context;
diff --git a/blockjob.c b/blockjob.c
index 0d59aba439..96fb9d9f73 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -111,8 +111,10 @@ static bool child_job_drained_poll(BdrvChild *c)
/* An inactive or completed job doesn't have any pending requests. Jobs
* with !job->busy are either already paused or have a pause point after
* being reentered, so no job driver code will run before they pause. */
- if (!job->busy || job_is_completed(job)) {
- return false;
+ WITH_JOB_LOCK_GUARD() {
+ if (!job->busy || job_is_completed_locked(job)) {
+ return false;
+ }
}
/* Otherwise, assume that it isn't fully stopped yet, but allow the job to
@@ -475,13 +477,15 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver,
job->ready_notifier.notify = block_job_event_ready;
job->idle_notifier.notify = block_job_on_idle;
- notifier_list_add(&job->job.on_finalize_cancelled,
- &job->finalize_cancelled_notifier);
- notifier_list_add(&job->job.on_finalize_completed,
- &job->finalize_completed_notifier);
- notifier_list_add(&job->job.on_pending, &job->pending_notifier);
- notifier_list_add(&job->job.on_ready, &job->ready_notifier);
- notifier_list_add(&job->job.on_idle, &job->idle_notifier);
+ WITH_JOB_LOCK_GUARD() {
+ notifier_list_add(&job->job.on_finalize_cancelled,
+ &job->finalize_cancelled_notifier);
+ notifier_list_add(&job->job.on_finalize_completed,
+ &job->finalize_completed_notifier);
+ notifier_list_add(&job->job.on_pending, &job->pending_notifier);
+ notifier_list_add(&job->job.on_ready, &job->ready_notifier);
+ notifier_list_add(&job->job.on_idle, &job->idle_notifier);
+ }
error_setg(&job->blocker, "block device is in use by block job: %s",
job_type_str(&job->job));
@@ -558,10 +562,15 @@ BlockErrorAction block_job_error_action(BlockJob *job, BlockdevOnError on_err,
action);
}
if (action == BLOCK_ERROR_ACTION_STOP) {
- if (!job->job.user_paused) {
- job_pause(&job->job);
- /* make the pause user visible, which will be resumed from QMP. */
- job->job.user_paused = true;
+ WITH_JOB_LOCK_GUARD() {
+ if (!job->job.user_paused) {
+ job_pause_locked(&job->job);
+ /*
+ * make the pause user visible, which will be
+ * resumed from QMP.
+ */
+ job->job.user_paused = true;
+ }
}
block_job_iostatus_set_err(job, error);
}
diff --git a/job-qmp.c b/job-qmp.c
index ac11a6c23c..cfaf34ffb7 100644
--- a/job-qmp.c
+++ b/job-qmp.c
@@ -194,7 +194,9 @@ JobInfoList *qmp_query_jobs(Error **errp)
JobInfoList *head = NULL, **tail = &head;
Job *job;
- for (job = job_next(NULL); job; job = job_next(job)) {
+ JOB_LOCK_GUARD();
+
+ for (job = job_next_locked(NULL); job; job = job_next_locked(job)) {
JobInfo *value;
AioContext *aio_context;
diff --git a/job.c b/job.c
index ebaa4e585b..b0729e2bb2 100644
--- a/job.c
+++ b/job.c
@@ -668,7 +668,7 @@ void coroutine_fn job_pause_point(Job *job)
job_pause_point_locked(job);
}
-void job_yield_locked(Job *job)
+static void job_yield_locked(Job *job)
{
assert(job->busy);
@@ -1041,11 +1041,14 @@ static void job_completed_txn_abort_locked(Job *job)
/* Called with job_mutex held, but releases it temporarily */
static int job_prepare_locked(Job *job)
{
+ int ret;
+
GLOBAL_STATE_CODE();
if (job->ret == 0 && job->driver->prepare) {
job_unlock();
- job->ret = job->driver->prepare(job);
+ ret = job->driver->prepare(job);
job_lock();
+ job->ret = ret;
job_update_rc_locked(job);
}
return job->ret;
diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index 1ebb89f46c..1897ed7a13 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -133,8 +133,11 @@ void qmp_cont(Error **errp)
blk_iostatus_reset(blk);
}
- for (job = block_job_next(NULL); job; job = block_job_next(job)) {
- block_job_iostatus_reset(job);
+ WITH_JOB_LOCK_GUARD() {
+ for (job = block_job_next_locked(NULL); job;
+ job = block_job_next_locked(job)) {
+ block_job_iostatus_reset_locked(job);
+ }
}
/* Continuing after completed migration. Images have been inactivated to
diff --git a/qemu-img.c b/qemu-img.c
index 4cf4d2423d..98c7662b0f 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -912,25 +912,30 @@ static void run_block_job(BlockJob *job, Error **errp)
int ret = 0;
aio_context_acquire(aio_context);
- job_ref(&job->job);
- do {
- float progress = 0.0f;
- aio_poll(aio_context, true);
+ WITH_JOB_LOCK_GUARD() {
+ job_ref_locked(&job->job);
+ do {
+ float progress = 0.0f;
+ job_unlock();
+ aio_poll(aio_context, true);
+
+ progress_get_snapshot(&job->job.progress, &progress_current,
+ &progress_total);
+ if (progress_total) {
+ progress = (float)progress_current / progress_total * 100.f;
+ }
+ qemu_progress_print(progress, 0);
+ job_lock();
+ } while (!job_is_ready_locked(&job->job) &&
+ !job_is_completed_locked(&job->job));
- progress_get_snapshot(&job->job.progress, &progress_current,
- &progress_total);
- if (progress_total) {
- progress = (float)progress_current / progress_total * 100.f;
+ if (!job_is_completed_locked(&job->job)) {
+ ret = job_complete_sync_locked(&job->job, errp);
+ } else {
+ ret = job->job.ret;
}
- qemu_progress_print(progress, 0);
- } while (!job_is_ready(&job->job) && !job_is_completed(&job->job));
-
- if (!job_is_completed(&job->job)) {
- ret = job_complete_sync(&job->job, errp);
- } else {
- ret = job->job.ret;
+ job_unref_locked(&job->job);
}
- job_unref(&job->job);
aio_context_release(aio_context);
/* publish completion progress only when success */
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 12/21] commit and mirror: create new nodes using bdrv_get_aio_context, and not the job aiocontext
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (10 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 11/21] jobs: group together API calls under the same job lock Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-05 8:14 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 13/21] job: detect change of aiocontext within job coroutine Emanuele Giuseppe Esposito
` (8 subsequent siblings)
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
We are always using the given bs AioContext, so there is no need
to take the job ones (which is identical anyways).
This also reduces the point we need to check when protecting
job.aio_context field.
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
block/commit.c | 4 ++--
block/mirror.c | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/block/commit.c b/block/commit.c
index 851d1c557a..336f799172 100644
--- a/block/commit.c
+++ b/block/commit.c
@@ -370,7 +370,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
goto fail;
}
- s->base = blk_new(s->common.job.aio_context,
+ s->base = blk_new(bdrv_get_aio_context(bs),
base_perms,
BLK_PERM_CONSISTENT_READ
| BLK_PERM_WRITE_UNCHANGED);
@@ -382,7 +382,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
s->base_bs = base;
/* Required permissions are already taken with block_job_add_bdrv() */
- s->top = blk_new(s->common.job.aio_context, 0, BLK_PERM_ALL);
+ s->top = blk_new(bdrv_get_aio_context(bs), 0, BLK_PERM_ALL);
ret = blk_insert_bs(s->top, top, errp);
if (ret < 0) {
goto fail;
diff --git a/block/mirror.c b/block/mirror.c
index b38676e19d..1977e25171 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -1728,7 +1728,7 @@ static BlockJob *mirror_start_job(
goto fail;
}
- s->target = blk_new(s->common.job.aio_context,
+ s->target = blk_new(bdrv_get_aio_context(bs),
target_perms, target_shared_perms);
ret = blk_insert_bs(s->target, target, errp);
if (ret < 0) {
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 13/21] job: detect change of aiocontext within job coroutine
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (11 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 12/21] commit and mirror: create new nodes using bdrv_get_aio_context, and not the job aiocontext Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-05 8:37 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex Emanuele Giuseppe Esposito
` (7 subsequent siblings)
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
From: Paolo Bonzini <pbonzini@redhat.com>
We want to make sure access of job->aio_context is always done
under either BQL or job_mutex. The problem is that using
aio_co_enter(job->aiocontext, job->co) in job_start and job_enter_cond
makes the coroutine immediately resume, so we can't hold the job lock.
And caching it is not safe either, as it might change.
job_start is under BQL, so it can freely read job->aiocontext, but
job_enter_cond is not. In order to fix this, use aio_co_wake():
the advantage is that it won't use job->aiocontext, but the
main disadvantage is that it won't be able to detect a change of
job AioContext.
Calling bdrv_try_set_aio_context() will issue the following calls
(simplified):
* in terms of bdrv callbacks:
.drained_begin -> .set_aio_context -> .drained_end
* in terms of child_job functions:
child_job_drained_begin -> child_job_set_aio_context -> child_job_drained_end
* in terms of job functions:
job_pause_locked -> job_set_aio_context -> job_resume_locked
We can see that after setting the new aio_context, job_resume_locked
calls again job_enter_cond, which then invokes aio_co_wake(). But
while job->aiocontext has been set in job_set_aio_context,
job->co->ctx has not changed, so the coroutine would be entering in
the wrong aiocontext.
Using aio_co_schedule in job_resume_locked() might seem as a valid
alternative, but the problem is that the bh resuming the coroutine
is not scheduled immediately, and if in the meanwhile another
bdrv_try_set_aio_context() is run (see test_propagate_mirror() in
test-block-iothread.c), we would have the first schedule in the
wrong aiocontext, and the second set of drains won't even manage
to schedule the coroutine, as job->busy would still be true from
the previous job_resume_locked().
The solution is to stick with aio_co_wake(), but then detect every time
the coroutine resumes back from yielding if job->aio_context
has changed. If so, we can reschedule it to the new context.
Check for the aiocontext change in job_do_yield_locked because:
1) aio_co_reschedule_self requires to be in the running coroutine
2) since child_job_set_aio_context allows changing the aiocontext only
while the job is paused, this is the exact place where the coroutine
resumes, before running JobDriver's code.
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
job.c | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/job.c b/job.c
index b0729e2bb2..ecec66b44e 100644
--- a/job.c
+++ b/job.c
@@ -585,7 +585,9 @@ void job_enter_cond_locked(Job *job, bool(*fn)(Job *job))
timer_del(&job->sleep_timer);
job->busy = true;
real_job_unlock();
- aio_co_enter(job->aio_context, job->co);
+ job_unlock();
+ aio_co_wake(job->co);
+ job_lock();
}
void job_enter_cond(Job *job, bool(*fn)(Job *job))
@@ -611,6 +613,8 @@ void job_enter(Job *job)
*/
static void coroutine_fn job_do_yield_locked(Job *job, uint64_t ns)
{
+ AioContext *next_aio_context;
+
real_job_lock();
if (ns != -1) {
timer_mod(&job->sleep_timer, ns);
@@ -622,7 +626,20 @@ static void coroutine_fn job_do_yield_locked(Job *job, uint64_t ns)
qemu_coroutine_yield();
job_lock();
- /* Set by job_enter_cond() before re-entering the coroutine. */
+ next_aio_context = job->aio_context;
+ /*
+ * Coroutine has resumed, but in the meanwhile the job AioContext
+ * might have changed via bdrv_try_set_aio_context(), so we need to move
+ * the coroutine too in the new aiocontext.
+ */
+ while (qemu_get_current_aio_context() != next_aio_context) {
+ job_unlock();
+ aio_co_reschedule_self(next_aio_context);
+ job_lock();
+ next_aio_context = job->aio_context;
+ }
+
+ /* Set by job_enter_cond_locked() before re-entering the coroutine. */
assert(job->busy);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (12 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 13/21] job: detect change of aiocontext within job coroutine Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-27 15:22 ` Vladimir Sementsov-Ogievskiy
2022-08-05 9:12 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 15/21] blockjob.h: categorize fields in struct BlockJob Emanuele Giuseppe Esposito
` (6 subsequent siblings)
20 siblings, 2 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito
In order to make it thread safe, implement a "fake rwlock",
where we allow reads under BQL *or* job_mutex held, but
writes only under BQL *and* job_mutex.
The only write we have is in child_job_set_aio_ctx, which always
happens under drain (so the job is paused).
For this reason, introduce job_set_aio_context and make sure that
the context is set under BQL, job_mutex and drain.
Also make sure all other places where the aiocontext is read
are protected.
Note: at this stage, job_{lock/unlock} and job lock guard macros
are *nop*.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
block/replication.c | 6 ++++--
blockjob.c | 3 ++-
include/qemu/job.h | 19 ++++++++++++++++++-
job.c | 12 ++++++++++++
4 files changed, 36 insertions(+), 4 deletions(-)
diff --git a/block/replication.c b/block/replication.c
index 55c8f894aa..2189863df1 100644
--- a/block/replication.c
+++ b/block/replication.c
@@ -148,8 +148,10 @@ static void replication_close(BlockDriverState *bs)
}
if (s->stage == BLOCK_REPLICATION_FAILOVER) {
commit_job = &s->commit_job->job;
- assert(commit_job->aio_context == qemu_get_current_aio_context());
- job_cancel_sync(commit_job, false);
+ WITH_JOB_LOCK_GUARD() {
+ assert(commit_job->aio_context == qemu_get_current_aio_context());
+ job_cancel_sync_locked(commit_job, false);
+ }
}
if (s->mode == REPLICATION_MODE_SECONDARY) {
diff --git a/blockjob.c b/blockjob.c
index 96fb9d9f73..9ff2727025 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -162,12 +162,13 @@ static void child_job_set_aio_ctx(BdrvChild *c, AioContext *ctx,
bdrv_set_aio_context_ignore(sibling->bs, ctx, ignore);
}
- job->job.aio_context = ctx;
+ job_set_aio_context(&job->job, ctx);
}
static AioContext *child_job_get_parent_aio_context(BdrvChild *c)
{
BlockJob *job = c->opaque;
+ assert(qemu_in_main_thread());
return job->job.aio_context;
}
diff --git a/include/qemu/job.h b/include/qemu/job.h
index 5709e8d4a8..c144aabefc 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -77,7 +77,12 @@ typedef struct Job {
/** Protected by AioContext lock */
- /** AioContext to run the job coroutine in */
+ /**
+ * AioContext to run the job coroutine in.
+ * This field can be read when holding either the BQL (so we are in
+ * the main loop) or the job_mutex.
+ * It can be only written when we hold *both* BQL and job_mutex.
+ */
AioContext *aio_context;
/** Reference count of the block job */
@@ -741,4 +746,16 @@ int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
int job_finish_sync_locked(Job *job, void (*finish)(Job *, Error **errp),
Error **errp);
+/**
+ * Sets the @job->aio_context.
+ * Called with job_mutex *not* held.
+ *
+ * This function must run in the main thread to protect against
+ * concurrent read in job_finish_sync_locked(),
+ * takes the job_mutex lock to protect against the read in
+ * job_do_yield_locked(), and must be called when the coroutine
+ * is quiescent.
+ */
+void job_set_aio_context(Job *job, AioContext *ctx);
+
#endif
diff --git a/job.c b/job.c
index ecec66b44e..0a857b1468 100644
--- a/job.c
+++ b/job.c
@@ -394,6 +394,17 @@ Job *job_get(const char *id)
return job_get_locked(id);
}
+void job_set_aio_context(Job *job, AioContext *ctx)
+{
+ /* protect against read in job_finish_sync_locked and job_start */
+ assert(qemu_in_main_thread());
+ /* protect against read in job_do_yield_locked */
+ JOB_LOCK_GUARD();
+ /* ensure the coroutine is quiescent while the AioContext is changed */
+ assert(job->pause_count > 0);
+ job->aio_context = ctx;
+}
+
/* Called with job_mutex *not* held. */
static void job_sleep_timer_cb(void *opaque)
{
@@ -1376,6 +1387,7 @@ int job_finish_sync_locked(Job *job,
{
Error *local_err = NULL;
int ret;
+ assert(qemu_in_main_thread());
job_ref_locked(job);
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 15/21] blockjob.h: categorize fields in struct BlockJob
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (13 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-05 9:21 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 16/21] blockjob: rename notifier callbacks as _locked Emanuele Giuseppe Esposito
` (5 subsequent siblings)
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
The same job lock is being used also to protect some of blockjob fields.
Categorize them just as done in job.h.
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
include/block/blockjob.h | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/include/block/blockjob.h b/include/block/blockjob.h
index 8b65d3949d..912e10b083 100644
--- a/include/block/blockjob.h
+++ b/include/block/blockjob.h
@@ -40,10 +40,16 @@ typedef struct BlockJobDriver BlockJobDriver;
* Long-running operation on a BlockDriverState.
*/
typedef struct BlockJob {
- /** Data belonging to the generic Job infrastructure */
+ /**
+ * Data belonging to the generic Job infrastructure.
+ * Protected by job mutex.
+ */
Job job;
- /** Status that is published by the query-block-jobs QMP API */
+ /**
+ * Status that is published by the query-block-jobs QMP API.
+ * Protected by job mutex.
+ */
BlockDeviceIoStatus iostatus;
/** Speed that was set with @block_job_set_speed. */
@@ -55,6 +61,8 @@ typedef struct BlockJob {
/** Block other operations when block job is running */
Error *blocker;
+ /** All notifiers are set once in block_job_create() and never modified. */
+
/** Called when a cancelled job is finalised. */
Notifier finalize_cancelled_notifier;
@@ -70,7 +78,10 @@ typedef struct BlockJob {
/** Called when the job coroutine yields or terminates */
Notifier idle_notifier;
- /** BlockDriverStates that are involved in this block job */
+ /**
+ * BlockDriverStates that are involved in this block job.
+ * Always modified and read under QEMU global mutex (GLOBAL_STATE_CODE)
+ */
GSList *nodes;
} BlockJob;
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 16/21] blockjob: rename notifier callbacks as _locked
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (14 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 15/21] blockjob.h: categorize fields in struct BlockJob Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-05 9:25 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 17/21] blockjob: protect iostatus field in BlockJob struct Emanuele Giuseppe Esposito
` (4 subsequent siblings)
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
They all are called with job_lock held, in job_event_*_locked()
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
blockjob.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/blockjob.c b/blockjob.c
index 9ff2727025..0663faee2c 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -250,7 +250,8 @@ int block_job_add_bdrv(BlockJob *job, const char *name, BlockDriverState *bs,
return 0;
}
-static void block_job_on_idle(Notifier *n, void *opaque)
+/* Called with job_mutex lock held. */
+static void block_job_on_idle_locked(Notifier *n, void *opaque)
{
aio_wait_kick();
}
@@ -370,7 +371,8 @@ static void block_job_iostatus_set_err(BlockJob *job, int error)
}
}
-static void block_job_event_cancelled(Notifier *n, void *opaque)
+/* Called with job_mutex lock held. */
+static void block_job_event_cancelled_locked(Notifier *n, void *opaque)
{
BlockJob *job = opaque;
uint64_t progress_current, progress_total;
@@ -389,7 +391,8 @@ static void block_job_event_cancelled(Notifier *n, void *opaque)
job->speed);
}
-static void block_job_event_completed(Notifier *n, void *opaque)
+/* Called with job_mutex lock held. */
+static void block_job_event_completed_locked(Notifier *n, void *opaque)
{
BlockJob *job = opaque;
const char *msg = NULL;
@@ -415,7 +418,8 @@ static void block_job_event_completed(Notifier *n, void *opaque)
msg);
}
-static void block_job_event_pending(Notifier *n, void *opaque)
+/* Called with job_mutex lock held. */
+static void block_job_event_pending_locked(Notifier *n, void *opaque)
{
BlockJob *job = opaque;
@@ -427,7 +431,8 @@ static void block_job_event_pending(Notifier *n, void *opaque)
job->job.id);
}
-static void block_job_event_ready(Notifier *n, void *opaque)
+/* Called with job_mutex lock held. */
+static void block_job_event_ready_locked(Notifier *n, void *opaque)
{
BlockJob *job = opaque;
uint64_t progress_current, progress_total;
@@ -472,11 +477,11 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver,
ratelimit_init(&job->limit);
- job->finalize_cancelled_notifier.notify = block_job_event_cancelled;
- job->finalize_completed_notifier.notify = block_job_event_completed;
- job->pending_notifier.notify = block_job_event_pending;
- job->ready_notifier.notify = block_job_event_ready;
- job->idle_notifier.notify = block_job_on_idle;
+ job->finalize_cancelled_notifier.notify = block_job_event_cancelled_locked;
+ job->finalize_completed_notifier.notify = block_job_event_completed_locked;
+ job->pending_notifier.notify = block_job_event_pending_locked;
+ job->ready_notifier.notify = block_job_event_ready_locked;
+ job->idle_notifier.notify = block_job_on_idle_locked;
WITH_JOB_LOCK_GUARD() {
notifier_list_add(&job->job.on_finalize_cancelled,
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 17/21] blockjob: protect iostatus field in BlockJob struct
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (15 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 16/21] blockjob: rename notifier callbacks as _locked Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-27 15:29 ` Vladimir Sementsov-Ogievskiy
2022-08-05 10:55 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks Emanuele Giuseppe Esposito
` (3 subsequent siblings)
20 siblings, 2 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito
iostatus is the only field (together with .job) that needs
protection using the job mutex.
It is set in the main loop (GLOBAL_STATE functions) but read
in I/O code (block_job_error_action).
In order to protect it, change block_job_iostatus_set_err
to block_job_iostatus_set_err_locked(), always called under
job lock.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
blockjob.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/blockjob.c b/blockjob.c
index 0663faee2c..448bdb5a53 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -363,7 +363,8 @@ BlockJobInfo *block_job_query(BlockJob *job, Error **errp)
return block_job_query_locked(job, errp);
}
-static void block_job_iostatus_set_err(BlockJob *job, int error)
+/* Called with job lock held */
+static void block_job_iostatus_set_err_locked(BlockJob *job, int error)
{
if (job->iostatus == BLOCK_DEVICE_IO_STATUS_OK) {
job->iostatus = error == ENOSPC ? BLOCK_DEVICE_IO_STATUS_NOSPACE :
@@ -577,8 +578,8 @@ BlockErrorAction block_job_error_action(BlockJob *job, BlockdevOnError on_err,
*/
job->job.user_paused = true;
}
+ block_job_iostatus_set_err_locked(job, error);
}
- block_job_iostatus_set_err(job, error);
}
return action;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (16 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 17/21] blockjob: protect iostatus field in BlockJob struct Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-07-27 15:53 ` Vladimir Sementsov-Ogievskiy
2022-08-05 13:01 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 19/21] block_job_query: remove atomic read Emanuele Giuseppe Esposito
` (2 subsequent siblings)
20 siblings, 2 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito
Change the job_{lock/unlock} and macros to use job_mutex.
Now that they are not nop anymore, remove the aiocontext
to avoid deadlocks.
Therefore:
- when possible, remove completely the aiocontext lock/unlock pair
- if it is used by some other function too, reduce the locking
section as much as possible, leaving the job API outside.
- change AIO_WAIT_WHILE in AIO_WAIT_WHILE_UNLOCKED, since we
are not using the aiocontext lock anymore
There is only one JobDriver callback, ->free() that assumes that
the aiocontext lock is held (because it calls bdrv_unref), so for
now keep that under aiocontext lock.
Also remove real_job_{lock/unlock}, as they are replaced by the
public functions.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
blockdev.c | 74 +++++-----------------------
include/qemu/job.h | 22 ++++-----
job-qmp.c | 46 +++--------------
job.c | 84 ++++++--------------------------
tests/unit/test-bdrv-drain.c | 4 +-
tests/unit/test-block-iothread.c | 2 +-
tests/unit/test-blockjob.c | 15 ++----
7 files changed, 52 insertions(+), 195 deletions(-)
diff --git a/blockdev.c b/blockdev.c
index 5b79093155..2cd84d206c 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -155,12 +155,7 @@ void blockdev_mark_auto_del(BlockBackend *blk)
for (job = block_job_next_locked(NULL); job;
job = block_job_next_locked(job)) {
if (block_job_has_bdrv(job, blk_bs(blk))) {
- AioContext *aio_context = job->job.aio_context;
- aio_context_acquire(aio_context);
-
job_cancel_locked(&job->job, false);
-
- aio_context_release(aio_context);
}
}
@@ -1836,14 +1831,7 @@ static void drive_backup_abort(BlkActionState *common)
DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
if (state->job) {
- AioContext *aio_context;
-
- aio_context = bdrv_get_aio_context(state->bs);
- aio_context_acquire(aio_context);
-
job_cancel_sync(&state->job->job, true);
-
- aio_context_release(aio_context);
}
}
@@ -1937,14 +1925,7 @@ static void blockdev_backup_abort(BlkActionState *common)
BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
if (state->job) {
- AioContext *aio_context;
-
- aio_context = bdrv_get_aio_context(state->bs);
- aio_context_acquire(aio_context);
-
job_cancel_sync(&state->job->job, true);
-
- aio_context_release(aio_context);
}
}
@@ -3306,19 +3287,14 @@ out:
}
/*
- * Get a block job using its ID and acquire its AioContext.
- * Called with job_mutex held.
+ * Get a block job using its ID. Called with job_mutex held.
*/
-static BlockJob *find_block_job_locked(const char *id,
- AioContext **aio_context,
- Error **errp)
+static BlockJob *find_block_job_locked(const char *id, Error **errp)
{
BlockJob *job;
assert(id != NULL);
- *aio_context = NULL;
-
job = block_job_get_locked(id);
if (!job) {
@@ -3327,36 +3303,30 @@ static BlockJob *find_block_job_locked(const char *id,
return NULL;
}
- *aio_context = block_job_get_aio_context(job);
- aio_context_acquire(*aio_context);
-
return job;
}
void qmp_block_job_set_speed(const char *device, int64_t speed, Error **errp)
{
- AioContext *aio_context;
BlockJob *job;
JOB_LOCK_GUARD();
- job = find_block_job_locked(device, &aio_context, errp);
+ job = find_block_job_locked(device, errp);
if (!job) {
return;
}
block_job_set_speed_locked(job, speed, errp);
- aio_context_release(aio_context);
}
void qmp_block_job_cancel(const char *device,
bool has_force, bool force, Error **errp)
{
- AioContext *aio_context;
BlockJob *job;
JOB_LOCK_GUARD();
- job = find_block_job_locked(device, &aio_context, errp);
+ job = find_block_job_locked(device, errp);
if (!job) {
return;
@@ -3369,22 +3339,19 @@ void qmp_block_job_cancel(const char *device,
if (job_user_paused_locked(&job->job) && !force) {
error_setg(errp, "The block job for device '%s' is currently paused",
device);
- goto out;
+ return;
}
trace_qmp_block_job_cancel(job);
job_user_cancel_locked(&job->job, force, errp);
-out:
- aio_context_release(aio_context);
}
void qmp_block_job_pause(const char *device, Error **errp)
{
- AioContext *aio_context;
BlockJob *job;
JOB_LOCK_GUARD();
- job = find_block_job_locked(device, &aio_context, errp);
+ job = find_block_job_locked(device, errp);
if (!job) {
return;
@@ -3392,16 +3359,14 @@ void qmp_block_job_pause(const char *device, Error **errp)
trace_qmp_block_job_pause(job);
job_user_pause_locked(&job->job, errp);
- aio_context_release(aio_context);
}
void qmp_block_job_resume(const char *device, Error **errp)
{
- AioContext *aio_context;
BlockJob *job;
JOB_LOCK_GUARD();
- job = find_block_job_locked(device, &aio_context, errp);
+ job = find_block_job_locked(device, errp);
if (!job) {
return;
@@ -3409,16 +3374,14 @@ void qmp_block_job_resume(const char *device, Error **errp)
trace_qmp_block_job_resume(job);
job_user_resume_locked(&job->job, errp);
- aio_context_release(aio_context);
}
void qmp_block_job_complete(const char *device, Error **errp)
{
- AioContext *aio_context;
BlockJob *job;
JOB_LOCK_GUARD();
- job = find_block_job_locked(device, &aio_context, errp);
+ job = find_block_job_locked(device, errp);
if (!job) {
return;
@@ -3426,16 +3389,14 @@ void qmp_block_job_complete(const char *device, Error **errp)
trace_qmp_block_job_complete(job);
job_complete_locked(&job->job, errp);
- aio_context_release(aio_context);
}
void qmp_block_job_finalize(const char *id, Error **errp)
{
- AioContext *aio_context;
BlockJob *job;
JOB_LOCK_GUARD();
- job = find_block_job_locked(id, &aio_context, errp);
+ job = find_block_job_locked(id, errp);
if (!job) {
return;
@@ -3445,24 +3406,16 @@ void qmp_block_job_finalize(const char *id, Error **errp)
job_ref_locked(&job->job);
job_finalize_locked(&job->job, errp);
- /*
- * Job's context might have changed via job_finalize (and job_txn_apply
- * automatically acquires the new one), so make sure we release the correct
- * one.
- */
- aio_context = block_job_get_aio_context(job);
job_unref_locked(&job->job);
- aio_context_release(aio_context);
}
void qmp_block_job_dismiss(const char *id, Error **errp)
{
- AioContext *aio_context;
BlockJob *bjob;
Job *job;
JOB_LOCK_GUARD();
- bjob = find_block_job_locked(id, &aio_context, errp);
+ bjob = find_block_job_locked(id, errp);
if (!bjob) {
return;
@@ -3471,7 +3424,6 @@ void qmp_block_job_dismiss(const char *id, Error **errp)
trace_qmp_block_job_dismiss(bjob);
job = &bjob->job;
job_dismiss_locked(&job, errp);
- aio_context_release(aio_context);
}
void qmp_change_backing_file(const char *device,
@@ -3753,15 +3705,11 @@ BlockJobInfoList *qmp_query_block_jobs(Error **errp)
for (job = block_job_next_locked(NULL); job;
job = block_job_next_locked(job)) {
BlockJobInfo *value;
- AioContext *aio_context;
if (block_job_is_internal(job)) {
continue;
}
- aio_context = block_job_get_aio_context(job);
- aio_context_acquire(aio_context);
- value = block_job_query(job, errp);
- aio_context_release(aio_context);
+ value = block_job_query_locked(job, errp);
if (!value) {
qapi_free_BlockJobInfoList(head);
return NULL;
diff --git a/include/qemu/job.h b/include/qemu/job.h
index c144aabefc..676f69bb2e 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -75,13 +75,14 @@ typedef struct Job {
ProgressMeter progress;
- /** Protected by AioContext lock */
+ /** Protected by job_mutex */
/**
* AioContext to run the job coroutine in.
- * This field can be read when holding either the BQL (so we are in
- * the main loop) or the job_mutex.
- * It can be only written when we hold *both* BQL and job_mutex.
+ * The job Aiocontext can be read when holding *either*
+ * the BQL (so we are in the main loop) or the job_mutex.
+ * It can only be written when we hold *both* BQL
+ * and the job_mutex.
*/
AioContext *aio_context;
@@ -106,7 +107,7 @@ typedef struct Job {
/**
* Set to false by the job while the coroutine has yielded and may be
* re-entered by job_enter(). There may still be I/O or event loop activity
- * pending. Accessed under block_job_mutex (in blockjob.c).
+ * pending. Accessed under job_mutex.
*
* When the job is deferred to the main loop, busy is true as long as the
* bottom half is still pending.
@@ -322,9 +323,9 @@ typedef enum JobCreateFlags {
extern QemuMutex job_mutex;
-#define JOB_LOCK_GUARD() /* QEMU_LOCK_GUARD(&job_mutex) */
+#define JOB_LOCK_GUARD() QEMU_LOCK_GUARD(&job_mutex)
-#define WITH_JOB_LOCK_GUARD() /* WITH_QEMU_LOCK_GUARD(&job_mutex) */
+#define WITH_JOB_LOCK_GUARD() WITH_QEMU_LOCK_GUARD(&job_mutex)
/**
* job_lock:
@@ -672,7 +673,7 @@ void job_user_cancel_locked(Job *job, bool force, Error **errp);
* Returns the return value from the job if the job actually completed
* during the call, or -ECANCELED if it was canceled.
*
- * Callers must hold the AioContext lock of job->aio_context.
+ * Called with job_lock held.
*/
int job_cancel_sync(Job *job, bool force);
@@ -697,8 +698,7 @@ void job_cancel_sync_all(void);
* function).
*
* Returns the return value from the job.
- *
- * Callers must hold the AioContext lock of job->aio_context.
+ * Called with job_lock held.
*/
int job_complete_sync(Job *job, Error **errp);
@@ -734,7 +734,7 @@ void job_dismiss_locked(Job **job, Error **errp);
* Returns 0 if the job is successfully completed, -ECANCELED if the job was
* cancelled before completing, and -errno in other error cases.
*
- * Callers must hold the AioContext lock of job->aio_context.
+ * Called with job_lock held.
*/
int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
Error **errp);
diff --git a/job-qmp.c b/job-qmp.c
index cfaf34ffb7..96d67246d2 100644
--- a/job-qmp.c
+++ b/job-qmp.c
@@ -30,36 +30,27 @@
#include "trace/trace-root.h"
/*
- * Get a block job using its ID and acquire its AioContext.
- * Called with job_mutex held.
+ * Get a block job using its ID. Called with job_mutex held.
*/
-static Job *find_job_locked(const char *id,
- AioContext **aio_context,
- Error **errp)
+static Job *find_job_locked(const char *id, Error **errp)
{
Job *job;
- *aio_context = NULL;
-
job = job_get_locked(id);
if (!job) {
error_setg(errp, "Job not found");
return NULL;
}
- *aio_context = job->aio_context;
- aio_context_acquire(*aio_context);
-
return job;
}
void qmp_job_cancel(const char *id, Error **errp)
{
- AioContext *aio_context;
Job *job;
JOB_LOCK_GUARD();
- job = find_job_locked(id, &aio_context, errp);
+ job = find_job_locked(id, errp);
if (!job) {
return;
@@ -67,16 +58,14 @@ void qmp_job_cancel(const char *id, Error **errp)
trace_qmp_job_cancel(job);
job_user_cancel_locked(job, true, errp);
- aio_context_release(aio_context);
}
void qmp_job_pause(const char *id, Error **errp)
{
- AioContext *aio_context;
Job *job;
JOB_LOCK_GUARD();
- job = find_job_locked(id, &aio_context, errp);
+ job = find_job_locked(id, errp);
if (!job) {
return;
@@ -84,16 +73,14 @@ void qmp_job_pause(const char *id, Error **errp)
trace_qmp_job_pause(job);
job_user_pause_locked(job, errp);
- aio_context_release(aio_context);
}
void qmp_job_resume(const char *id, Error **errp)
{
- AioContext *aio_context;
Job *job;
JOB_LOCK_GUARD();
- job = find_job_locked(id, &aio_context, errp);
+ job = find_job_locked(id, errp);
if (!job) {
return;
@@ -101,16 +88,14 @@ void qmp_job_resume(const char *id, Error **errp)
trace_qmp_job_resume(job);
job_user_resume_locked(job, errp);
- aio_context_release(aio_context);
}
void qmp_job_complete(const char *id, Error **errp)
{
- AioContext *aio_context;
Job *job;
JOB_LOCK_GUARD();
- job = find_job_locked(id, &aio_context, errp);
+ job = find_job_locked(id, errp);
if (!job) {
return;
@@ -118,16 +103,14 @@ void qmp_job_complete(const char *id, Error **errp)
trace_qmp_job_complete(job);
job_complete_locked(job, errp);
- aio_context_release(aio_context);
}
void qmp_job_finalize(const char *id, Error **errp)
{
- AioContext *aio_context;
Job *job;
JOB_LOCK_GUARD();
- job = find_job_locked(id, &aio_context, errp);
+ job = find_job_locked(id, errp);
if (!job) {
return;
@@ -137,23 +120,15 @@ void qmp_job_finalize(const char *id, Error **errp)
job_ref_locked(job);
job_finalize_locked(job, errp);
- /*
- * Job's context might have changed via job_finalize (and job_txn_apply
- * automatically acquires the new one), so make sure we release the correct
- * one.
- */
- aio_context = job->aio_context;
job_unref_locked(job);
- aio_context_release(aio_context);
}
void qmp_job_dismiss(const char *id, Error **errp)
{
- AioContext *aio_context;
Job *job;
JOB_LOCK_GUARD();
- job = find_job_locked(id, &aio_context, errp);
+ job = find_job_locked(id, errp);
if (!job) {
return;
@@ -161,7 +136,6 @@ void qmp_job_dismiss(const char *id, Error **errp)
trace_qmp_job_dismiss(job);
job_dismiss_locked(&job, errp);
- aio_context_release(aio_context);
}
static JobInfo *job_query_single(Job *job, Error **errp)
@@ -198,15 +172,11 @@ JobInfoList *qmp_query_jobs(Error **errp)
for (job = job_next_locked(NULL); job; job = job_next_locked(job)) {
JobInfo *value;
- AioContext *aio_context;
if (job_is_internal(job)) {
continue;
}
- aio_context = job->aio_context;
- aio_context_acquire(aio_context);
value = job_query_single(job, errp);
- aio_context_release(aio_context);
if (!value) {
qapi_free_JobInfoList(head);
return NULL;
diff --git a/job.c b/job.c
index 0a857b1468..9797b934d9 100644
--- a/job.c
+++ b/job.c
@@ -96,21 +96,11 @@ struct JobTxn {
};
void job_lock(void)
-{
- /* nop */
-}
-
-void job_unlock(void)
-{
- /* nop */
-}
-
-static void real_job_lock(void)
{
qemu_mutex_lock(&job_mutex);
}
-static void real_job_unlock(void)
+void job_unlock(void)
{
qemu_mutex_unlock(&job_mutex);
}
@@ -185,7 +175,6 @@ static void job_txn_del_job_locked(Job *job)
/* Called with job_mutex held, but releases it temporarily. */
static int job_txn_apply_locked(Job *job, int fn(Job *))
{
- AioContext *inner_ctx;
Job *other_job, *next;
JobTxn *txn = job->txn;
int rc = 0;
@@ -197,23 +186,14 @@ static int job_txn_apply_locked(Job *job, int fn(Job *))
* break AIO_WAIT_WHILE from within fn.
*/
job_ref_locked(job);
- aio_context_release(job->aio_context);
QLIST_FOREACH_SAFE(other_job, &txn->jobs, txn_list, next) {
- inner_ctx = other_job->aio_context;
- aio_context_acquire(inner_ctx);
rc = fn(other_job);
- aio_context_release(inner_ctx);
if (rc) {
break;
}
}
- /*
- * Note that job->aio_context might have been changed by calling fn, so we
- * can't use a local variable to cache it.
- */
- aio_context_acquire(job->aio_context);
job_unref_locked(job);
return rc;
}
@@ -501,8 +481,12 @@ void job_unref_locked(Job *job)
assert(!job->txn);
if (job->driver->free) {
+ AioContext *aio_context = job->aio_context;
job_unlock();
+ /* FIXME: aiocontext lock is required because cb calls blk_unref */
+ aio_context_acquire(aio_context);
job->driver->free(job);
+ aio_context_release(aio_context);
job_lock();
}
@@ -581,21 +565,17 @@ void job_enter_cond_locked(Job *job, bool(*fn)(Job *job))
return;
}
- real_job_lock();
if (job->busy) {
- real_job_unlock();
return;
}
if (fn && !fn(job)) {
- real_job_unlock();
return;
}
assert(!job->deferred_to_main_loop);
timer_del(&job->sleep_timer);
job->busy = true;
- real_job_unlock();
job_unlock();
aio_co_wake(job->co);
job_lock();
@@ -626,13 +606,11 @@ static void coroutine_fn job_do_yield_locked(Job *job, uint64_t ns)
{
AioContext *next_aio_context;
- real_job_lock();
if (ns != -1) {
timer_mod(&job->sleep_timer, ns);
}
job->busy = false;
job_event_idle_locked(job);
- real_job_unlock();
job_unlock();
qemu_coroutine_yield();
job_lock();
@@ -922,6 +900,7 @@ static void job_clean(Job *job)
static int job_finalize_single_locked(Job *job)
{
int job_ret;
+ AioContext *ctx = job->aio_context;
assert(job_is_completed_locked(job));
@@ -929,6 +908,7 @@ static int job_finalize_single_locked(Job *job)
job_update_rc_locked(job);
job_unlock();
+ aio_context_acquire(ctx);
if (!job->ret) {
job_commit(job);
@@ -937,6 +917,7 @@ static int job_finalize_single_locked(Job *job)
}
job_clean(job);
+ aio_context_release(ctx);
job_lock();
if (job->cb) {
@@ -1002,7 +983,6 @@ static void job_cancel_async_locked(Job *job, bool force)
/* Called with job_mutex held, but releases it temporarily. */
static void job_completed_txn_abort_locked(Job *job)
{
- AioContext *ctx;
JobTxn *txn = job->txn;
Job *other_job;
@@ -1015,54 +995,31 @@ static void job_completed_txn_abort_locked(Job *job)
txn->aborting = true;
job_txn_ref_locked(txn);
- /*
- * We can only hold the single job's AioContext lock while calling
- * job_finalize_single() because the finalization callbacks can involve
- * calls of AIO_WAIT_WHILE(), which could deadlock otherwise.
- * Note that the job's AioContext may change when it is finalized.
- */
job_ref_locked(job);
- aio_context_release(job->aio_context);
/* Other jobs are effectively cancelled by us, set the status for
* them; this job, however, may or may not be cancelled, depending
* on the caller, so leave it. */
QLIST_FOREACH(other_job, &txn->jobs, txn_list) {
if (other_job != job) {
- ctx = other_job->aio_context;
- aio_context_acquire(ctx);
/*
* This is a transaction: If one job failed, no result will matter.
* Therefore, pass force=true to terminate all other jobs as quickly
* as possible.
*/
job_cancel_async_locked(other_job, true);
- aio_context_release(ctx);
}
}
while (!QLIST_EMPTY(&txn->jobs)) {
other_job = QLIST_FIRST(&txn->jobs);
- /*
- * The job's AioContext may change, so store it in @ctx so we
- * release the same context that we have acquired before.
- */
- ctx = other_job->aio_context;
- aio_context_acquire(ctx);
if (!job_is_completed_locked(other_job)) {
assert(job_cancel_requested_locked(other_job));
job_finish_sync_locked(other_job, NULL, NULL);
}
job_finalize_single_locked(other_job);
- aio_context_release(ctx);
}
- /*
- * Use job_ref()/job_unref() so we can read the AioContext here
- * even if the job went away during job_finalize_single().
- */
- aio_context_acquire(job->aio_context);
job_unref_locked(job);
-
job_txn_unref_locked(txn);
}
@@ -1070,15 +1027,20 @@ static void job_completed_txn_abort_locked(Job *job)
static int job_prepare_locked(Job *job)
{
int ret;
+ AioContext *ctx = job->aio_context;
GLOBAL_STATE_CODE();
+
if (job->ret == 0 && job->driver->prepare) {
job_unlock();
+ aio_context_acquire(ctx);
ret = job->driver->prepare(job);
+ aio_context_release(ctx);
job_lock();
job->ret = ret;
job_update_rc_locked(job);
}
+
return job->ret;
}
@@ -1183,11 +1145,8 @@ static void job_completed_locked(Job *job)
static void job_exit(void *opaque)
{
Job *job = (Job *)opaque;
- AioContext *ctx;
JOB_LOCK_GUARD();
-
job_ref_locked(job);
- aio_context_acquire(job->aio_context);
/* This is a lie, we're not quiescent, but still doing the completion
* callbacks. However, completion callbacks tend to involve operations that
@@ -1197,16 +1156,7 @@ static void job_exit(void *opaque)
job_event_idle_locked(job);
job_completed_locked(job);
-
- /*
- * Note that calling job_completed can move the job to a different
- * aio_context, so we cannot cache from above. job_txn_apply takes care of
- * acquiring the new lock, and we ref/unref to avoid job_completed freeing
- * the job underneath us.
- */
- ctx = job->aio_context;
job_unref_locked(job);
- aio_context_release(ctx);
}
/**
@@ -1334,14 +1284,10 @@ int job_cancel_sync(Job *job, bool force)
void job_cancel_sync_all(void)
{
Job *job;
- AioContext *aio_context;
JOB_LOCK_GUARD();
while ((job = job_next_locked(NULL))) {
- aio_context = job->aio_context;
- aio_context_acquire(aio_context);
job_cancel_sync_locked(job, true);
- aio_context_release(aio_context);
}
}
@@ -1401,8 +1347,8 @@ int job_finish_sync_locked(Job *job,
}
job_unlock();
- AIO_WAIT_WHILE(job->aio_context,
- (job_enter(job), !job_is_completed(job)));
+ AIO_WAIT_WHILE_UNLOCKED(job->aio_context,
+ (job_enter(job), !job_is_completed(job)));
job_lock();
ret = (job_is_cancelled_locked(job) && job->ret == 0)
diff --git a/tests/unit/test-bdrv-drain.c b/tests/unit/test-bdrv-drain.c
index 0db056ea63..4924ceb562 100644
--- a/tests/unit/test-bdrv-drain.c
+++ b/tests/unit/test-bdrv-drain.c
@@ -930,9 +930,9 @@ static void test_blockjob_common_drain_node(enum drain_type drain_type,
tjob->prepare_ret = -EIO;
break;
}
+ aio_context_release(ctx);
job_start(&job->job);
- aio_context_release(ctx);
if (use_iothread) {
/* job_co_entry() is run in the I/O thread, wait for the actual job
@@ -1016,12 +1016,12 @@ static void test_blockjob_common_drain_node(enum drain_type drain_type,
g_assert_true(job->job.busy); /* We're in qemu_co_sleep_ns() */
}
- aio_context_acquire(ctx);
WITH_JOB_LOCK_GUARD() {
ret = job_complete_sync_locked(&job->job, &error_abort);
}
g_assert_cmpint(ret, ==, (result == TEST_JOB_SUCCESS ? 0 : -EIO));
+ aio_context_acquire(ctx);
if (use_iothread) {
blk_set_aio_context(blk_src, qemu_get_aio_context(), &error_abort);
assert(blk_get_aio_context(blk_target) == qemu_get_aio_context());
diff --git a/tests/unit/test-block-iothread.c b/tests/unit/test-block-iothread.c
index 89e7f0fffb..9d7c8be00f 100644
--- a/tests/unit/test-block-iothread.c
+++ b/tests/unit/test-block-iothread.c
@@ -455,10 +455,10 @@ static void test_attach_blockjob(void)
aio_poll(qemu_get_aio_context(), false);
}
- aio_context_acquire(ctx);
WITH_JOB_LOCK_GUARD() {
job_complete_sync_locked(&tjob->common.job, &error_abort);
}
+ aio_context_acquire(ctx);
blk_set_aio_context(blk, qemu_get_aio_context(), &error_abort);
aio_context_release(ctx);
diff --git a/tests/unit/test-blockjob.c b/tests/unit/test-blockjob.c
index b0cd06c529..8a9350078f 100644
--- a/tests/unit/test-blockjob.c
+++ b/tests/unit/test-blockjob.c
@@ -228,10 +228,6 @@ static void cancel_common(CancelJob *s)
BlockJob *job = &s->common;
BlockBackend *blk = s->blk;
JobStatus sts = job->job.status;
- AioContext *ctx;
-
- ctx = job->job.aio_context;
- aio_context_acquire(ctx);
job_cancel_sync(&job->job, true);
WITH_JOB_LOCK_GUARD() {
@@ -244,7 +240,6 @@ static void cancel_common(CancelJob *s)
}
destroy_blk(blk);
- aio_context_release(ctx);
}
static void test_cancel_created(void)
@@ -384,12 +379,10 @@ static void test_cancel_concluded(void)
aio_poll(qemu_get_aio_context(), true);
assert_job_status_is(job, JOB_STATUS_PENDING);
- aio_context_acquire(job->aio_context);
WITH_JOB_LOCK_GUARD() {
job_finalize_locked(job, &error_abort);
+ assert(job->status == JOB_STATUS_CONCLUDED);
}
- aio_context_release(job->aio_context);
- assert_job_status_is(job, JOB_STATUS_CONCLUDED);
cancel_common(s);
}
@@ -482,13 +475,11 @@ static void test_complete_in_standby(void)
/* Wait for the job to become READY */
job_start(job);
- aio_context_acquire(ctx);
/*
* Here we are waiting for the status to change, so don't bother
* protecting the read every time.
*/
- AIO_WAIT_WHILE(ctx, job->status != JOB_STATUS_READY);
- aio_context_release(ctx);
+ AIO_WAIT_WHILE_UNLOCKED(ctx, job->status != JOB_STATUS_READY);
/* Begin the drained section, pausing the job */
bdrv_drain_all_begin();
@@ -507,6 +498,7 @@ static void test_complete_in_standby(void)
job_complete_locked(job, &error_abort);
/* The test is done now, clean up. */
+ aio_context_release(ctx);
job_finish_sync_locked(job, NULL, &error_abort);
assert(job->status == JOB_STATUS_PENDING);
@@ -516,6 +508,7 @@ static void test_complete_in_standby(void)
job_dismiss_locked(&job, &error_abort);
}
+ aio_context_acquire(ctx);
destroy_blk(blk);
aio_context_release(ctx);
iothread_join(iothread);
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 19/21] block_job_query: remove atomic read
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (17 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-05 13:01 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 20/21] blockjob: remove unused functions Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 21/21] job: " Emanuele Giuseppe Esposito
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito, Vladimir Sementsov-Ogievskiy
Not sure what the atomic here was supposed to do, since job.busy
is protected by the job lock. Since the whole function
is called under job_mutex, just remove the atomic.
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
---
blockjob.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/blockjob.c b/blockjob.c
index 448bdb5a53..f0ae466c34 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -338,7 +338,7 @@ BlockJobInfo *block_job_query_locked(BlockJob *job, Error **errp)
info = g_new0(BlockJobInfo, 1);
info->type = g_strdup(job_type_str(&job->job));
info->device = g_strdup(job->job.id);
- info->busy = qatomic_read(&job->job.busy);
+ info->busy = job->job.busy;
info->paused = job->job.pause_count > 0;
info->offset = progress_current;
info->len = progress_total;
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 20/21] blockjob: remove unused functions
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (18 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 19/21] block_job_query: remove atomic read Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-05 13:05 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 21/21] job: " Emanuele Giuseppe Esposito
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito
These public functions are not used anywhere, thus can be dropped.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
blockjob.c | 16 ++--------------
include/block/blockjob.h | 31 ++++++++++++-------------------
2 files changed, 14 insertions(+), 33 deletions(-)
diff --git a/blockjob.c b/blockjob.c
index f0ae466c34..375c90e4b8 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -56,12 +56,6 @@ BlockJob *block_job_next_locked(BlockJob *bjob)
return job ? container_of(job, BlockJob, job) : NULL;
}
-BlockJob *block_job_next(BlockJob *bjob)
-{
- JOB_LOCK_GUARD();
- return block_job_next_locked(bjob);
-}
-
BlockJob *block_job_get_locked(const char *id)
{
Job *job = job_get_locked(id);
@@ -308,7 +302,7 @@ bool block_job_set_speed_locked(BlockJob *job, int64_t speed, Error **errp)
return true;
}
-bool block_job_set_speed(BlockJob *job, int64_t speed, Error **errp)
+static bool block_job_set_speed(BlockJob *job, int64_t speed, Error **errp)
{
JOB_LOCK_GUARD();
return block_job_set_speed_locked(job, speed, errp);
@@ -357,12 +351,6 @@ BlockJobInfo *block_job_query_locked(BlockJob *job, Error **errp)
return info;
}
-BlockJobInfo *block_job_query(BlockJob *job, Error **errp)
-{
- JOB_LOCK_GUARD();
- return block_job_query_locked(job, errp);
-}
-
/* Called with job lock held */
static void block_job_iostatus_set_err_locked(BlockJob *job, int error)
{
@@ -525,7 +513,7 @@ void block_job_iostatus_reset_locked(BlockJob *job)
job->iostatus = BLOCK_DEVICE_IO_STATUS_OK;
}
-void block_job_iostatus_reset(BlockJob *job)
+static void block_job_iostatus_reset(BlockJob *job)
{
JOB_LOCK_GUARD();
block_job_iostatus_reset_locked(job);
diff --git a/include/block/blockjob.h b/include/block/blockjob.h
index 912e10b083..e499d2bc90 100644
--- a/include/block/blockjob.h
+++ b/include/block/blockjob.h
@@ -93,17 +93,15 @@ typedef struct BlockJob {
*/
/**
- * block_job_next:
+ * block_job_next_locked:
* @job: A block job, or %NULL.
*
* Get the next element from the list of block jobs after @job, or the
* first one if @job is %NULL.
*
* Returns the requested job, or %NULL if there are no more jobs left.
+ * Called with job lock held.
*/
-BlockJob *block_job_next(BlockJob *job);
-
-/* Same as block_job_next(), but called with job lock held. */
BlockJob *block_job_next_locked(BlockJob *job);
/**
@@ -113,6 +111,7 @@ BlockJob *block_job_next_locked(BlockJob *job);
* Get the block job identified by @id (which must not be %NULL).
*
* Returns the requested job, or %NULL if it doesn't exist.
+ * Called with job lock *not* held.
*/
BlockJob *block_job_get(const char *id);
@@ -152,43 +151,37 @@ void block_job_remove_all_bdrv(BlockJob *job);
bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs);
/**
- * block_job_set_speed:
+ * block_job_set_speed_locked:
* @job: The job to set the speed for.
* @speed: The new value
* @errp: Error object.
*
* Set a rate-limiting parameter for the job; the actual meaning may
* vary depending on the job type.
- */
-bool block_job_set_speed(BlockJob *job, int64_t speed, Error **errp);
-
-/*
- * Same as block_job_set_speed(), but called with job lock held.
- * Might release the lock temporarily.
+ *
+ * Called with job lock held, but might release it temporarily.
*/
bool block_job_set_speed_locked(BlockJob *job, int64_t speed, Error **errp);
/**
- * block_job_query:
+ * block_job_query_locked:
* @job: The job to get information about.
*
* Return information about a job.
+ *
+ * Called with job lock held.
*/
-BlockJobInfo *block_job_query(BlockJob *job, Error **errp);
-
-/* Same as block_job_query(), but called with job lock held. */
BlockJobInfo *block_job_query_locked(BlockJob *job, Error **errp);
/**
- * block_job_iostatus_reset:
+ * block_job_iostatus_reset_locked:
* @job: The job whose I/O status should be reset.
*
* Reset I/O status on @job and on BlockDriverState objects it uses,
* other than job->blk.
+ *
+ * Called with job lock held.
*/
-void block_job_iostatus_reset(BlockJob *job);
-
-/* Same as block_job_iostatus_reset(), but called with job lock held. */
void block_job_iostatus_reset_locked(BlockJob *job);
/*
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* [PATCH v10 21/21] job: remove unused functions
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
` (19 preceding siblings ...)
2022-07-25 7:38 ` [PATCH v10 20/21] blockjob: remove unused functions Emanuele Giuseppe Esposito
@ 2022-07-25 7:38 ` Emanuele Giuseppe Esposito
2022-08-05 13:09 ` Kevin Wolf
20 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-07-25 7:38 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Emanuele Giuseppe Esposito
These public functions are not used anywhere, thus can be dropped.
Also, since this is the final job API that doesn't use AioContext
lock and replaces it with job_lock, adjust all remaining function
documentation to clearly specify if the job lock is taken or not.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
include/qemu/job.h | 97 ++++++++++++++-----------------------------
job.c | 101 ++-------------------------------------------
2 files changed, 35 insertions(+), 163 deletions(-)
diff --git a/include/qemu/job.h b/include/qemu/job.h
index 676f69bb2e..e1c0ed5940 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -361,6 +361,8 @@ JobTxn *job_txn_new(void);
/**
* Release a reference that was previously acquired with job_txn_add_job or
* job_txn_new. If it's the last reference to the object, it will be freed.
+ *
+ * Called with job lock *not* held.
*/
void job_txn_unref(JobTxn *txn);
@@ -390,19 +392,17 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
/**
* Add a reference to Job refcnt, it will be decreased with job_unref, and then
* be freed if it comes to be the last reference.
+ *
+ * Called with job lock held.
*/
-void job_ref(Job *job);
-
-/* Same as job_ref(), but called with job lock held. */
void job_ref_locked(Job *job);
/**
* Release a reference that was previously acquired with job_ref() or
* job_create(). If it's the last reference to the object, it will be freed.
+ *
+ * Called with job lock held.
*/
-void job_unref(Job *job);
-
-/* Same as job_unref(), but called with job lock held. */
void job_unref_locked(Job *job);
/**
@@ -448,12 +448,8 @@ void job_progress_increase_remaining(Job *job, uint64_t delta);
* Conditionally enter the job coroutine if the job is ready to run, not
* already busy and fn() returns true. fn() is called while under the job_lock
* critical section.
- */
-void job_enter_cond(Job *job, bool(*fn)(Job *job));
-
-/*
- * Same as job_enter_cond(), but called with job lock held.
- * Might release the lock temporarily.
+ *
+ * Called with job lock held, but might release it temporarily.
*/
void job_enter_cond_locked(Job *job, bool(*fn)(Job *job));
@@ -532,11 +528,8 @@ bool job_cancel_requested(Job *job);
/**
* Returns whether the job is in a completed state.
- * Called with job_mutex *not* held.
+ * Called with job lock held.
*/
-bool job_is_completed(Job *job);
-
-/* Same as job_is_completed(), but called with job lock held. */
bool job_is_completed_locked(Job *job);
/**
@@ -552,13 +545,15 @@ bool job_is_ready_locked(Job *job);
* Request @job to pause at the next pause point. Must be paired with
* job_resume(). If the job is supposed to be resumed by user action, call
* job_user_pause() instead.
+ *
+ * Called with job lock *not* held.
*/
void job_pause(Job *job);
/* Same as job_pause(), but called with job lock held. */
void job_pause_locked(Job *job);
-/** Resumes a @job paused with job_pause. */
+/** Resumes a @job paused with job_pause. Called with job lock *not* held. */
void job_resume(Job *job);
/*
@@ -570,27 +565,20 @@ void job_resume_locked(Job *job);
/**
* Asynchronously pause the specified @job.
* Do not allow a resume until a matching call to job_user_resume.
+ * Called with job lock held.
*/
-void job_user_pause(Job *job, Error **errp);
-
-/* Same as job_user_pause(), but called with job lock held. */
void job_user_pause_locked(Job *job, Error **errp);
-/** Returns true if the job is user-paused. */
-bool job_user_paused(Job *job);
-
-/* Same as job_user_paused(), but called with job lock held. */
+/**
+ * Returns true if the job is user-paused.
+ * Called with job lock held.
+ */
bool job_user_paused_locked(Job *job);
/**
* Resume the specified @job.
* Must be paired with a preceding job_user_pause.
- */
-void job_user_resume(Job *job, Error **errp);
-
-/*
- * Same as job_user_resume(), but called with job lock held.
- * Might release the lock temporarily.
+ * Called with job lock held, but might release it temporarily.
*/
void job_user_resume_locked(Job *job, Error **errp);
@@ -599,6 +587,7 @@ void job_user_resume_locked(Job *job, Error **errp);
* first one if @job is %NULL.
*
* Returns the requested job, or %NULL if there are no more jobs left.
+ * Called with job lock *not* held.
*/
Job *job_next(Job *job);
@@ -609,20 +598,17 @@ Job *job_next_locked(Job *job);
* Get the job identified by @id (which must not be %NULL).
*
* Returns the requested job, or %NULL if it doesn't exist.
+ * Called with job lock held.
*/
-Job *job_get(const char *id);
-
-/* Same as job_get(), but called with job lock held. */
Job *job_get_locked(const char *id);
/**
* Check whether the verb @verb can be applied to @job in its current state.
* Returns 0 if the verb can be applied; otherwise errp is set and -EPERM
* returned.
+ *
+ * Called with job lock held.
*/
-int job_apply_verb(Job *job, JobVerb verb, Error **errp);
-
-/* Same as job_apply_verb, but called with job lock held. */
int job_apply_verb_locked(Job *job, JobVerb verb, Error **errp);
/**
@@ -637,31 +623,24 @@ void job_early_fail(Job *job);
*/
void job_transition_to_ready(Job *job);
-/** Asynchronously complete the specified @job. */
-void job_complete(Job *job, Error **errp);
-
-/*
- * Same as job_complete(), but called with job lock held.
- * Might release the lock temporarily.
+/**
+ * Asynchronously complete the specified @job.
+ * Called with job lock held, but might release it temporarily.
*/
void job_complete_locked(Job *job, Error **errp);
/**
* Asynchronously cancel the specified @job. If @force is true, the job should
* be cancelled immediately without waiting for a consistent state.
+ * Called with job lock held.
*/
-void job_cancel(Job *job, bool force);
-
-/* Same as job_cancel(), but called with job lock held. */
void job_cancel_locked(Job *job, bool force);
/**
* Cancels the specified job like job_cancel(), but may refuse to do so if the
* operation isn't meaningful in the current state of the job.
+ * Called with job lock held.
*/
-void job_user_cancel(Job *job, bool force, Error **errp);
-
-/* Same as job_user_cancel(), but called with job lock held. */
void job_user_cancel_locked(Job *job, bool force, Error **errp);
/**
@@ -700,9 +679,6 @@ void job_cancel_sync_all(void);
* Returns the return value from the job.
* Called with job_lock held.
*/
-int job_complete_sync(Job *job, Error **errp);
-
-/* Same as job_complete_sync, but called with job lock held. */
int job_complete_sync_locked(Job *job, Error **errp);
/**
@@ -712,19 +688,17 @@ int job_complete_sync_locked(Job *job, Error **errp);
* FIXME: Make the below statement universally true:
* For jobs that support the manual workflow mode, all graph changes that occur
* as a result will occur after this command and before a successful reply.
+ *
+ * Called with job lock held.
*/
-void job_finalize(Job *job, Error **errp);
-
-/* Same as job_finalize(), but called with job lock held. */
void job_finalize_locked(Job *job, Error **errp);
/**
* Remove the concluded @job from the query list and resets the passed pointer
* to %NULL. Returns an error if the job is not actually concluded.
+ *
+ * Called with job lock held.
*/
-void job_dismiss(Job **job, Error **errp);
-
-/* Same as job_dismiss(), but called with job lock held. */
void job_dismiss_locked(Job **job, Error **errp);
/**
@@ -734,14 +708,7 @@ void job_dismiss_locked(Job **job, Error **errp);
* Returns 0 if the job is successfully completed, -ECANCELED if the job was
* cancelled before completing, and -errno in other error cases.
*
- * Called with job_lock held.
- */
-int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
- Error **errp);
-
-/*
- * Same as job_finish_sync(), but called with job lock held.
- * Might release the lock temporarily.
+ * Called with job_lock held, but might release it temporarily.
*/
int job_finish_sync_locked(Job *job, void (*finish)(Job *, Error **errp),
Error **errp);
diff --git a/job.c b/job.c
index 9797b934d9..747871c2e5 100644
--- a/job.c
+++ b/job.c
@@ -233,12 +233,6 @@ int job_apply_verb_locked(Job *job, JobVerb verb, Error **errp)
return -EPERM;
}
-int job_apply_verb(Job *job, JobVerb verb, Error **errp)
-{
- JOB_LOCK_GUARD();
- return job_apply_verb_locked(job, verb, errp);
-}
-
JobType job_type(const Job *job)
{
return job->driver->job_type;
@@ -324,7 +318,7 @@ bool job_is_completed_locked(Job *job)
return false;
}
-bool job_is_completed(Job *job)
+static bool job_is_completed(Job *job)
{
JOB_LOCK_GUARD();
return job_is_completed_locked(job);
@@ -368,12 +362,6 @@ Job *job_get_locked(const char *id)
return NULL;
}
-Job *job_get(const char *id)
-{
- JOB_LOCK_GUARD();
- return job_get_locked(id);
-}
-
void job_set_aio_context(Job *job, AioContext *ctx)
{
/* protect against read in job_finish_sync_locked and job_start */
@@ -465,12 +453,6 @@ void job_ref_locked(Job *job)
++job->refcnt;
}
-void job_ref(Job *job)
-{
- JOB_LOCK_GUARD();
- job_ref_locked(job);
-}
-
void job_unref_locked(Job *job)
{
GLOBAL_STATE_CODE();
@@ -499,12 +481,6 @@ void job_unref_locked(Job *job)
}
}
-void job_unref(Job *job)
-{
- JOB_LOCK_GUARD();
- job_unref_locked(job);
-}
-
void job_progress_update(Job *job, uint64_t done)
{
progress_work_done(&job->progress, done);
@@ -581,12 +557,6 @@ void job_enter_cond_locked(Job *job, bool(*fn)(Job *job))
job_lock();
}
-void job_enter_cond(Job *job, bool(*fn)(Job *job))
-{
- JOB_LOCK_GUARD();
- job_enter_cond_locked(job, fn);
-}
-
void job_enter(Job *job)
{
JOB_LOCK_GUARD();
@@ -674,8 +644,9 @@ void coroutine_fn job_pause_point(Job *job)
job_pause_point_locked(job);
}
-static void job_yield_locked(Job *job)
+void job_yield(Job *job)
{
+ JOB_LOCK_GUARD();
assert(job->busy);
/* Check cancellation *before* setting busy = false, too! */
@@ -690,12 +661,6 @@ static void job_yield_locked(Job *job)
job_pause_point_locked(job);
}
-void job_yield(Job *job)
-{
- JOB_LOCK_GUARD();
- job_yield_locked(job);
-}
-
void coroutine_fn job_sleep_ns(Job *job, int64_t ns)
{
JOB_LOCK_GUARD();
@@ -764,23 +729,11 @@ void job_user_pause_locked(Job *job, Error **errp)
job_pause_locked(job);
}
-void job_user_pause(Job *job, Error **errp)
-{
- JOB_LOCK_GUARD();
- job_user_pause_locked(job, errp);
-}
-
bool job_user_paused_locked(Job *job)
{
return job->user_paused;
}
-bool job_user_paused(Job *job)
-{
- JOB_LOCK_GUARD();
- return job_user_paused_locked(job);
-}
-
void job_user_resume_locked(Job *job, Error **errp)
{
assert(job);
@@ -801,12 +754,6 @@ void job_user_resume_locked(Job *job, Error **errp)
job_resume_locked(job);
}
-void job_user_resume(Job *job, Error **errp)
-{
- JOB_LOCK_GUARD();
- job_user_resume_locked(job, errp);
-}
-
/* Called with job_mutex held, but releases it temporarily. */
static void job_do_dismiss_locked(Job *job)
{
@@ -834,12 +781,6 @@ void job_dismiss_locked(Job **jobptr, Error **errp)
*jobptr = NULL;
}
-void job_dismiss(Job **jobptr, Error **errp)
-{
- JOB_LOCK_GUARD();
- job_dismiss_locked(jobptr, errp);
-}
-
void job_early_fail(Job *job)
{
JOB_LOCK_GUARD();
@@ -1074,12 +1015,6 @@ void job_finalize_locked(Job *job, Error **errp)
job_do_finalize_locked(job);
}
-void job_finalize(Job *job, Error **errp)
-{
- JOB_LOCK_GUARD();
- job_finalize_locked(job, errp);
-}
-
/* Called with job_mutex held. */
static int job_transition_to_pending_locked(Job *job)
{
@@ -1226,12 +1161,6 @@ void job_cancel_locked(Job *job, bool force)
}
}
-void job_cancel(Job *job, bool force)
-{
- JOB_LOCK_GUARD();
- job_cancel_locked(job, force);
-}
-
void job_user_cancel_locked(Job *job, bool force, Error **errp)
{
if (job_apply_verb_locked(job, JOB_VERB_CANCEL, errp)) {
@@ -1240,12 +1169,6 @@ void job_user_cancel_locked(Job *job, bool force, Error **errp)
job_cancel_locked(job, force);
}
-void job_user_cancel(Job *job, bool force, Error **errp)
-{
- JOB_LOCK_GUARD();
- job_user_cancel_locked(job, force, errp);
-}
-
/* A wrapper around job_cancel() taking an Error ** parameter so it may be
* used with job_finish_sync() without the need for (rather nasty) function
* pointer casts there.
@@ -1296,12 +1219,6 @@ int job_complete_sync_locked(Job *job, Error **errp)
return job_finish_sync_locked(job, job_complete_locked, errp);
}
-int job_complete_sync(Job *job, Error **errp)
-{
- JOB_LOCK_GUARD();
- return job_complete_sync_locked(job, errp);
-}
-
void job_complete_locked(Job *job, Error **errp)
{
/* Should not be reachable via external interface for internal jobs */
@@ -1321,12 +1238,6 @@ void job_complete_locked(Job *job, Error **errp)
job_lock();
}
-void job_complete(Job *job, Error **errp)
-{
- JOB_LOCK_GUARD();
- job_complete_locked(job, errp);
-}
-
int job_finish_sync_locked(Job *job,
void (*finish)(Job *, Error **errp),
Error **errp)
@@ -1356,9 +1267,3 @@ int job_finish_sync_locked(Job *job,
job_unref_locked(job);
return ret;
}
-
-int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp)
-{
- JOB_LOCK_GUARD();
- return job_finish_sync_locked(job, finish, errp);
-}
--
2.31.1
^ permalink raw reply related [flat|nested] 71+ messages in thread
* Re: [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact
2022-07-25 7:38 ` [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact Emanuele Giuseppe Esposito
@ 2022-07-27 10:45 ` Vladimir Sementsov-Ogievskiy
2022-07-29 13:33 ` Kevin Wolf
2022-08-16 18:31 ` Stefan Hajnoczi
2 siblings, 0 replies; 71+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2022-07-27 10:45 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
On 7/25/22 10:38, Emanuele Giuseppe Esposito wrote:
> With "intact" we mean that all job.h functions implicitly
> take the lock. Therefore API callers are unmodified.
>
> This means that:
> - many static functions that will be always called with job lock held
> become _locked, and call _locked functions
> - all public functions take the lock internally if needed, and call _locked
> functions
> - all public functions called internally by other functions in job.c will have a
> _locked counterpart (sometimes public), to avoid deadlocks (job lock already taken).
> These functions are not used for now.
> - some public functions called only from exernal files (not job.c) do not
> have _locked() counterpart and take the lock inside. Others won't need
> the lock at all because use fields only set at initialization and
> never modified.
>
> job_{lock/unlock} is independent from real_job_{lock/unlock}.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Honestly, you've changed the patch enough to drop my r-b.
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> ---
> include/qemu/job.h | 138 ++++++++++-
[..]
> +/* Called with job_mutex held, but releases it temporarily */
> +static int job_finalize_single_locked(Job *job)
> {
> - assert(job_is_completed(job));
> + int job_ret;
> +
> + assert(job_is_completed_locked(job));
>
> /* Ensure abort is called for late-transactional failures */
> - job_update_rc(job);
> + job_update_rc_locked(job);
> +
> + job_unlock();
With this new variant of code you read job->ret not under mutex.. Is it correct?
Probably it's OK, as here we shouldn't race with something other.. But it's simple to just move job_unlock() to beginning of "if" body, and copy to the beginning of "else" body.
>
> if (!job->ret) {
> job_commit(job);
> @@ -739,29 +895,37 @@ static int job_finalize_single(Job *job)
> }
> job_clean(job);
>
> + job_lock();
> +
> if (job->cb) {
> - job->cb(job->opaque, job->ret);
> + job_ret = job->ret;
> + job_unlock();
> + job->cb(job->opaque, job_ret);
> + job_lock();
> }
>
[..]
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 09/21] jobs: use job locks also in the unit tests
2022-07-25 7:38 ` [PATCH v10 09/21] jobs: use job locks also in the unit tests Emanuele Giuseppe Esposito
@ 2022-07-27 14:29 ` Vladimir Sementsov-Ogievskiy
2022-08-04 11:56 ` Kevin Wolf
1 sibling, 0 replies; 71+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2022-07-27 14:29 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
On 7/25/22 10:38, Emanuele Giuseppe Esposito wrote:
> Add missing job synchronization in the unit tests, with
> explicit locks.
>
> We are deliberately using _locked functions wrapped by a guard
> instead of a normal call because the normal call will be removed
> in future, as the only usage is limited to the tests.
>
> In other words, if a function like job_pause() is/will be only used
> in tests to avoid:
>
> WITH_JOB_LOCK_GUARD(){
> job_pause_locked();
> }
>
> then it is not worth keeping job_pause(), and just use the guard.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
actually, patch is modified enough to drop r-b.
> ---
> tests/unit/test-bdrv-drain.c | 76 +++++++++++++-------
> tests/unit/test-block-iothread.c | 8 ++-
> tests/unit/test-blockjob-txn.c | 24 ++++---
> tests/unit/test-blockjob.c | 116 +++++++++++++++++++------------
> 4 files changed, 141 insertions(+), 83 deletions(-)
[..]
> @@ -459,36 +477,44 @@ static void test_complete_in_standby(void)
> bjob = mk_job(blk, "job", &test_yielding_driver, true,
> JOB_MANUAL_FINALIZE | JOB_MANUAL_DISMISS);
> job = &bjob->job;
> - assert(job->status == JOB_STATUS_CREATED);
> + /* Job did not start, so status is safe to read*/
comment is not needed now, let's drop it.
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 11/21] jobs: group together API calls under the same job lock
2022-07-25 7:38 ` [PATCH v10 11/21] jobs: group together API calls under the same job lock Emanuele Giuseppe Esposito
@ 2022-07-27 14:50 ` Vladimir Sementsov-Ogievskiy
2022-08-04 17:10 ` Kevin Wolf
1 sibling, 0 replies; 71+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2022-07-27 14:50 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
On 7/25/22 10:38, Emanuele Giuseppe Esposito wrote:
> Now that the API offers also _locked() functions, take advantage
> of it and give also the caller control to take the lock and call
> _locked functions.
>
> This makes sense especially when we have for loops, because it
> makes no sense to have:
>
> for(job = job_next(); ...)
>
> where each job_next() takes the lock internally.
> Instead we want
>
> JOB_LOCK_GUARD();
> for(job = job_next_locked(); ...)
>
> In addition, protect also direct field accesses, by either creating a
> new critical section or widening the existing ones.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> ---
> block.c | 17 ++++++++++-------
> blockdev.c | 12 +++++++++---
> blockjob.c | 35 ++++++++++++++++++++++-------------
> job-qmp.c | 4 +++-
> job.c | 7 +++++--
> monitor/qmp-cmds.c | 7 +++++--
> qemu-img.c | 37 +++++++++++++++++++++----------------
> 7 files changed, 75 insertions(+), 44 deletions(-)
>
> diff --git a/block.c b/block.c
> index 2c00dddd80..7559965dbc 100644
> --- a/block.c
> +++ b/block.c
> @@ -4978,8 +4978,8 @@ static void bdrv_close(BlockDriverState *bs)
>
> void bdrv_close_all(void)
> {
> - assert(job_next(NULL) == NULL);
> GLOBAL_STATE_CODE();
> + assert(job_next(NULL) == NULL);
>
> /* Drop references from requests still in flight, such as canceled block
> * jobs whose AIO context has not been polled yet */
> @@ -6165,13 +6165,16 @@ XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp)
> }
> }
>
> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> - GSList *el;
> + WITH_JOB_LOCK_GUARD() {
> + for (job = block_job_next_locked(NULL); job;
> + job = block_job_next_locked(job)) {
> + GSList *el;
>
> - xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
> - job->job.id);
> - for (el = job->nodes; el; el = el->next) {
> - xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
> + xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
> + job->job.id);
> + for (el = job->nodes; el; el = el->next) {
> + xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
> + }
> }
> }
>
> diff --git a/blockdev.c b/blockdev.c
> index 71f793c4ab..5b79093155 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -150,12 +150,15 @@ void blockdev_mark_auto_del(BlockBackend *blk)
> return;
> }
>
> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> + JOB_LOCK_GUARD();
> +
> + for (job = block_job_next_locked(NULL); job;
> + job = block_job_next_locked(job)) {
> if (block_job_has_bdrv(job, blk_bs(blk))) {
> AioContext *aio_context = job->job.aio_context;
> aio_context_acquire(aio_context);
>
> - job_cancel(&job->job, false);
> + job_cancel_locked(&job->job, false);
>
> aio_context_release(aio_context);
> }
> @@ -3745,7 +3748,10 @@ BlockJobInfoList *qmp_query_block_jobs(Error **errp)
> BlockJobInfoList *head = NULL, **tail = &head;
> BlockJob *job;
>
> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> + JOB_LOCK_GUARD();
> +
> + for (job = block_job_next_locked(NULL); job;
> + job = block_job_next_locked(job)) {
> BlockJobInfo *value;
> AioContext *aio_context;
below block_job_query should be changed to block_job_query_locked()
>
> diff --git a/blockjob.c b/blockjob.c
> index 0d59aba439..96fb9d9f73 100644
> --- a/blockjob.c
> +++ b/blockjob.c
> @@ -111,8 +111,10 @@ static bool child_job_drained_poll(BdrvChild *c)
> /* An inactive or completed job doesn't have any pending requests. Jobs
> * with !job->busy are either already paused or have a pause point after
> * being reentered, so no job driver code will run before they pause. */
> - if (!job->busy || job_is_completed(job)) {
> - return false;
> + WITH_JOB_LOCK_GUARD() {
> + if (!job->busy || job_is_completed_locked(job)) {
> + return false;
> + }
> }
>
> /* Otherwise, assume that it isn't fully stopped yet, but allow the job to
> @@ -475,13 +477,15 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver,
> job->ready_notifier.notify = block_job_event_ready;
> job->idle_notifier.notify = block_job_on_idle;
>
> - notifier_list_add(&job->job.on_finalize_cancelled,
> - &job->finalize_cancelled_notifier);
> - notifier_list_add(&job->job.on_finalize_completed,
> - &job->finalize_completed_notifier);
> - notifier_list_add(&job->job.on_pending, &job->pending_notifier);
> - notifier_list_add(&job->job.on_ready, &job->ready_notifier);
> - notifier_list_add(&job->job.on_idle, &job->idle_notifier);
> + WITH_JOB_LOCK_GUARD() {
> + notifier_list_add(&job->job.on_finalize_cancelled,
> + &job->finalize_cancelled_notifier);
> + notifier_list_add(&job->job.on_finalize_completed,
> + &job->finalize_completed_notifier);
> + notifier_list_add(&job->job.on_pending, &job->pending_notifier);
> + notifier_list_add(&job->job.on_ready, &job->ready_notifier);
> + notifier_list_add(&job->job.on_idle, &job->idle_notifier);
> + }
>
> error_setg(&job->blocker, "block device is in use by block job: %s",
> job_type_str(&job->job));
> @@ -558,10 +562,15 @@ BlockErrorAction block_job_error_action(BlockJob *job, BlockdevOnError on_err,
> action);
> }
> if (action == BLOCK_ERROR_ACTION_STOP) {
> - if (!job->job.user_paused) {
> - job_pause(&job->job);
> - /* make the pause user visible, which will be resumed from QMP. */
> - job->job.user_paused = true;
> + WITH_JOB_LOCK_GUARD() {
> + if (!job->job.user_paused) {
> + job_pause_locked(&job->job);
> + /*
> + * make the pause user visible, which will be
> + * resumed from QMP.
> + */
> + job->job.user_paused = true;
> + }
> }
> block_job_iostatus_set_err(job, error);
> }
> diff --git a/job-qmp.c b/job-qmp.c
> index ac11a6c23c..cfaf34ffb7 100644
> --- a/job-qmp.c
> +++ b/job-qmp.c
> @@ -194,7 +194,9 @@ JobInfoList *qmp_query_jobs(Error **errp)
> JobInfoList *head = NULL, **tail = &head;
> Job *job;
>
> - for (job = job_next(NULL); job; job = job_next(job)) {
> + JOB_LOCK_GUARD();
> +
> + for (job = job_next_locked(NULL); job; job = job_next_locked(job)) {
> JobInfo *value;
> AioContext *aio_context;
>
> diff --git a/job.c b/job.c
> index ebaa4e585b..b0729e2bb2 100644
> --- a/job.c
> +++ b/job.c
Two hunks of job.c should be here, they are actually from patch 05.
> @@ -668,7 +668,7 @@ void coroutine_fn job_pause_point(Job *job)
> job_pause_point_locked(job);
> }
>
> -void job_yield_locked(Job *job)
> +static void job_yield_locked(Job *job)
> {
> assert(job->busy);
>
> @@ -1041,11 +1041,14 @@ static void job_completed_txn_abort_locked(Job *job)
> /* Called with job_mutex held, but releases it temporarily */
> static int job_prepare_locked(Job *job)
> {
> + int ret;
> +
> GLOBAL_STATE_CODE();
> if (job->ret == 0 && job->driver->prepare) {
> job_unlock();
> - job->ret = job->driver->prepare(job);
> + ret = job->driver->prepare(job);
> job_lock();
> + job->ret = ret;
> job_update_rc_locked(job);
> }
> return job->ret;
> diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
> index 1ebb89f46c..1897ed7a13 100644
> --- a/monitor/qmp-cmds.c
> +++ b/monitor/qmp-cmds.c
> @@ -133,8 +133,11 @@ void qmp_cont(Error **errp)
> blk_iostatus_reset(blk);
> }
>
> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> - block_job_iostatus_reset(job);
> + WITH_JOB_LOCK_GUARD() {
> + for (job = block_job_next_locked(NULL); job;
> + job = block_job_next_locked(job)) {
> + block_job_iostatus_reset_locked(job);
> + }
> }
>
> /* Continuing after completed migration. Images have been inactivated to
> diff --git a/qemu-img.c b/qemu-img.c
> index 4cf4d2423d..98c7662b0f 100644
> --- a/qemu-img.c
> +++ b/qemu-img.c
> @@ -912,25 +912,30 @@ static void run_block_job(BlockJob *job, Error **errp)
> int ret = 0;
>
> aio_context_acquire(aio_context);
> - job_ref(&job->job);
> - do {
> - float progress = 0.0f;
> - aio_poll(aio_context, true);
> + WITH_JOB_LOCK_GUARD() {
> + job_ref_locked(&job->job);
> + do {
> + float progress = 0.0f;
> + job_unlock();
> + aio_poll(aio_context, true);
> +
> + progress_get_snapshot(&job->job.progress, &progress_current,
> + &progress_total);
indent broken
> + if (progress_total) {
> + progress = (float)progress_current / progress_total * 100.f;
> + }
> + qemu_progress_print(progress, 0);
> + job_lock();
> + } while (!job_is_ready_locked(&job->job) &&
> + !job_is_completed_locked(&job->job));
>
> - progress_get_snapshot(&job->job.progress, &progress_current,
> - &progress_total);
> - if (progress_total) {
> - progress = (float)progress_current / progress_total * 100.f;
> + if (!job_is_completed_locked(&job->job)) {
> + ret = job_complete_sync_locked(&job->job, errp);
> + } else {
> + ret = job->job.ret;
> }
> - qemu_progress_print(progress, 0);
> - } while (!job_is_ready(&job->job) && !job_is_completed(&job->job));
> -
> - if (!job_is_completed(&job->job)) {
> - ret = job_complete_sync(&job->job, errp);
> - } else {
> - ret = job->job.ret;
> + job_unref_locked(&job->job);
> }
> - job_unref(&job->job);
> aio_context_release(aio_context);
>
> /* publish completion progress only when success */
With my notes fixed:
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex
2022-07-25 7:38 ` [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex Emanuele Giuseppe Esposito
@ 2022-07-27 15:22 ` Vladimir Sementsov-Ogievskiy
2022-08-05 9:12 ` Kevin Wolf
1 sibling, 0 replies; 71+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2022-07-27 15:22 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
On 7/25/22 10:38, Emanuele Giuseppe Esposito wrote:
> In order to make it thread safe, implement a "fake rwlock",
> where we allow reads under BQL *or* job_mutex held, but
> writes only under BQL *and* job_mutex.
>
> The only write we have is in child_job_set_aio_ctx, which always
> happens under drain (so the job is paused).
> For this reason, introduce job_set_aio_context and make sure that
> the context is set under BQL, job_mutex and drain.
actually, we only make sure that pause was scheduled
> Also make sure all other places where the aiocontext is read
> are protected.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> block/replication.c | 6 ++++--
> blockjob.c | 3 ++-
> include/qemu/job.h | 19 ++++++++++++++++++-
> job.c | 12 ++++++++++++
> 4 files changed, 36 insertions(+), 4 deletions(-)
>
> diff --git a/block/replication.c b/block/replication.c
> index 55c8f894aa..2189863df1 100644
> --- a/block/replication.c
> +++ b/block/replication.c
> @@ -148,8 +148,10 @@ static void replication_close(BlockDriverState *bs)
> }
> if (s->stage == BLOCK_REPLICATION_FAILOVER) {
> commit_job = &s->commit_job->job;
> - assert(commit_job->aio_context == qemu_get_current_aio_context());
> - job_cancel_sync(commit_job, false);
> + WITH_JOB_LOCK_GUARD() {
> + assert(commit_job->aio_context == qemu_get_current_aio_context());
> + job_cancel_sync_locked(commit_job, false);
> + }
> }
>
> if (s->mode == REPLICATION_MODE_SECONDARY) {
> diff --git a/blockjob.c b/blockjob.c
> index 96fb9d9f73..9ff2727025 100644
> --- a/blockjob.c
> +++ b/blockjob.c
> @@ -162,12 +162,13 @@ static void child_job_set_aio_ctx(BdrvChild *c, AioContext *ctx,
> bdrv_set_aio_context_ignore(sibling->bs, ctx, ignore);
> }
>
> - job->job.aio_context = ctx;
> + job_set_aio_context(&job->job, ctx);
> }
>
> static AioContext *child_job_get_parent_aio_context(BdrvChild *c)
> {
> BlockJob *job = c->opaque;
> + assert(qemu_in_main_thread());
>
> return job->job.aio_context;
> }
> diff --git a/include/qemu/job.h b/include/qemu/job.h
> index 5709e8d4a8..c144aabefc 100644
> --- a/include/qemu/job.h
> +++ b/include/qemu/job.h
> @@ -77,7 +77,12 @@ typedef struct Job {
>
> /** Protected by AioContext lock */
>
> - /** AioContext to run the job coroutine in */
> + /**
> + * AioContext to run the job coroutine in.
> + * This field can be read when holding either the BQL (so we are in
> + * the main loop) or the job_mutex.
> + * It can be only written when we hold *both* BQL and job_mutex.
> + */
> AioContext *aio_context;
>
> /** Reference count of the block job */
> @@ -741,4 +746,16 @@ int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
> int job_finish_sync_locked(Job *job, void (*finish)(Job *, Error **errp),
> Error **errp);
>
> +/**
> + * Sets the @job->aio_context.
> + * Called with job_mutex *not* held.
> + *
> + * This function must run in the main thread to protect against
> + * concurrent read in job_finish_sync_locked(),
> + * takes the job_mutex lock to protect against the read in
> + * job_do_yield_locked(), and must be called when the coroutine
> + * is quiescent.
What means "coroutine is quescent"? Could we just swap it by "job is paused"?
> + */
> +void job_set_aio_context(Job *job, AioContext *ctx);
> +
> #endif
> diff --git a/job.c b/job.c
> index ecec66b44e..0a857b1468 100644
> --- a/job.c
> +++ b/job.c
> @@ -394,6 +394,17 @@ Job *job_get(const char *id)
> return job_get_locked(id);
> }
>
> +void job_set_aio_context(Job *job, AioContext *ctx)
> +{
> + /* protect against read in job_finish_sync_locked and job_start */
> + assert(qemu_in_main_thread());
> + /* protect against read in job_do_yield_locked */
> + JOB_LOCK_GUARD();
> + /* ensure the coroutine is quiescent while the AioContext is changed */
pause_count means pause was scheduled. Job may be paused already, or may be not. I'm not against the assertion, it helps. I just think that we don't have the guarantee that comment claims. (And I still don't understand what means coroutine is quiescent. And not that there are may be several job related coroutines: the main one and some worker coroutines).
> + assert(job->pause_count > 0);
> + job->aio_context = ctx;
> +}
> +
> /* Called with job_mutex *not* held. */
> static void job_sleep_timer_cb(void *opaque)
> {
> @@ -1376,6 +1387,7 @@ int job_finish_sync_locked(Job *job,
> {
> Error *local_err = NULL;
> int ret;
> + assert(qemu_in_main_thread());
>
> job_ref_locked(job);
>
without "quiescent coroutine" concept:
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 17/21] blockjob: protect iostatus field in BlockJob struct
2022-07-25 7:38 ` [PATCH v10 17/21] blockjob: protect iostatus field in BlockJob struct Emanuele Giuseppe Esposito
@ 2022-07-27 15:29 ` Vladimir Sementsov-Ogievskiy
2022-08-16 12:39 ` Emanuele Giuseppe Esposito
2022-08-05 10:55 ` Kevin Wolf
1 sibling, 1 reply; 71+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2022-07-27 15:29 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
On 7/25/22 10:38, Emanuele Giuseppe Esposito wrote:
> iostatus is the only field (together with .job) that needs
> protection using the job mutex.
>
> It is set in the main loop (GLOBAL_STATE functions) but read
> in I/O code (block_job_error_action).
Hmm, block_job_error_action doesn't read iostatus..
Also, iostatus is read by by mirror_run, which is not protected.
>
> In order to protect it, change block_job_iostatus_set_err
> to block_job_iostatus_set_err_locked(), always called under
> job lock.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> ---
> blockjob.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/blockjob.c b/blockjob.c
> index 0663faee2c..448bdb5a53 100644
> --- a/blockjob.c
> +++ b/blockjob.c
> @@ -363,7 +363,8 @@ BlockJobInfo *block_job_query(BlockJob *job, Error **errp)
> return block_job_query_locked(job, errp);
> }
>
> -static void block_job_iostatus_set_err(BlockJob *job, int error)
> +/* Called with job lock held */
> +static void block_job_iostatus_set_err_locked(BlockJob *job, int error)
> {
> if (job->iostatus == BLOCK_DEVICE_IO_STATUS_OK) {
> job->iostatus = error == ENOSPC ? BLOCK_DEVICE_IO_STATUS_NOSPACE :
> @@ -577,8 +578,8 @@ BlockErrorAction block_job_error_action(BlockJob *job, BlockdevOnError on_err,
> */
> job->job.user_paused = true;
> }
> + block_job_iostatus_set_err_locked(job, error);
> }
> - block_job_iostatus_set_err(job, error);
> }
> return action;
> }
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks
2022-07-25 7:38 ` [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks Emanuele Giuseppe Esposito
@ 2022-07-27 15:53 ` Vladimir Sementsov-Ogievskiy
2022-08-16 12:52 ` Emanuele Giuseppe Esposito
2022-08-16 12:53 ` Emanuele Giuseppe Esposito
2022-08-05 13:01 ` Kevin Wolf
1 sibling, 2 replies; 71+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2022-07-27 15:53 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
On 7/25/22 10:38, Emanuele Giuseppe Esposito wrote:
> Change the job_{lock/unlock} and macros to use job_mutex.
>
> Now that they are not nop anymore, remove the aiocontext
> to avoid deadlocks.
>
> Therefore:
> - when possible, remove completely the aiocontext lock/unlock pair
> - if it is used by some other function too, reduce the locking
> section as much as possible, leaving the job API outside.
> - change AIO_WAIT_WHILE in AIO_WAIT_WHILE_UNLOCKED, since we
> are not using the aiocontext lock anymore
>
> There is only one JobDriver callback, ->free() that assumes that
> the aiocontext lock is held (because it calls bdrv_unref), so for
> now keep that under aiocontext lock.
>
> Also remove real_job_{lock/unlock}, as they are replaced by the
> public functions.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> blockdev.c | 74 +++++-----------------------
> include/qemu/job.h | 22 ++++-----
> job-qmp.c | 46 +++--------------
> job.c | 84 ++++++--------------------------
> tests/unit/test-bdrv-drain.c | 4 +-
> tests/unit/test-block-iothread.c | 2 +-
> tests/unit/test-blockjob.c | 15 ++----
> 7 files changed, 52 insertions(+), 195 deletions(-)
>
> diff --git a/blockdev.c b/blockdev.c
> index 5b79093155..2cd84d206c 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -155,12 +155,7 @@ void blockdev_mark_auto_del(BlockBackend *blk)
> for (job = block_job_next_locked(NULL); job;
> job = block_job_next_locked(job)) {
> if (block_job_has_bdrv(job, blk_bs(blk))) {
> - AioContext *aio_context = job->job.aio_context;
> - aio_context_acquire(aio_context);
> -
> job_cancel_locked(&job->job, false);
> -
> - aio_context_release(aio_context);
> }
> }
>
> @@ -1836,14 +1831,7 @@ static void drive_backup_abort(BlkActionState *common)
> DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
>
> if (state->job) {
> - AioContext *aio_context;
> -
> - aio_context = bdrv_get_aio_context(state->bs);
> - aio_context_acquire(aio_context);
> -
> job_cancel_sync(&state->job->job, true);
> -
> - aio_context_release(aio_context);
> }
> }
>
> @@ -1937,14 +1925,7 @@ static void blockdev_backup_abort(BlkActionState *common)
> BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
>
> if (state->job) {
> - AioContext *aio_context;
> -
> - aio_context = bdrv_get_aio_context(state->bs);
> - aio_context_acquire(aio_context);
> -
> job_cancel_sync(&state->job->job, true);
> -
> - aio_context_release(aio_context);
> }
> }
>
> @@ -3306,19 +3287,14 @@ out:
> }
>
> /*
> - * Get a block job using its ID and acquire its AioContext.
> - * Called with job_mutex held.
> + * Get a block job using its ID. Called with job_mutex held.
> */
> -static BlockJob *find_block_job_locked(const char *id,
> - AioContext **aio_context,
> - Error **errp)
> +static BlockJob *find_block_job_locked(const char *id, Error **errp)
> {
> BlockJob *job;
>
> assert(id != NULL);
>
> - *aio_context = NULL;
> -
> job = block_job_get_locked(id);
>
> if (!job) {
> @@ -3327,36 +3303,30 @@ static BlockJob *find_block_job_locked(const char *id,
> return NULL;
> }
>
> - *aio_context = block_job_get_aio_context(job);
> - aio_context_acquire(*aio_context);
> -
> return job;
> }
>
> void qmp_block_job_set_speed(const char *device, int64_t speed, Error **errp)
> {
> - AioContext *aio_context;
> BlockJob *job;
>
> JOB_LOCK_GUARD();
> - job = find_block_job_locked(device, &aio_context, errp);
> + job = find_block_job_locked(device, errp);
>
> if (!job) {
> return;
> }
>
> block_job_set_speed_locked(job, speed, errp);
> - aio_context_release(aio_context);
> }
>
> void qmp_block_job_cancel(const char *device,
> bool has_force, bool force, Error **errp)
> {
> - AioContext *aio_context;
> BlockJob *job;
>
> JOB_LOCK_GUARD();
> - job = find_block_job_locked(device, &aio_context, errp);
> + job = find_block_job_locked(device, errp);
>
> if (!job) {
> return;
> @@ -3369,22 +3339,19 @@ void qmp_block_job_cancel(const char *device,
> if (job_user_paused_locked(&job->job) && !force) {
> error_setg(errp, "The block job for device '%s' is currently paused",
> device);
> - goto out;
> + return;
> }
>
> trace_qmp_block_job_cancel(job);
> job_user_cancel_locked(&job->job, force, errp);
> -out:
> - aio_context_release(aio_context);
> }
>
> void qmp_block_job_pause(const char *device, Error **errp)
> {
> - AioContext *aio_context;
> BlockJob *job;
>
> JOB_LOCK_GUARD();
> - job = find_block_job_locked(device, &aio_context, errp);
> + job = find_block_job_locked(device, errp);
>
> if (!job) {
> return;
> @@ -3392,16 +3359,14 @@ void qmp_block_job_pause(const char *device, Error **errp)
>
> trace_qmp_block_job_pause(job);
> job_user_pause_locked(&job->job, errp);
> - aio_context_release(aio_context);
> }
>
> void qmp_block_job_resume(const char *device, Error **errp)
> {
> - AioContext *aio_context;
> BlockJob *job;
>
> JOB_LOCK_GUARD();
> - job = find_block_job_locked(device, &aio_context, errp);
> + job = find_block_job_locked(device, errp);
>
> if (!job) {
> return;
> @@ -3409,16 +3374,14 @@ void qmp_block_job_resume(const char *device, Error **errp)
>
> trace_qmp_block_job_resume(job);
> job_user_resume_locked(&job->job, errp);
> - aio_context_release(aio_context);
> }
>
> void qmp_block_job_complete(const char *device, Error **errp)
> {
> - AioContext *aio_context;
> BlockJob *job;
>
> JOB_LOCK_GUARD();
> - job = find_block_job_locked(device, &aio_context, errp);
> + job = find_block_job_locked(device, errp);
>
> if (!job) {
> return;
> @@ -3426,16 +3389,14 @@ void qmp_block_job_complete(const char *device, Error **errp)
>
> trace_qmp_block_job_complete(job);
> job_complete_locked(&job->job, errp);
> - aio_context_release(aio_context);
> }
>
> void qmp_block_job_finalize(const char *id, Error **errp)
> {
> - AioContext *aio_context;
> BlockJob *job;
>
> JOB_LOCK_GUARD();
> - job = find_block_job_locked(id, &aio_context, errp);
> + job = find_block_job_locked(id, errp);
>
> if (!job) {
> return;
> @@ -3445,24 +3406,16 @@ void qmp_block_job_finalize(const char *id, Error **errp)
> job_ref_locked(&job->job);
> job_finalize_locked(&job->job, errp);
>
> - /*
> - * Job's context might have changed via job_finalize (and job_txn_apply
> - * automatically acquires the new one), so make sure we release the correct
> - * one.
> - */
> - aio_context = block_job_get_aio_context(job);
> job_unref_locked(&job->job);
> - aio_context_release(aio_context);
> }
>
> void qmp_block_job_dismiss(const char *id, Error **errp)
> {
> - AioContext *aio_context;
> BlockJob *bjob;
> Job *job;
>
> JOB_LOCK_GUARD();
> - bjob = find_block_job_locked(id, &aio_context, errp);
> + bjob = find_block_job_locked(id, errp);
>
> if (!bjob) {
> return;
> @@ -3471,7 +3424,6 @@ void qmp_block_job_dismiss(const char *id, Error **errp)
> trace_qmp_block_job_dismiss(bjob);
> job = &bjob->job;
> job_dismiss_locked(&job, errp);
> - aio_context_release(aio_context);
> }
>
> void qmp_change_backing_file(const char *device,
> @@ -3753,15 +3705,11 @@ BlockJobInfoList *qmp_query_block_jobs(Error **errp)
> for (job = block_job_next_locked(NULL); job;
> job = block_job_next_locked(job)) {
> BlockJobInfo *value;
> - AioContext *aio_context;
>
> if (block_job_is_internal(job)) {
> continue;
> }
> - aio_context = block_job_get_aio_context(job);
> - aio_context_acquire(aio_context);
> - value = block_job_query(job, errp);
> - aio_context_release(aio_context);
> + value = block_job_query_locked(job, errp);
> if (!value) {
> qapi_free_BlockJobInfoList(head);
> return NULL;
> diff --git a/include/qemu/job.h b/include/qemu/job.h
> index c144aabefc..676f69bb2e 100644
> --- a/include/qemu/job.h
> +++ b/include/qemu/job.h
> @@ -75,13 +75,14 @@ typedef struct Job {
> ProgressMeter progress;
>
>
> - /** Protected by AioContext lock */
> + /** Protected by job_mutex */
>
> /**
> * AioContext to run the job coroutine in.
> - * This field can be read when holding either the BQL (so we are in
> - * the main loop) or the job_mutex.
> - * It can be only written when we hold *both* BQL and job_mutex.
> + * The job Aiocontext can be read when holding *either*
> + * the BQL (so we are in the main loop) or the job_mutex.
> + * It can only be written when we hold *both* BQL
> + * and the job_mutex.
You reword comment you've added some patches ago. Could you please merge this to original patch?
> */
> AioContext *aio_context;
>
> @@ -106,7 +107,7 @@ typedef struct Job {
> /**
> * Set to false by the job while the coroutine has yielded and may be
> * re-entered by job_enter(). There may still be I/O or event loop activity
> - * pending. Accessed under block_job_mutex (in blockjob.c).
> + * pending. Accessed under job_mutex.
> *
> * When the job is deferred to the main loop, busy is true as long as the
> * bottom half is still pending.
> @@ -322,9 +323,9 @@ typedef enum JobCreateFlags {
>
> extern QemuMutex job_mutex;
>
> -#define JOB_LOCK_GUARD() /* QEMU_LOCK_GUARD(&job_mutex) */
> +#define JOB_LOCK_GUARD() QEMU_LOCK_GUARD(&job_mutex)
>
> -#define WITH_JOB_LOCK_GUARD() /* WITH_QEMU_LOCK_GUARD(&job_mutex) */
> +#define WITH_JOB_LOCK_GUARD() WITH_QEMU_LOCK_GUARD(&job_mutex)
>
> /**
> * job_lock:
> @@ -672,7 +673,7 @@ void job_user_cancel_locked(Job *job, bool force, Error **errp);
> * Returns the return value from the job if the job actually completed
> * during the call, or -ECANCELED if it was canceled.
> *
> - * Callers must hold the AioContext lock of job->aio_context.
> + * Called with job_lock held.
That's wrong, it should be called with job_lock not held :)
> */
> int job_cancel_sync(Job *job, bool force);
>
> @@ -697,8 +698,7 @@ void job_cancel_sync_all(void);
> * function).
> *
> * Returns the return value from the job.
> - *
> - * Callers must hold the AioContext lock of job->aio_context.
> + * Called with job_lock held.
and this,
> */
> int job_complete_sync(Job *job, Error **errp);
>
> @@ -734,7 +734,7 @@ void job_dismiss_locked(Job **job, Error **errp);
> * Returns 0 if the job is successfully completed, -ECANCELED if the job was
> * cancelled before completing, and -errno in other error cases.
> *
> - * Callers must hold the AioContext lock of job->aio_context.
> + * Called with job_lock held.
and this.
> */
> int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
> Error **errp);
> diff --git a/job-qmp.c b/job-qmp.c
> index cfaf34ffb7..96d67246d2 100644
> --- a/job-qmp.c
> +++ b/job-qmp.c
> @@ -30,36 +30,27 @@
[..]
> }
> @@ -501,8 +481,12 @@ void job_unref_locked(Job *job)> assert(!job->txn);
>
> if (job->driver->free) {
> + AioContext *aio_context = job->aio_context;
> job_unlock();
> + /* FIXME: aiocontext lock is required because cb calls blk_unref */
> + aio_context_acquire(aio_context);
> job->driver->free(job);
> + aio_context_release(aio_context);
So, job_unref_locked() must be called with aio_context not locked, otherwise we dead-lock here? That should be documented in function declaration comment.
For example in qemu-img.c in run_block_job() we do exactly that: call job_unref_locked() inside aio-context lock critical seaction..
> job_lock();
> }
>
> @@ -581,21 +565,17 @@ void job_enter_cond_locked(Job *job, bool(*fn)(Job *job))
> return;
> }
>
> - real_job_lock();
> if (job->busy) {
> - real_job_unlock();
> return;
> }
>
> if (fn && !fn(job)) {
> - real_job_unlock();
> return;
> }
>
> assert(!job->deferred_to_main_loop);
> timer_del(&job->sleep_timer);
> job->busy = true;
> - real_job_unlock();
> job_unlock();
> aio_co_wake(job->co);
> job_lock();
> @@ -626,13 +606,11 @@ static void coroutine_fn job_do_yield_locked(Job *job, uint64_t ns)
> {
> AioContext *next_aio_context;
>
> - real_job_lock();
> if (ns != -1) {
> timer_mod(&job->sleep_timer, ns);
> }
> job->busy = false;
> job_event_idle_locked(job);
> - real_job_unlock();
> job_unlock();
> qemu_coroutine_yield();
> job_lock();
> @@ -922,6 +900,7 @@ static void job_clean(Job *job)
> static int job_finalize_single_locked(Job *job)
> {
> int job_ret;
> + AioContext *ctx = job->aio_context;
>
> assert(job_is_completed_locked(job));
>
> @@ -929,6 +908,7 @@ static int job_finalize_single_locked(Job *job)
> job_update_rc_locked(job);
>
> job_unlock();
> + aio_context_acquire(ctx);
Hmm, and this function and all its callers now should be called with aio-context lock not locked?
For example job_exit is scheduled as as BH. Aren't BHs called with aio-context lock held?
>
> if (!job->ret) {
> job_commit(job);
> @@ -937,6 +917,7 @@ static int job_finalize_single_locked(Job *job)
> }
> job_clean(job);
>
> + aio_context_release(ctx);
> job_lock();
>
> if (job->cb) {
[..]
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 02/21] job.h: categorize fields in struct Job
2022-07-25 7:38 ` [PATCH v10 02/21] job.h: categorize fields in struct Job Emanuele Giuseppe Esposito
@ 2022-07-29 12:30 ` Kevin Wolf
2022-08-16 18:28 ` Stefan Hajnoczi
1 sibling, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-07-29 12:30 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> Categorize the fields in struct Job to understand which ones
> need to be protected by the job mutex and which don't.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 03/21] job.c: API functions not used outside should be static
2022-07-25 7:38 ` [PATCH v10 03/21] job.c: API functions not used outside should be static Emanuele Giuseppe Esposito
@ 2022-07-29 12:30 ` Kevin Wolf
0 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-07-29 12:30 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> job_event_* functions can all be static, as they are not used
> outside job.c.
>
> Same applies for job_txn_add_job().
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 04/21] aio-wait.h: introduce AIO_WAIT_WHILE_UNLOCKED
2022-07-25 7:38 ` [PATCH v10 04/21] aio-wait.h: introduce AIO_WAIT_WHILE_UNLOCKED Emanuele Giuseppe Esposito
@ 2022-07-29 12:33 ` Kevin Wolf
0 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-07-29 12:33 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> Same as AIO_WAIT_WHILE macro, but if we are in the Main loop
> do not release and then acquire ctx_ 's aiocontext.
>
> Once all Aiocontext locks go away, this macro will replace
> AIO_WAIT_WHILE.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> ---
> include/block/aio-wait.h | 17 +++++++++++++----
> 1 file changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/include/block/aio-wait.h b/include/block/aio-wait.h
> index 54840f8622..a61f82c617 100644
> --- a/include/block/aio-wait.h
> +++ b/include/block/aio-wait.h
> @@ -59,10 +59,13 @@ typedef struct {
> extern AioWait global_aio_wait;
>
> /**
> - * AIO_WAIT_WHILE:
> + * _AIO_WAIT_WHILE:
> * @ctx: the aio context, or NULL if multiple aio contexts (for which the
> * caller does not hold a lock) are involved in the polling condition.
> * @cond: wait while this conditional expression is true
> + * @unlock: whether to unlock and then lock again @ctx. This apples
> + * only when waiting for another AioContext from the main loop.
> + * Otherwise it's ignored.
> *
> * Wait while a condition is true. Use this to implement synchronous
> * operations that require event loop activity.
> @@ -75,7 +78,7 @@ extern AioWait global_aio_wait;
> * wait on conditions between two IOThreads since that could lead to deadlock,
> * go via the main loop instead.
> */
> -#define AIO_WAIT_WHILE(ctx, cond) ({ \
> +#define _AIO_WAIT_WHILE(ctx, cond, unlock) ({ \
"All identifiers that begin with an underscore and either an uppercase
letter or another underscore are always reserved for any use." (C11,
7.1.3)
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact
2022-07-25 7:38 ` [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact Emanuele Giuseppe Esposito
2022-07-27 10:45 ` Vladimir Sementsov-Ogievskiy
@ 2022-07-29 13:33 ` Kevin Wolf
2022-08-16 18:31 ` Stefan Hajnoczi
2 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-07-29 13:33 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> With "intact" we mean that all job.h functions implicitly
> take the lock. Therefore API callers are unmodified.
>
> This means that:
> - many static functions that will be always called with job lock held
> become _locked, and call _locked functions
> - all public functions take the lock internally if needed, and call _locked
> functions
> - all public functions called internally by other functions in job.c will have a
> _locked counterpart (sometimes public), to avoid deadlocks (job lock already taken).
> These functions are not used for now.
> - some public functions called only from exernal files (not job.c) do not
> have _locked() counterpart and take the lock inside. Others won't need
> the lock at all because use fields only set at initialization and
> never modified.
>
> job_{lock/unlock} is independent from real_job_{lock/unlock}.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Would be nice to fix the access Vladimir found, but I think it's not
actually a bug because we know that nobody else is going to write to
job->ret. So with or without that fix:
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 06/21] job: move and update comments from blockjob.c
2022-07-25 7:38 ` [PATCH v10 06/21] job: move and update comments from blockjob.c Emanuele Giuseppe Esposito
@ 2022-08-03 15:47 ` Kevin Wolf
2022-08-16 18:32 ` Stefan Hajnoczi
1 sibling, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-03 15:47 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> This comment applies more on job, it was left in blockjob as in the past
> the whole job logic was implemented there.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> No functional change intended.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> ---
> blockjob.c | 20 --------------------
> job.c | 14 ++++++++++++++
> 2 files changed, 14 insertions(+), 20 deletions(-)
>
> diff --git a/blockjob.c b/blockjob.c
> index 4868453d74..7da59a1f1c 100644
> --- a/blockjob.c
> +++ b/blockjob.c
> @@ -36,21 +36,6 @@
> #include "qemu/main-loop.h"
> #include "qemu/timer.h"
>
> -/*
> - * The block job API is composed of two categories of functions.
> - *
> - * The first includes functions used by the monitor. The monitor is
> - * peculiar in that it accesses the block job list with block_job_get, and
> - * therefore needs consistency across block_job_get and the actual operation
> - * (e.g. block_job_set_speed). The consistency is achieved with
> - * aio_context_acquire/release. These functions are declared in blockjob.h.
> - *
> - * The second includes functions used by the block job drivers and sometimes
> - * by the core block layer. These do not care about locking, because the
> - * whole coroutine runs under the AioContext lock, and are declared in
> - * blockjob_int.h.
> - */
> -
> static bool is_block_job(Job *job)
> {
> return job_type(job) == JOB_TYPE_BACKUP ||
> @@ -433,11 +418,6 @@ static void block_job_event_ready(Notifier *n, void *opaque)
> }
>
>
> -/*
> - * API for block job drivers and the block layer. These functions are
> - * declared in blockjob_int.h.
> - */
> -
> void *block_job_create(const char *job_id, const BlockJobDriver *driver,
> JobTxn *txn, BlockDriverState *bs, uint64_t perm,
> uint64_t shared_perm, int64_t speed, int flags,
> diff --git a/job.c b/job.c
> index ae25db97ac..ebaa4e585b 100644
> --- a/job.c
> +++ b/job.c
> @@ -32,6 +32,20 @@
> #include "trace/trace-root.h"
> #include "qapi/qapi-events-job.h"
>
> +/*
> + * The job API is composed of two categories of functions.
> + *
> + * The first includes functions used by the monitor. The monitor is
> + * peculiar in that it accesses the block job list with job_get, and
s/block job/job/
> + * therefore needs consistency across job_get and the actual operation
> + * (e.g. job_user_cancel). To achieve this consistency, the caller
> + * calls job_lock/job_unlock itself around the whole operation.
> + *
> + *
> + * The second includes functions used by the block job drivers and sometimes
Same here.
> + * by the core block layer. These delegate the locking to the callee instead.
> + */
Unless I'm missing something, this comment (specifically the part with
calling job_lock/job_unlock outside of job.c) is actually not true at
this point in the series. I would suggest adding a comment to this
effect, like:
* TODO Actually make this true
Then we know that when you remove the comment, we need to review that
it's actually true at that point in the series.
For now, I'll just try to remember checking this later.
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 07/21] blockjob: introduce block_job _locked() APIs
2022-07-25 7:38 ` [PATCH v10 07/21] blockjob: introduce block_job _locked() APIs Emanuele Giuseppe Esposito
@ 2022-08-03 15:52 ` Kevin Wolf
2022-08-16 18:33 ` Stefan Hajnoczi
1 sibling, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-03 15:52 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> Just as done with job.h, create _locked() functions in blockjob.h
>
> These functions will be later useful when caller has already taken
> the lock. All blockjob _locked functions call job _locked functions.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 08/21] jobs: add job lock in find_* functions
2022-07-25 7:38 ` [PATCH v10 08/21] jobs: add job lock in find_* functions Emanuele Giuseppe Esposito
@ 2022-08-04 11:47 ` Kevin Wolf
0 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-04 11:47 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> Both blockdev.c and job-qmp.c have TOC/TOU conditions, because
> they first search for the job and then perform an action on it.
> Therefore, we need to do the search + action under the same
> job mutex critical section.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> diff --git a/job-qmp.c b/job-qmp.c
> index 829a28aa70..ac11a6c23c 100644
> --- a/job-qmp.c
> +++ b/job-qmp.c
> @@ -29,14 +29,19 @@
> #include "qapi/error.h"
> #include "trace/trace-root.h"
>
> -/* Get a job using its ID and acquire its AioContext */
> -static Job *find_job(const char *id, AioContext **aio_context, Error **errp)
> +/*
> + * Get a block job using its ID and acquire its AioContext.
This should still be just a job, not specifically a block job.
> + * Called with job_mutex held.
> + */
> +static Job *find_job_locked(const char *id,
> + AioContext **aio_context,
> + Error **errp)
> {
> Job *job;
>
> *aio_context = NULL;
>
> - job = job_get(id);
> + job = job_get_locked(id);
> if (!job) {
> error_setg(errp, "Job not found");
> return NULL;
With this fixed:
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 09/21] jobs: use job locks also in the unit tests
2022-07-25 7:38 ` [PATCH v10 09/21] jobs: use job locks also in the unit tests Emanuele Giuseppe Esposito
2022-07-27 14:29 ` Vladimir Sementsov-Ogievskiy
@ 2022-08-04 11:56 ` Kevin Wolf
1 sibling, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-04 11:56 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> Add missing job synchronization in the unit tests, with
> explicit locks.
>
> We are deliberately using _locked functions wrapped by a guard
> instead of a normal call because the normal call will be removed
> in future, as the only usage is limited to the tests.
>
> In other words, if a function like job_pause() is/will be only used
> in tests to avoid:
>
> WITH_JOB_LOCK_GUARD(){
> job_pause_locked();
> }
>
> then it is not worth keeping job_pause(), and just use the guard.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 10/21] block/mirror.c: use of job helpers in drivers to avoid TOC/TOU
2022-07-25 7:38 ` [PATCH v10 10/21] block/mirror.c: use of job helpers in drivers to avoid TOC/TOU Emanuele Giuseppe Esposito
@ 2022-08-04 16:35 ` Kevin Wolf
2022-08-16 14:23 ` Emanuele Giuseppe Esposito
0 siblings, 1 reply; 71+ messages in thread
From: Kevin Wolf @ 2022-08-04 16:35 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> Once job lock is used and aiocontext is removed, mirror has
> to perform job operations under the same critical section,
> using the helpers prepared in previous commit.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Can you explain in the commit message what the TOC/TOU case is that this
patch is addressing? It's not obvious to me why you picked exactly these
places to add locking.
> diff --git a/block/mirror.c b/block/mirror.c
> index d8ecb9efa2..b38676e19d 100644
> --- a/block/mirror.c
> +++ b/block/mirror.c
> @@ -654,9 +654,13 @@ static int mirror_exit_common(Job *job)
> BlockDriverState *target_bs;
> BlockDriverState *mirror_top_bs;
> Error *local_err = NULL;
> - bool abort = job->ret < 0;
> + bool abort;
> int ret = 0;
>
> + WITH_JOB_LOCK_GUARD() {
> + abort = job->ret < 0;
> + }
This is the most mysterious hunk to me. The only thing that should
modify job->ret is the caller of this function anyway, but let's assume
for a moment that another thread could write to it.
Then why is it only important that we hold the lock when we're reading
the value, but not any more when we are actually using it? And what is
the TOC/TOU that this fixes?
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 11/21] jobs: group together API calls under the same job lock
2022-07-25 7:38 ` [PATCH v10 11/21] jobs: group together API calls under the same job lock Emanuele Giuseppe Esposito
2022-07-27 14:50 ` Vladimir Sementsov-Ogievskiy
@ 2022-08-04 17:10 ` Kevin Wolf
2022-08-16 14:54 ` Emanuele Giuseppe Esposito
1 sibling, 1 reply; 71+ messages in thread
From: Kevin Wolf @ 2022-08-04 17:10 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> Now that the API offers also _locked() functions, take advantage
> of it and give also the caller control to take the lock and call
> _locked functions.
>
> This makes sense especially when we have for loops, because it
> makes no sense to have:
>
> for(job = job_next(); ...)
>
> where each job_next() takes the lock internally.
> Instead we want
>
> JOB_LOCK_GUARD();
> for(job = job_next_locked(); ...)
>
> In addition, protect also direct field accesses, by either creating a
> new critical section or widening the existing ones.
"In addition" sounds like it should be a separate patch. I was indeed
surprised when after a few for loops where you just pulled the existing
locking up a bit, I saw some hunks that add completely new locking.
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> ---
> block.c | 17 ++++++++++-------
> blockdev.c | 12 +++++++++---
> blockjob.c | 35 ++++++++++++++++++++++-------------
> job-qmp.c | 4 +++-
> job.c | 7 +++++--
> monitor/qmp-cmds.c | 7 +++++--
> qemu-img.c | 37 +++++++++++++++++++++----------------
> 7 files changed, 75 insertions(+), 44 deletions(-)
>
> diff --git a/block.c b/block.c
> index 2c00dddd80..7559965dbc 100644
> --- a/block.c
> +++ b/block.c
> @@ -4978,8 +4978,8 @@ static void bdrv_close(BlockDriverState *bs)
>
> void bdrv_close_all(void)
> {
> - assert(job_next(NULL) == NULL);
> GLOBAL_STATE_CODE();
> + assert(job_next(NULL) == NULL);
>
> /* Drop references from requests still in flight, such as canceled block
> * jobs whose AIO context has not been polled yet */
> @@ -6165,13 +6165,16 @@ XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp)
> }
> }
>
> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> - GSList *el;
> + WITH_JOB_LOCK_GUARD() {
> + for (job = block_job_next_locked(NULL); job;
> + job = block_job_next_locked(job)) {
> + GSList *el;
>
> - xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
> - job->job.id);
> - for (el = job->nodes; el; el = el->next) {
> - xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
> + xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
> + job->job.id);
> + for (el = job->nodes; el; el = el->next) {
> + xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
> + }
> }
> }
>
> diff --git a/blockdev.c b/blockdev.c
> index 71f793c4ab..5b79093155 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -150,12 +150,15 @@ void blockdev_mark_auto_del(BlockBackend *blk)
> return;
> }
>
> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> + JOB_LOCK_GUARD();
> +
> + for (job = block_job_next_locked(NULL); job;
> + job = block_job_next_locked(job)) {
> if (block_job_has_bdrv(job, blk_bs(blk))) {
Should this be renamed to block_job_has_bdrv_locked() now?
It looks to me like it does need the locking. (Which actually makes
this patch a fix and not just an optimisation as the commit message
suggests.)
> AioContext *aio_context = job->job.aio_context;
> aio_context_acquire(aio_context);
>
> - job_cancel(&job->job, false);
> + job_cancel_locked(&job->job, false);
>
> aio_context_release(aio_context);
> }
> @@ -3745,7 +3748,10 @@ BlockJobInfoList *qmp_query_block_jobs(Error **errp)
> BlockJobInfoList *head = NULL, **tail = &head;
> BlockJob *job;
>
> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> + JOB_LOCK_GUARD();
> +
> + for (job = block_job_next_locked(NULL); job;
> + job = block_job_next_locked(job)) {
> BlockJobInfo *value;
> AioContext *aio_context;
More context:
BlockJobInfo *value;
AioContext *aio_context;
if (block_job_is_internal(job)) {
continue;
}
aio_context = block_job_get_aio_context(job);
aio_context_acquire(aio_context);
value = block_job_query(job, errp);
aio_context_release(aio_context);
This should become block_job_query_locked(). (You do that in patch 18,
but it looks a bit out of place there - which is precisely because it
really belongs in this one.)
> diff --git a/blockjob.c b/blockjob.c
> index 0d59aba439..96fb9d9f73 100644
> --- a/blockjob.c
> +++ b/blockjob.c
> @@ -111,8 +111,10 @@ static bool child_job_drained_poll(BdrvChild *c)
> /* An inactive or completed job doesn't have any pending requests. Jobs
> * with !job->busy are either already paused or have a pause point after
> * being reentered, so no job driver code will run before they pause. */
> - if (!job->busy || job_is_completed(job)) {
> - return false;
> + WITH_JOB_LOCK_GUARD() {
> + if (!job->busy || job_is_completed_locked(job)) {
> + return false;
> + }
> }
>
> /* Otherwise, assume that it isn't fully stopped yet, but allow the job to
Assuming that the job status can actually change, don't we need the
locking for the rest of the function, too? Otherwise we might call
drv->drained_poll() for a job that has already paused or completed.
Of course, this goes against the assumption that all callbacks are
called without holding the job lock. Maybe it's not a good assumption.
> @@ -475,13 +477,15 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver,
> job->ready_notifier.notify = block_job_event_ready;
> job->idle_notifier.notify = block_job_on_idle;
>
> - notifier_list_add(&job->job.on_finalize_cancelled,
> - &job->finalize_cancelled_notifier);
> - notifier_list_add(&job->job.on_finalize_completed,
> - &job->finalize_completed_notifier);
> - notifier_list_add(&job->job.on_pending, &job->pending_notifier);
> - notifier_list_add(&job->job.on_ready, &job->ready_notifier);
> - notifier_list_add(&job->job.on_idle, &job->idle_notifier);
> + WITH_JOB_LOCK_GUARD() {
> + notifier_list_add(&job->job.on_finalize_cancelled,
> + &job->finalize_cancelled_notifier);
> + notifier_list_add(&job->job.on_finalize_completed,
> + &job->finalize_completed_notifier);
> + notifier_list_add(&job->job.on_pending, &job->pending_notifier);
> + notifier_list_add(&job->job.on_ready, &job->ready_notifier);
> + notifier_list_add(&job->job.on_idle, &job->idle_notifier);
> + }
>
> error_setg(&job->blocker, "block device is in use by block job: %s",
> job_type_str(&job->job));
Why is this the right scope for the lock? It looks very arbitrary to
lock only here, but not for the assignments above or the function calls
below.
Given that job_create() already puts the job in the job_list so it
becomes visible for other code, should we not keep the job lock from the
moment that we create the job until it is fully initialised?
> @@ -558,10 +562,15 @@ BlockErrorAction block_job_error_action(BlockJob *job, BlockdevOnError on_err,
> action);
> }
> if (action == BLOCK_ERROR_ACTION_STOP) {
> - if (!job->job.user_paused) {
> - job_pause(&job->job);
> - /* make the pause user visible, which will be resumed from QMP. */
> - job->job.user_paused = true;
> + WITH_JOB_LOCK_GUARD() {
> + if (!job->job.user_paused) {
> + job_pause_locked(&job->job);
> + /*
> + * make the pause user visible, which will be
> + * resumed from QMP.
> + */
> + job->job.user_paused = true;
> + }
> }
> block_job_iostatus_set_err(job, error);
Why is this call not in the critical section? It accesses job->iostatus.
> }
> diff --git a/job-qmp.c b/job-qmp.c
> index ac11a6c23c..cfaf34ffb7 100644
> --- a/job-qmp.c
> +++ b/job-qmp.c
> @@ -194,7 +194,9 @@ JobInfoList *qmp_query_jobs(Error **errp)
> JobInfoList *head = NULL, **tail = &head;
> Job *job;
>
> - for (job = job_next(NULL); job; job = job_next(job)) {
> + JOB_LOCK_GUARD();
> +
> + for (job = job_next_locked(NULL); job; job = job_next_locked(job)) {
> JobInfo *value;
> AioContext *aio_context;
Should job_query_single() be renamed to job_query_single_locked()?
> diff --git a/job.c b/job.c
> index ebaa4e585b..b0729e2bb2 100644
> --- a/job.c
> +++ b/job.c
> @@ -668,7 +668,7 @@ void coroutine_fn job_pause_point(Job *job)
> job_pause_point_locked(job);
> }
>
> -void job_yield_locked(Job *job)
> +static void job_yield_locked(Job *job)
> {
> assert(job->busy);
It was already unused outside of job.c before this patch. Should it have
been static from the start?
> @@ -1041,11 +1041,14 @@ static void job_completed_txn_abort_locked(Job *job)
> /* Called with job_mutex held, but releases it temporarily */
> static int job_prepare_locked(Job *job)
> {
> + int ret;
> +
> GLOBAL_STATE_CODE();
> if (job->ret == 0 && job->driver->prepare) {
> job_unlock();
> - job->ret = job->driver->prepare(job);
> + ret = job->driver->prepare(job);
> job_lock();
> + job->ret = ret;
> job_update_rc_locked(job);
> }
> return job->ret;
> diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
> index 1ebb89f46c..1897ed7a13 100644
> --- a/monitor/qmp-cmds.c
> +++ b/monitor/qmp-cmds.c
> @@ -133,8 +133,11 @@ void qmp_cont(Error **errp)
> blk_iostatus_reset(blk);
> }
>
> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> - block_job_iostatus_reset(job);
> + WITH_JOB_LOCK_GUARD() {
> + for (job = block_job_next_locked(NULL); job;
> + job = block_job_next_locked(job)) {
> + block_job_iostatus_reset_locked(job);
> + }
> }
>
> /* Continuing after completed migration. Images have been inactivated to
> diff --git a/qemu-img.c b/qemu-img.c
> index 4cf4d2423d..98c7662b0f 100644
> --- a/qemu-img.c
> +++ b/qemu-img.c
> @@ -912,25 +912,30 @@ static void run_block_job(BlockJob *job, Error **errp)
> int ret = 0;
>
> aio_context_acquire(aio_context);
> - job_ref(&job->job);
> - do {
> - float progress = 0.0f;
> - aio_poll(aio_context, true);
> + WITH_JOB_LOCK_GUARD() {
> + job_ref_locked(&job->job);
> + do {
> + float progress = 0.0f;
> + job_unlock();
This might be more a question of style, but mixing WITH_JOB_LOCK_GUARD()
with manual job_unlock()/job_lock() feels dangerous. What if someone
added a break between the unlock/lock pair? The lock guard would try to
unlock a mutex that is already unlocked, which probably means an
assertion failure.
I feel we should just use manual job_lock()/job_unlock() for everything
in this function.
> + aio_poll(aio_context, true);
> +
> + progress_get_snapshot(&job->job.progress, &progress_current,
> + &progress_total);
> + if (progress_total) {
> + progress = (float)progress_current / progress_total * 100.f;
> + }
> + qemu_progress_print(progress, 0);
> + job_lock();
> + } while (!job_is_ready_locked(&job->job) &&
> + !job_is_completed_locked(&job->job));
>
> - progress_get_snapshot(&job->job.progress, &progress_current,
> - &progress_total);
> - if (progress_total) {
> - progress = (float)progress_current / progress_total * 100.f;
> + if (!job_is_completed_locked(&job->job)) {
> + ret = job_complete_sync_locked(&job->job, errp);
> + } else {
> + ret = job->job.ret;
> }
> - qemu_progress_print(progress, 0);
> - } while (!job_is_ready(&job->job) && !job_is_completed(&job->job));
> -
> - if (!job_is_completed(&job->job)) {
> - ret = job_complete_sync(&job->job, errp);
> - } else {
> - ret = job->job.ret;
> + job_unref_locked(&job->job);
> }
> - job_unref(&job->job);
> aio_context_release(aio_context);
>
> /* publish completion progress only when success */
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 12/21] commit and mirror: create new nodes using bdrv_get_aio_context, and not the job aiocontext
2022-07-25 7:38 ` [PATCH v10 12/21] commit and mirror: create new nodes using bdrv_get_aio_context, and not the job aiocontext Emanuele Giuseppe Esposito
@ 2022-08-05 8:14 ` Kevin Wolf
2022-08-16 14:57 ` Emanuele Giuseppe Esposito
0 siblings, 1 reply; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 8:14 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> We are always using the given bs AioContext, so there is no need
> to take the job ones (which is identical anyways).
> This also reduces the point we need to check when protecting
> job.aio_context field.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
I'm not sure against which scenario this is actually protecting. The
only reason for not using job.aio_context seems to be if someone else
can change the job AioContext in parallel. However, if that is the case
(I don't think it is, but for the hypothetical case this patch seems to
address), the AioContext is not identical any more and the change is
wrong because we actually want things to run in the job AioContext -
otherwise accessing the BlockBackend from the job coroutine wouldn't be
possible.
So I believe the current code expresses how we actually want to use the
BlockBackend and the change doesn't only protect nothing, but is even
misleading because it implies that the job can work with any AioContext,
which is not true.
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 13/21] job: detect change of aiocontext within job coroutine
2022-07-25 7:38 ` [PATCH v10 13/21] job: detect change of aiocontext within job coroutine Emanuele Giuseppe Esposito
@ 2022-08-05 8:37 ` Kevin Wolf
2022-08-16 15:09 ` Emanuele Giuseppe Esposito
0 siblings, 1 reply; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 8:37 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> From: Paolo Bonzini <pbonzini@redhat.com>
>
> We want to make sure access of job->aio_context is always done
> under either BQL or job_mutex.
Is this the goal of this series? If so, it would have been useful to
state somewhere more obvious, because I had assumed that holding the BQL
would not be considered enough, but everyone needs to hold the job_mutex.
> The problem is that using
> aio_co_enter(job->aiocontext, job->co) in job_start and job_enter_cond
> makes the coroutine immediately resume, so we can't hold the job lock.
> And caching it is not safe either, as it might change.
>
> job_start is under BQL, so it can freely read job->aiocontext, but
> job_enter_cond is not. In order to fix this, use aio_co_wake():
> the advantage is that it won't use job->aiocontext, but the
> main disadvantage is that it won't be able to detect a change of
> job AioContext.
>
> Calling bdrv_try_set_aio_context() will issue the following calls
> (simplified):
> * in terms of bdrv callbacks:
> .drained_begin -> .set_aio_context -> .drained_end
> * in terms of child_job functions:
> child_job_drained_begin -> child_job_set_aio_context -> child_job_drained_end
> * in terms of job functions:
> job_pause_locked -> job_set_aio_context -> job_resume_locked
>
> We can see that after setting the new aio_context, job_resume_locked
> calls again job_enter_cond, which then invokes aio_co_wake(). But
> while job->aiocontext has been set in job_set_aio_context,
> job->co->ctx has not changed, so the coroutine would be entering in
> the wrong aiocontext.
>
> Using aio_co_schedule in job_resume_locked() might seem as a valid
> alternative, but the problem is that the bh resuming the coroutine
> is not scheduled immediately, and if in the meanwhile another
> bdrv_try_set_aio_context() is run (see test_propagate_mirror() in
> test-block-iothread.c), we would have the first schedule in the
> wrong aiocontext, and the second set of drains won't even manage
> to schedule the coroutine, as job->busy would still be true from
> the previous job_resume_locked().
>
> The solution is to stick with aio_co_wake(), but then detect every time
> the coroutine resumes back from yielding if job->aio_context
> has changed. If so, we can reschedule it to the new context.
Hm, but with this in place, what does aio_co_wake() actually buy us
compared to aio_co_enter()?
I guess it's a bit simpler code because you don't have to explicitly
specify the AioContext, but we're still going to enter the coroutine in
the wrong AioContext occasionally and have to reschedule it, just like
in the existing code (except that the rescheduling doesn't exist there
yet).
So while I don't disagree with the change, I don't think the
justification in the commit message is right for this part.
> Check for the aiocontext change in job_do_yield_locked because:
> 1) aio_co_reschedule_self requires to be in the running coroutine
> 2) since child_job_set_aio_context allows changing the aiocontext only
> while the job is paused, this is the exact place where the coroutine
> resumes, before running JobDriver's code.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> job.c | 21 +++++++++++++++++++--
> 1 file changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/job.c b/job.c
> index b0729e2bb2..ecec66b44e 100644
> --- a/job.c
> +++ b/job.c
> @@ -585,7 +585,9 @@ void job_enter_cond_locked(Job *job, bool(*fn)(Job *job))
> timer_del(&job->sleep_timer);
> job->busy = true;
> real_job_unlock();
> - aio_co_enter(job->aio_context, job->co);
> + job_unlock();
> + aio_co_wake(job->co);
> + job_lock();
The addition of job_unlock/lock is unrelated, this was necessary even
before this patch.
> }
>
> void job_enter_cond(Job *job, bool(*fn)(Job *job))
> @@ -611,6 +613,8 @@ void job_enter(Job *job)
> */
> static void coroutine_fn job_do_yield_locked(Job *job, uint64_t ns)
> {
> + AioContext *next_aio_context;
> +
> real_job_lock();
> if (ns != -1) {
> timer_mod(&job->sleep_timer, ns);
> @@ -622,7 +626,20 @@ static void coroutine_fn job_do_yield_locked(Job *job, uint64_t ns)
> qemu_coroutine_yield();
> job_lock();
>
> - /* Set by job_enter_cond() before re-entering the coroutine. */
> + next_aio_context = job->aio_context;
> + /*
> + * Coroutine has resumed, but in the meanwhile the job AioContext
> + * might have changed via bdrv_try_set_aio_context(), so we need to move
> + * the coroutine too in the new aiocontext.
> + */
> + while (qemu_get_current_aio_context() != next_aio_context) {
> + job_unlock();
> + aio_co_reschedule_self(next_aio_context);
> + job_lock();
> + next_aio_context = job->aio_context;
> + }
> +
> + /* Set by job_enter_cond_locked() before re-entering the coroutine. */
> assert(job->busy);
> }
The actual code changes look good.
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex
2022-07-25 7:38 ` [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex Emanuele Giuseppe Esposito
2022-07-27 15:22 ` Vladimir Sementsov-Ogievskiy
@ 2022-08-05 9:12 ` Kevin Wolf
2022-08-17 8:04 ` Emanuele Giuseppe Esposito
1 sibling, 1 reply; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 9:12 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> In order to make it thread safe, implement a "fake rwlock",
> where we allow reads under BQL *or* job_mutex held, but
> writes only under BQL *and* job_mutex.
Oh, so the "or BQL" part is only for job.aio_context? Okay.
> The only write we have is in child_job_set_aio_ctx, which always
> happens under drain (so the job is paused).
> For this reason, introduce job_set_aio_context and make sure that
> the context is set under BQL, job_mutex and drain.
> Also make sure all other places where the aiocontext is read
> are protected.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> block/replication.c | 6 ++++--
> blockjob.c | 3 ++-
> include/qemu/job.h | 19 ++++++++++++++++++-
> job.c | 12 ++++++++++++
> 4 files changed, 36 insertions(+), 4 deletions(-)
>
> diff --git a/block/replication.c b/block/replication.c
> index 55c8f894aa..2189863df1 100644
> --- a/block/replication.c
> +++ b/block/replication.c
> @@ -148,8 +148,10 @@ static void replication_close(BlockDriverState *bs)
> }
> if (s->stage == BLOCK_REPLICATION_FAILOVER) {
> commit_job = &s->commit_job->job;
> - assert(commit_job->aio_context == qemu_get_current_aio_context());
> - job_cancel_sync(commit_job, false);
> + WITH_JOB_LOCK_GUARD() {
> + assert(commit_job->aio_context == qemu_get_current_aio_context());
> + job_cancel_sync_locked(commit_job, false);
> + }
> }
.bdrv_close runs under the BQL, so why is this needed? Maybe a
GLOBAL_STATE_CODE() annotation would be helpful, though.
> if (s->mode == REPLICATION_MODE_SECONDARY) {
> diff --git a/blockjob.c b/blockjob.c
> index 96fb9d9f73..9ff2727025 100644
> --- a/blockjob.c
> +++ b/blockjob.c
> @@ -162,12 +162,13 @@ static void child_job_set_aio_ctx(BdrvChild *c, AioContext *ctx,
> bdrv_set_aio_context_ignore(sibling->bs, ctx, ignore);
> }
>
> - job->job.aio_context = ctx;
> + job_set_aio_context(&job->job, ctx);
> }
>
> static AioContext *child_job_get_parent_aio_context(BdrvChild *c)
> {
> BlockJob *job = c->opaque;
> + assert(qemu_in_main_thread());
Any reason not to use GLOBAL_STATE_CODE()?
> return job->job.aio_context;
> }
> diff --git a/include/qemu/job.h b/include/qemu/job.h
> index 5709e8d4a8..c144aabefc 100644
> --- a/include/qemu/job.h
> +++ b/include/qemu/job.h
> @@ -77,7 +77,12 @@ typedef struct Job {
>
> /** Protected by AioContext lock */
I think this section comment should move down below aio_context now.
> - /** AioContext to run the job coroutine in */
> + /**
> + * AioContext to run the job coroutine in.
> + * This field can be read when holding either the BQL (so we are in
> + * the main loop) or the job_mutex.
> + * It can be only written when we hold *both* BQL and job_mutex.
> + */
> AioContext *aio_context;
>
> /** Reference count of the block job */
> @@ -741,4 +746,16 @@ int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
> int job_finish_sync_locked(Job *job, void (*finish)(Job *, Error **errp),
> Error **errp);
>
> +/**
> + * Sets the @job->aio_context.
> + * Called with job_mutex *not* held.
> + *
> + * This function must run in the main thread to protect against
> + * concurrent read in job_finish_sync_locked(),
Odd line break here in the middle of a sentence.
> + * takes the job_mutex lock to protect against the read in
> + * job_do_yield_locked(), and must be called when the coroutine
> + * is quiescent.
> + */
> +void job_set_aio_context(Job *job, AioContext *ctx);
> +
> #endif
> diff --git a/job.c b/job.c
> index ecec66b44e..0a857b1468 100644
> --- a/job.c
> +++ b/job.c
> @@ -394,6 +394,17 @@ Job *job_get(const char *id)
> return job_get_locked(id);
> }
>
> +void job_set_aio_context(Job *job, AioContext *ctx)
> +{
> + /* protect against read in job_finish_sync_locked and job_start */
> + assert(qemu_in_main_thread());
Same question about GLOBAL_STATE_CODE().
> + /* protect against read in job_do_yield_locked */
> + JOB_LOCK_GUARD();
> + /* ensure the coroutine is quiescent while the AioContext is changed */
> + assert(job->pause_count > 0);
job->pause_count only shows that pausing was requested. The coroutine is
only really quiescent if job->busy == false, too.
Or maybe job->paused is actually the one you want here.
> + job->aio_context = ctx;
> +}
> +
> /* Called with job_mutex *not* held. */
> static void job_sleep_timer_cb(void *opaque)
> {
> @@ -1376,6 +1387,7 @@ int job_finish_sync_locked(Job *job,
> {
> Error *local_err = NULL;
> int ret;
> + assert(qemu_in_main_thread());
>
> job_ref_locked(job);
Another GLOBAL_STATE_CODE()?
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 15/21] blockjob.h: categorize fields in struct BlockJob
2022-07-25 7:38 ` [PATCH v10 15/21] blockjob.h: categorize fields in struct BlockJob Emanuele Giuseppe Esposito
@ 2022-08-05 9:21 ` Kevin Wolf
0 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 9:21 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> The same job lock is being used also to protect some of blockjob fields.
> Categorize them just as done in job.h.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Why do speed, limit and blocker stay uncategorised? I think we should
document the locking rules for all fields.
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 16/21] blockjob: rename notifier callbacks as _locked
2022-07-25 7:38 ` [PATCH v10 16/21] blockjob: rename notifier callbacks as _locked Emanuele Giuseppe Esposito
@ 2022-08-05 9:25 ` Kevin Wolf
0 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 9:25 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> They all are called with job_lock held, in job_event_*_locked()
Worth documenting that for the notifier lists in struct Job, too?
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 17/21] blockjob: protect iostatus field in BlockJob struct
2022-07-25 7:38 ` [PATCH v10 17/21] blockjob: protect iostatus field in BlockJob struct Emanuele Giuseppe Esposito
2022-07-27 15:29 ` Vladimir Sementsov-Ogievskiy
@ 2022-08-05 10:55 ` Kevin Wolf
1 sibling, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 10:55 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> iostatus is the only field (together with .job) that needs
> protection using the job mutex.
>
> It is set in the main loop (GLOBAL_STATE functions) but read
> in I/O code (block_job_error_action).
>
> In order to protect it, change block_job_iostatus_set_err
> to block_job_iostatus_set_err_locked(), always called under
> job lock.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> ---
> blockjob.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/blockjob.c b/blockjob.c
> index 0663faee2c..448bdb5a53 100644
> --- a/blockjob.c
> +++ b/blockjob.c
> @@ -363,7 +363,8 @@ BlockJobInfo *block_job_query(BlockJob *job, Error **errp)
> return block_job_query_locked(job, errp);
> }
>
> -static void block_job_iostatus_set_err(BlockJob *job, int error)
> +/* Called with job lock held */
> +static void block_job_iostatus_set_err_locked(BlockJob *job, int error)
> {
> if (job->iostatus == BLOCK_DEVICE_IO_STATUS_OK) {
> job->iostatus = error == ENOSPC ? BLOCK_DEVICE_IO_STATUS_NOSPACE :
> @@ -577,8 +578,8 @@ BlockErrorAction block_job_error_action(BlockJob *job, BlockdevOnError on_err,
> */
> job->job.user_paused = true;
> }
> + block_job_iostatus_set_err_locked(job, error);
> }
> - block_job_iostatus_set_err(job, error);
> }
> return action;
> }
Ah, so this patch does what I asked for in an earlier patch. I wonder if
it should be squashed there.
Vladimir's finding that we have an access in mirror may need a fix, but
while incomplete, this patch isn't wrong:
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks
2022-07-25 7:38 ` [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks Emanuele Giuseppe Esposito
2022-07-27 15:53 ` Vladimir Sementsov-Ogievskiy
@ 2022-08-05 13:01 ` Kevin Wolf
2022-08-17 12:45 ` Emanuele Giuseppe Esposito
1 sibling, 1 reply; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 13:01 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> Change the job_{lock/unlock} and macros to use job_mutex.
>
> Now that they are not nop anymore, remove the aiocontext
> to avoid deadlocks.
Okay, so this is the big bad patch where we need to verify the
completeness of all changes made so far. Let's see...
> Therefore:
> - when possible, remove completely the aiocontext lock/unlock pair
> - if it is used by some other function too, reduce the locking
> section as much as possible, leaving the job API outside.
> - change AIO_WAIT_WHILE in AIO_WAIT_WHILE_UNLOCKED, since we
> are not using the aiocontext lock anymore
Does this imply that there is a new rule that job_*() must not be called
with the AioContext lock held? Or is it optional now?
If it's a rule, it should be documented.
(Coming back after reviewing more of the patch:)
It doesn't seem to be a rule, or at least not a rule that is obeyed.
Actually each function in job.c belongs in one of at most three
categories: Must hold the AioContext (because it calls callbacks that
need it), may hold the AioContext optionally, and must not hold it
(everything that would cause the deadlocks you're alluding to, but not
explaining, in the commit message).
It is not obvious which function is in which category. So I maintain
that we need some documentation for the assumptions made.
All coroutine_fns (which are called from the job coroutine) should be in
the category that they still run with the AioContext locked, but that
covers only a small minority of functions.
The driver callbacks look mostly correct at least with respect to
AioContext locking, even if their status isn't documented. I suppose
this is the most important part.
> There is only one JobDriver callback, ->free() that assumes that
> the aiocontext lock is held (because it calls bdrv_unref), so for
> now keep that under aiocontext lock.
>
> Also remove real_job_{lock/unlock}, as they are replaced by the
> public functions.
At least this part is easily verified. All real_job_lock() sections are
part of a larger job_lock() section.
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> blockdev.c | 74 +++++-----------------------
> include/qemu/job.h | 22 ++++-----
> job-qmp.c | 46 +++--------------
> job.c | 84 ++++++--------------------------
> tests/unit/test-bdrv-drain.c | 4 +-
> tests/unit/test-block-iothread.c | 2 +-
> tests/unit/test-blockjob.c | 15 ++----
> 7 files changed, 52 insertions(+), 195 deletions(-)
> diff --git a/include/qemu/job.h b/include/qemu/job.h
> index c144aabefc..676f69bb2e 100644
> --- a/include/qemu/job.h
> +++ b/include/qemu/job.h
> @@ -75,13 +75,14 @@ typedef struct Job {
> ProgressMeter progress;
>
>
> - /** Protected by AioContext lock */
> + /** Protected by job_mutex */
>
> /**
> * AioContext to run the job coroutine in.
> - * This field can be read when holding either the BQL (so we are in
> - * the main loop) or the job_mutex.
> - * It can be only written when we hold *both* BQL and job_mutex.
> + * The job Aiocontext can be read when holding *either*
> + * the BQL (so we are in the main loop) or the job_mutex.
> + * It can only be written when we hold *both* BQL
> + * and the job_mutex.
Old and new version seem to say the same apart from stylistic
differences, so this feels like unnecessary churn.
> */
> AioContext *aio_context;
>
> @@ -106,7 +107,7 @@ typedef struct Job {
> /**
> * Set to false by the job while the coroutine has yielded and may be
> * re-entered by job_enter(). There may still be I/O or event loop activity
> - * pending. Accessed under block_job_mutex (in blockjob.c).
> + * pending. Accessed under job_mutex.
> *
> * When the job is deferred to the main loop, busy is true as long as the
> * bottom half is still pending.
> @@ -322,9 +323,9 @@ typedef enum JobCreateFlags {
>
> extern QemuMutex job_mutex;
>
> -#define JOB_LOCK_GUARD() /* QEMU_LOCK_GUARD(&job_mutex) */
> +#define JOB_LOCK_GUARD() QEMU_LOCK_GUARD(&job_mutex)
>
> -#define WITH_JOB_LOCK_GUARD() /* WITH_QEMU_LOCK_GUARD(&job_mutex) */
> +#define WITH_JOB_LOCK_GUARD() WITH_QEMU_LOCK_GUARD(&job_mutex)
>
> /**
> * job_lock:
> @@ -672,7 +673,7 @@ void job_user_cancel_locked(Job *job, bool force, Error **errp);
> * Returns the return value from the job if the job actually completed
> * during the call, or -ECANCELED if it was canceled.
> *
> - * Callers must hold the AioContext lock of job->aio_context.
> + * Called with job_lock held.
> */
> int job_cancel_sync(Job *job, bool force);
This doesn't make sense, it describes job_cancel_sync_locked().
>
> @@ -697,8 +698,7 @@ void job_cancel_sync_all(void);
> * function).
> *
> * Returns the return value from the job.
> - *
> - * Callers must hold the AioContext lock of job->aio_context.
> + * Called with job_lock held.
Same.
> */
> int job_complete_sync(Job *job, Error **errp);
>
> @@ -734,7 +734,7 @@ void job_dismiss_locked(Job **job, Error **errp);
> * Returns 0 if the job is successfully completed, -ECANCELED if the job was
> * cancelled before completing, and -errno in other error cases.
> *
> - * Callers must hold the AioContext lock of job->aio_context.
> + * Called with job_lock held.
Same.
> */
> int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp),
> Error **errp);
> diff --git a/job.c b/job.c
> index 0a857b1468..9797b934d9 100644
> --- a/job.c
> +++ b/job.c
> @@ -96,21 +96,11 @@ struct JobTxn {
> };
>
> void job_lock(void)
> -{
> - /* nop */
> -}
> -
> -void job_unlock(void)
> -{
> - /* nop */
> -}
> -
> -static void real_job_lock(void)
> {
> qemu_mutex_lock(&job_mutex);
> }
>
> -static void real_job_unlock(void)
> +void job_unlock(void)
> {
> qemu_mutex_unlock(&job_mutex);
> }
> @@ -185,7 +175,6 @@ static void job_txn_del_job_locked(Job *job)
> /* Called with job_mutex held, but releases it temporarily. */
> static int job_txn_apply_locked(Job *job, int fn(Job *))
> {
> - AioContext *inner_ctx;
> Job *other_job, *next;
> JobTxn *txn = job->txn;
> int rc = 0;
> @@ -197,23 +186,14 @@ static int job_txn_apply_locked(Job *job, int fn(Job *))
> * break AIO_WAIT_WHILE from within fn.
> */
> job_ref_locked(job);
> - aio_context_release(job->aio_context);
>
> QLIST_FOREACH_SAFE(other_job, &txn->jobs, txn_list, next) {
> - inner_ctx = other_job->aio_context;
> - aio_context_acquire(inner_ctx);
> rc = fn(other_job);
> - aio_context_release(inner_ctx);
Okay, so fn() is now called without the AioContext lock while it was
called with it previously. This requires checking all callers.
What isn't immediately clear, but seems to be true, is that all notifiers
don't need the AioContext lock. Probably worth documenting in struct
Job. (Needed because of job_transition_to_pending_locked(), which is
passed as fn.)
> if (rc) {
> break;
> }
> }
>
> - /*
> - * Note that job->aio_context might have been changed by calling fn, so we
> - * can't use a local variable to cache it.
> - */
> - aio_context_acquire(job->aio_context);
> job_unref_locked(job);
> return rc;
> }
> @@ -501,8 +481,12 @@ void job_unref_locked(Job *job)
> assert(!job->txn);
>
> if (job->driver->free) {
> + AioContext *aio_context = job->aio_context;
> job_unlock();
> + /* FIXME: aiocontext lock is required because cb calls blk_unref */
> + aio_context_acquire(aio_context);
> job->driver->free(job);
> + aio_context_release(aio_context);
The documentation of JobDriver doesn't specify which callbacks are
called with the AioContext locked and which are called without it. It
probably should.
(A good part of the documentation clarifications I'm asking for in this
review should probably done in a patch before this one, so that
reviewing the documentation involves checking that the requirements of
the callback match what we're documenting, and then the review of this
patch can focus on that the documented contract is still obeyed.)
> job_lock();
> }
>
> @@ -922,6 +900,7 @@ static void job_clean(Job *job)
> static int job_finalize_single_locked(Job *job)
> {
> int job_ret;
> + AioContext *ctx = job->aio_context;
>
> assert(job_is_completed_locked(job));
>
> @@ -929,6 +908,7 @@ static int job_finalize_single_locked(Job *job)
> job_update_rc_locked(job);
>
> job_unlock();
> + aio_context_acquire(ctx);
>
> if (!job->ret) {
> job_commit(job);
> @@ -937,6 +917,7 @@ static int job_finalize_single_locked(Job *job)
> }
> job_clean(job);
>
> + aio_context_release(ctx);
> job_lock();
Let's add comments to job_commit(), job_abort() and job_clean() that
they are called with the AioContext lock held.
A few lines below we are now calling job->cb() without the AioContext
lock even though previously it was called with it. Which way is right?
The intended behaviour should be documented in struct Job.
> if (job->cb) {
> @@ -1002,7 +983,6 @@ static void job_cancel_async_locked(Job *job, bool force)
> /* Called with job_mutex held, but releases it temporarily. */
> static void job_completed_txn_abort_locked(Job *job)
> {
> - AioContext *ctx;
> JobTxn *txn = job->txn;
> Job *other_job;
>
> @@ -1015,54 +995,31 @@ static void job_completed_txn_abort_locked(Job *job)
> txn->aborting = true;
> job_txn_ref_locked(txn);
>
> - /*
> - * We can only hold the single job's AioContext lock while calling
> - * job_finalize_single() because the finalization callbacks can involve
> - * calls of AIO_WAIT_WHILE(), which could deadlock otherwise.
> - * Note that the job's AioContext may change when it is finalized.
> - */
> job_ref_locked(job);
> - aio_context_release(job->aio_context);
>
> /* Other jobs are effectively cancelled by us, set the status for
> * them; this job, however, may or may not be cancelled, depending
> * on the caller, so leave it. */
> QLIST_FOREACH(other_job, &txn->jobs, txn_list) {
> if (other_job != job) {
> - ctx = other_job->aio_context;
> - aio_context_acquire(ctx);
> /*
> * This is a transaction: If one job failed, no result will matter.
> * Therefore, pass force=true to terminate all other jobs as quickly
> * as possible.
> */
> job_cancel_async_locked(other_job, true);
job_cancel_async_locked() calls job->driver->cancel() without the
AioContext lock now. Some implementations call bdrv_cancel_in_flight().
Generally bdrv_*() are called with the AioContext lock held.
If we want to make bdrv_cancel_in_flight() an exception, at the very
least this need to be documented.
The more obvious solution would be to acquire the AioContext lock in
job_cancel_async_locked() around the callback.
> - aio_context_release(ctx);
> }
> }
> while (!QLIST_EMPTY(&txn->jobs)) {
> other_job = QLIST_FIRST(&txn->jobs);
> - /*
> - * The job's AioContext may change, so store it in @ctx so we
> - * release the same context that we have acquired before.
> - */
> - ctx = other_job->aio_context;
> - aio_context_acquire(ctx);
> if (!job_is_completed_locked(other_job)) {
> assert(job_cancel_requested_locked(other_job));
> job_finish_sync_locked(other_job, NULL, NULL);
> }
> job_finalize_single_locked(other_job);
> - aio_context_release(ctx);
> }
>
> - /*
> - * Use job_ref()/job_unref() so we can read the AioContext here
> - * even if the job went away during job_finalize_single().
> - */
> - aio_context_acquire(job->aio_context);
> job_unref_locked(job);
> -
> job_txn_unref_locked(txn);
> }
>
> @@ -1070,15 +1027,20 @@ static void job_completed_txn_abort_locked(Job *job)
> static int job_prepare_locked(Job *job)
> {
> int ret;
> + AioContext *ctx = job->aio_context;
>
> GLOBAL_STATE_CODE();
> +
> if (job->ret == 0 && job->driver->prepare) {
> job_unlock();
> + aio_context_acquire(ctx);
> ret = job->driver->prepare(job);
> + aio_context_release(ctx);
> job_lock();
> job->ret = ret;
> job_update_rc_locked(job);
> }
> +
> return job->ret;
> }
>
> @@ -1183,11 +1145,8 @@ static void job_completed_locked(Job *job)
> static void job_exit(void *opaque)
> {
> Job *job = (Job *)opaque;
> - AioContext *ctx;
> JOB_LOCK_GUARD();
> -
> job_ref_locked(job);
> - aio_context_acquire(job->aio_context);
>
> /* This is a lie, we're not quiescent, but still doing the completion
> * callbacks. However, completion callbacks tend to involve operations that
> @@ -1197,16 +1156,7 @@ static void job_exit(void *opaque)
> job_event_idle_locked(job);
>
> job_completed_locked(job);
> -
> - /*
> - * Note that calling job_completed can move the job to a different
> - * aio_context, so we cannot cache from above. job_txn_apply takes care of
> - * acquiring the new lock, and we ref/unref to avoid job_completed freeing
> - * the job underneath us.
> - */
> - ctx = job->aio_context;
> job_unref_locked(job);
> - aio_context_release(ctx);
> }
>
> /**
> @@ -1334,14 +1284,10 @@ int job_cancel_sync(Job *job, bool force)
> void job_cancel_sync_all(void)
> {
> Job *job;
> - AioContext *aio_context;
> JOB_LOCK_GUARD();
>
> while ((job = job_next_locked(NULL))) {
> - aio_context = job->aio_context;
> - aio_context_acquire(aio_context);
> job_cancel_sync_locked(job, true);
> - aio_context_release(aio_context);
> }
> }
>
> @@ -1401,8 +1347,8 @@ int job_finish_sync_locked(Job *job,
> }
>
> job_unlock();
> - AIO_WAIT_WHILE(job->aio_context,
> - (job_enter(job), !job_is_completed(job)));
> + AIO_WAIT_WHILE_UNLOCKED(job->aio_context,
> + (job_enter(job), !job_is_completed(job)));
> job_lock();
>
> ret = (job_is_cancelled_locked(job) && job->ret == 0)
> diff --git a/tests/unit/test-bdrv-drain.c b/tests/unit/test-bdrv-drain.c
> index 0db056ea63..4924ceb562 100644
> --- a/tests/unit/test-bdrv-drain.c
> +++ b/tests/unit/test-bdrv-drain.c
> @@ -930,9 +930,9 @@ static void test_blockjob_common_drain_node(enum drain_type drain_type,
> tjob->prepare_ret = -EIO;
> break;
> }
> + aio_context_release(ctx);
>
> job_start(&job->job);
> - aio_context_release(ctx);
>
> if (use_iothread) {
> /* job_co_entry() is run in the I/O thread, wait for the actual job
> @@ -1016,12 +1016,12 @@ static void test_blockjob_common_drain_node(enum drain_type drain_type,
> g_assert_true(job->job.busy); /* We're in qemu_co_sleep_ns() */
> }
>
> - aio_context_acquire(ctx);
> WITH_JOB_LOCK_GUARD() {
> ret = job_complete_sync_locked(&job->job, &error_abort);
> }
> g_assert_cmpint(ret, ==, (result == TEST_JOB_SUCCESS ? 0 : -EIO));
>
> + aio_context_acquire(ctx);
> if (use_iothread) {
> blk_set_aio_context(blk_src, qemu_get_aio_context(), &error_abort);
> assert(blk_get_aio_context(blk_target) == qemu_get_aio_context());
> diff --git a/tests/unit/test-block-iothread.c b/tests/unit/test-block-iothread.c
> index 89e7f0fffb..9d7c8be00f 100644
> --- a/tests/unit/test-block-iothread.c
> +++ b/tests/unit/test-block-iothread.c
> @@ -455,10 +455,10 @@ static void test_attach_blockjob(void)
> aio_poll(qemu_get_aio_context(), false);
> }
>
> - aio_context_acquire(ctx);
> WITH_JOB_LOCK_GUARD() {
> job_complete_sync_locked(&tjob->common.job, &error_abort);
> }
> + aio_context_acquire(ctx);
> blk_set_aio_context(blk, qemu_get_aio_context(), &error_abort);
> aio_context_release(ctx);
>
> diff --git a/tests/unit/test-blockjob.c b/tests/unit/test-blockjob.c
> index b0cd06c529..8a9350078f 100644
> --- a/tests/unit/test-blockjob.c
> +++ b/tests/unit/test-blockjob.c
> @@ -228,10 +228,6 @@ static void cancel_common(CancelJob *s)
> BlockJob *job = &s->common;
> BlockBackend *blk = s->blk;
> JobStatus sts = job->job.status;
> - AioContext *ctx;
> -
> - ctx = job->job.aio_context;
> - aio_context_acquire(ctx);
>
> job_cancel_sync(&job->job, true);
> WITH_JOB_LOCK_GUARD() {
> @@ -244,7 +240,6 @@ static void cancel_common(CancelJob *s)
> }
> destroy_blk(blk);
>
> - aio_context_release(ctx);
destroy_blk() requires the AioContext to be locked.
> }
>
> static void test_cancel_created(void)
> @@ -384,12 +379,10 @@ static void test_cancel_concluded(void)
> aio_poll(qemu_get_aio_context(), true);
> assert_job_status_is(job, JOB_STATUS_PENDING);
>
> - aio_context_acquire(job->aio_context);
> WITH_JOB_LOCK_GUARD() {
> job_finalize_locked(job, &error_abort);
> + assert(job->status == JOB_STATUS_CONCLUDED);
> }
> - aio_context_release(job->aio_context);
> - assert_job_status_is(job, JOB_STATUS_CONCLUDED);
>
> cancel_common(s);
> }
> @@ -482,13 +475,11 @@ static void test_complete_in_standby(void)
>
> /* Wait for the job to become READY */
> job_start(job);
> - aio_context_acquire(ctx);
> /*
> * Here we are waiting for the status to change, so don't bother
> * protecting the read every time.
> */
> - AIO_WAIT_WHILE(ctx, job->status != JOB_STATUS_READY);
> - aio_context_release(ctx);
> + AIO_WAIT_WHILE_UNLOCKED(ctx, job->status != JOB_STATUS_READY);
>
> /* Begin the drained section, pausing the job */
> bdrv_drain_all_begin();
> @@ -507,6 +498,7 @@ static void test_complete_in_standby(void)
> job_complete_locked(job, &error_abort);
>
> /* The test is done now, clean up. */
> + aio_context_release(ctx);
job_complete_locked() is not supposed to be called with the AioContext
locked (otherwise blockdev.c would be wrong).
> job_finish_sync_locked(job, NULL, &error_abort);
> assert(job->status == JOB_STATUS_PENDING);
>
> @@ -516,6 +508,7 @@ static void test_complete_in_standby(void)
> job_dismiss_locked(&job, &error_abort);
> }
>
> + aio_context_acquire(ctx);
> destroy_blk(blk);
> aio_context_release(ctx);
> iothread_join(iothread);
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 19/21] block_job_query: remove atomic read
2022-07-25 7:38 ` [PATCH v10 19/21] block_job_query: remove atomic read Emanuele Giuseppe Esposito
@ 2022-08-05 13:01 ` Kevin Wolf
0 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 13:01 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> Not sure what the atomic here was supposed to do, since job.busy
> is protected by the job lock. Since the whole function
> is called under job_mutex, just remove the atomic.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 20/21] blockjob: remove unused functions
2022-07-25 7:38 ` [PATCH v10 20/21] blockjob: remove unused functions Emanuele Giuseppe Esposito
@ 2022-08-05 13:05 ` Kevin Wolf
0 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 13:05 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> These public functions are not used anywhere, thus can be dropped.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> @@ -113,6 +111,7 @@ BlockJob *block_job_next_locked(BlockJob *job);
> * Get the block job identified by @id (which must not be %NULL).
> *
> * Returns the requested job, or %NULL if it doesn't exist.
> + * Called with job lock *not* held.
> */
> BlockJob *block_job_get(const char *id);
Unrelated hunk. Whatever:
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 21/21] job: remove unused functions
2022-07-25 7:38 ` [PATCH v10 21/21] job: " Emanuele Giuseppe Esposito
@ 2022-08-05 13:09 ` Kevin Wolf
0 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-05 13:09 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> These public functions are not used anywhere, thus can be dropped.
> Also, since this is the final job API that doesn't use AioContext
> lock and replaces it with job_lock, adjust all remaining function
> documentation to clearly specify if the job lock is taken or not.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
You're also documenting the locking requirements for a few functions
where you don't remove a second version.
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 17/21] blockjob: protect iostatus field in BlockJob struct
2022-07-27 15:29 ` Vladimir Sementsov-Ogievskiy
@ 2022-08-16 12:39 ` Emanuele Giuseppe Esposito
0 siblings, 0 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-16 12:39 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 27/07/2022 um 17:29 schrieb Vladimir Sementsov-Ogievskiy:
> On 7/25/22 10:38, Emanuele Giuseppe Esposito wrote:
>> iostatus is the only field (together with .job) that needs
>> protection using the job mutex.
>>
>> It is set in the main loop (GLOBAL_STATE functions) but read
>> in I/O code (block_job_error_action).
>
> Hmm, block_job_error_action doesn't read iostatus..
block_job_error_action -> block_job_iostatus_set_err_locked -> reads
iostatus
>
> Also, iostatus is read by by mirror_run, which is not protected.
Will do, thanks
Emanuele
>>
>> In order to protect it, change block_job_iostatus_set_err
>> to block_job_iostatus_set_err_locked(), always called under
>> job lock.
>>
>> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
>> ---
>> blockjob.c | 5 +++--
>> 1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> diff --git a/blockjob.c b/blockjob.c
>> index 0663faee2c..448bdb5a53 100644
>> --- a/blockjob.c
>> +++ b/blockjob.c
>> @@ -363,7 +363,8 @@ BlockJobInfo *block_job_query(BlockJob *job, Error
>> **errp)
>> return block_job_query_locked(job, errp);
>> }
>> -static void block_job_iostatus_set_err(BlockJob *job, int error)
>> +/* Called with job lock held */
>> +static void block_job_iostatus_set_err_locked(BlockJob *job, int error)
>> {
>> if (job->iostatus == BLOCK_DEVICE_IO_STATUS_OK) {
>> job->iostatus = error == ENOSPC ?
>> BLOCK_DEVICE_IO_STATUS_NOSPACE :
>> @@ -577,8 +578,8 @@ BlockErrorAction block_job_error_action(BlockJob
>> *job, BlockdevOnError on_err,
>> */
>> job->job.user_paused = true;
>> }
>> + block_job_iostatus_set_err_locked(job, error);
>> }
>> - block_job_iostatus_set_err(job, error);
>> }
>> return action;
>> }
>
>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks
2022-07-27 15:53 ` Vladimir Sementsov-Ogievskiy
@ 2022-08-16 12:52 ` Emanuele Giuseppe Esposito
2022-08-17 18:54 ` Vladimir Sementsov-Ogievskiy
2022-08-16 12:53 ` Emanuele Giuseppe Esposito
1 sibling, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-16 12:52 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
>
>> }
>> @@ -501,8 +481,12 @@ void job_unref_locked(Job *job)>
>> assert(!job->txn);
>> if (job->driver->free) {
>> + AioContext *aio_context = job->aio_context;
>> job_unlock();
>> + /* FIXME: aiocontext lock is required because cb calls
>> blk_unref */
>> + aio_context_acquire(aio_context);
>> job->driver->free(job);
>> + aio_context_release(aio_context);
>
> So, job_unref_locked() must be called with aio_context not locked,
> otherwise we dead-lock here? That should be documented in function
> declaration comment.
>
> For example in qemu-img.c in run_block_job() we do exactly that: call
> job_unref_locked() inside aio-context lock critical seaction..
No, job_unref_locked has also status and refcnt and all the other fields
that need to be protectd. Only free must be called without job lock held.
>
>
>> job_lock();
>> }
>> @@ -581,21 +565,17 @@ void job_enter_cond_locked(Job *job,
>> bool(*fn)(Job *job))
>> return;
>> }
>> - real_job_lock();
>> if (job->busy) {
>> - real_job_unlock();
>> return;
>> }
>> if (fn && !fn(job)) {
>> - real_job_unlock();
>> return;
>> }
>> assert(!job->deferred_to_main_loop);
>> timer_del(&job->sleep_timer);
>> job->busy = true;
>> - real_job_unlock();
>> job_unlock();
>> aio_co_wake(job->co);
>> job_lock();
>> @@ -626,13 +606,11 @@ static void coroutine_fn job_do_yield_locked(Job
>> *job, uint64_t ns)
>> {
>> AioContext *next_aio_context;
>> - real_job_lock();
>> if (ns != -1) {
>> timer_mod(&job->sleep_timer, ns);
>> }
>> job->busy = false;
>> job_event_idle_locked(job);
>> - real_job_unlock();
>> job_unlock();
>> qemu_coroutine_yield();
>> job_lock();
>> @@ -922,6 +900,7 @@ static void job_clean(Job *job)
>> static int job_finalize_single_locked(Job *job)
>> {
>> int job_ret;
>> + AioContext *ctx = job->aio_context;
>> assert(job_is_completed_locked(job));
>> @@ -929,6 +908,7 @@ static int job_finalize_single_locked(Job *job)
>> job_update_rc_locked(job);
>> job_unlock();
>> + aio_context_acquire(ctx);
>
> Hmm, and this function and all its callers now should be called with
> aio-context lock not locked?
Why not leave it as it is?
>
> For example job_exit is scheduled as as BH. Aren't BHs called with
> aio-context lock held?
I see no aiocontext call in aio_bh_schedule_oneshot and callees...
So summing up, no, I don't think I will apply your suggestions for this
patch here (assume the opposite for all the others).
Emanuele
>
>> if (!job->ret) {
>> job_commit(job);
>> @@ -937,6 +917,7 @@ static int job_finalize_single_locked(Job *job)
>> }
>> job_clean(job);
>> + aio_context_release(ctx);
>> job_lock();
>> if (job->cb) {
>
> [..]
>
>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks
2022-07-27 15:53 ` Vladimir Sementsov-Ogievskiy
2022-08-16 12:52 ` Emanuele Giuseppe Esposito
@ 2022-08-16 12:53 ` Emanuele Giuseppe Esposito
1 sibling, 0 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-16 12:53 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 27/07/2022 um 17:53 schrieb Vladimir Sementsov-Ogievskiy:
>> * job_lock:
>> @@ -672,7 +673,7 @@ void job_user_cancel_locked(Job *job, bool force,
>> Error **errp);
>> * Returns the return value from the job if the job actually completed
>> * during the call, or -ECANCELED if it was canceled.
>> *
>> - * Callers must hold the AioContext lock of job->aio_context.
>> + * Called with job_lock held.
>
> That's wrong, it should be called with job_lock not held :)
>
>> */
>> int job_cancel_sync(Job *job, bool force);
>> @@ -697,8 +698,7 @@ void job_cancel_sync_all(void);
>> * function).
>> *
>> * Returns the return value from the job.
>> - *
>> - * Callers must hold the AioContext lock of job->aio_context.
>> + * Called with job_lock held.
>
> and this,
>
>> */
>> int job_complete_sync(Job *job, Error **errp);
>> @@ -734,7 +734,7 @@ void job_dismiss_locked(Job **job, Error **errp);
>> * Returns 0 if the job is successfully completed, -ECANCELED if the
>> job was
>> * cancelled before completing, and -errno in other error cases.
>> *
>> - * Callers must hold the AioContext lock of job->aio_context.
>> + * Called with job_lock held.
>
> and this.
Well, except for this part here :) You're right here.
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 10/21] block/mirror.c: use of job helpers in drivers to avoid TOC/TOU
2022-08-04 16:35 ` Kevin Wolf
@ 2022-08-16 14:23 ` Emanuele Giuseppe Esposito
0 siblings, 0 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-16 14:23 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 04/08/2022 um 18:35 schrieb Kevin Wolf:
> Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
>> Once job lock is used and aiocontext is removed, mirror has
>> to perform job operations under the same critical section,
>> using the helpers prepared in previous commit.
>>
>> Note: at this stage, job_{lock/unlock} and job lock guard macros
>> are *nop*.
>>
>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
>
> Can you explain in the commit message what the TOC/TOU case is that this
> patch is addressing? It's not obvious to me why you picked exactly these
> places to add locking.
Well after thinking about it, mentioning TOC/TOU doesn't really make
sense here. I'll remove the "to avoid TOC/TOU" in the title.
>
>> diff --git a/block/mirror.c b/block/mirror.c
>> index d8ecb9efa2..b38676e19d 100644
>> --- a/block/mirror.c
>> +++ b/block/mirror.c
>> @@ -654,9 +654,13 @@ static int mirror_exit_common(Job *job)
>> BlockDriverState *target_bs;
>> BlockDriverState *mirror_top_bs;
>> Error *local_err = NULL;
>> - bool abort = job->ret < 0;
>> + bool abort;
>> int ret = 0;
>>
>> + WITH_JOB_LOCK_GUARD() {
>> + abort = job->ret < 0;
>> + }
>
> This is the most mysterious hunk to me. The only thing that should
> modify job->ret is the caller of this function anyway, but let's assume
> for a moment that another thread could write to it.
>
> Then why is it only important that we hold the lock when we're reading
> the value, but not any more when we are actually using it? And what is
> the TOC/TOU that this fixes?
No TOC/TOU, no sense for this fix too. I'll remove this hunk too.
Emanuele
>
> Kevin
>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 11/21] jobs: group together API calls under the same job lock
2022-08-04 17:10 ` Kevin Wolf
@ 2022-08-16 14:54 ` Emanuele Giuseppe Esposito
2022-08-17 8:46 ` Kevin Wolf
0 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-16 14:54 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 04/08/2022 um 19:10 schrieb Kevin Wolf:
> Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
>> Now that the API offers also _locked() functions, take advantage
>> of it and give also the caller control to take the lock and call
>> _locked functions.
>>
>> This makes sense especially when we have for loops, because it
>> makes no sense to have:
>>
>> for(job = job_next(); ...)
>>
>> where each job_next() takes the lock internally.
>> Instead we want
>>
>> JOB_LOCK_GUARD();
>> for(job = job_next_locked(); ...)
>>
>> In addition, protect also direct field accesses, by either creating a
>> new critical section or widening the existing ones.
>
> "In addition" sounds like it should be a separate patch. I was indeed
> surprised when after a few for loops where you just pulled the existing
> locking up a bit, I saw some hunks that add completely new locking.
Would it be okay if we don't split it in two? There would be two
microscopical patches.
>
>> Note: at this stage, job_{lock/unlock} and job lock guard macros
>> are *nop*.
>>
>> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
>> ---
>> block.c | 17 ++++++++++-------
>> blockdev.c | 12 +++++++++---
>> blockjob.c | 35 ++++++++++++++++++++++-------------
>> job-qmp.c | 4 +++-
>> job.c | 7 +++++--
>> monitor/qmp-cmds.c | 7 +++++--
>> qemu-img.c | 37 +++++++++++++++++++++----------------
>> 7 files changed, 75 insertions(+), 44 deletions(-)
>>
>> diff --git a/block.c b/block.c
>> index 2c00dddd80..7559965dbc 100644
>> --- a/block.c
>> +++ b/block.c
>> @@ -4978,8 +4978,8 @@ static void bdrv_close(BlockDriverState *bs)
>>
>> void bdrv_close_all(void)
>> {
>> - assert(job_next(NULL) == NULL);
>> GLOBAL_STATE_CODE();
>> + assert(job_next(NULL) == NULL);
>>
>> /* Drop references from requests still in flight, such as canceled block
>> * jobs whose AIO context has not been polled yet */
>> @@ -6165,13 +6165,16 @@ XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp)
>> }
>> }
>>
>> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
>> - GSList *el;
>> + WITH_JOB_LOCK_GUARD() {
>> + for (job = block_job_next_locked(NULL); job;
>> + job = block_job_next_locked(job)) {
>> + GSList *el;
>>
>> - xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
>> - job->job.id);
>> - for (el = job->nodes; el; el = el->next) {
>> - xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
>> + xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
>> + job->job.id);
>> + for (el = job->nodes; el; el = el->next) {
>> + xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
>> + }
>> }
>> }
>>
>> diff --git a/blockdev.c b/blockdev.c
>> index 71f793c4ab..5b79093155 100644
>> --- a/blockdev.c
>> +++ b/blockdev.c
>> @@ -150,12 +150,15 @@ void blockdev_mark_auto_del(BlockBackend *blk)
>> return;
>> }
>>
>> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
>> + JOB_LOCK_GUARD();
>> +
>> + for (job = block_job_next_locked(NULL); job;
>> + job = block_job_next_locked(job)) {
>> if (block_job_has_bdrv(job, blk_bs(blk))) {
>
> Should this be renamed to block_job_has_bdrv_locked() now?
>
> It looks to me like it does need the locking. (Which actually makes
> this patch a fix and not just an optimisation as the commit message
> suggests.)
Nope, as GSList *nodes; is always read and written under BQL.
>
>> AioContext *aio_context = job->job.aio_context;
>> aio_context_acquire(aio_context);
>>
>> - job_cancel(&job->job, false);
>> + job_cancel_locked(&job->job, false);
>>
>> aio_context_release(aio_context);
>> }
>> @@ -3745,7 +3748,10 @@ BlockJobInfoList *qmp_query_block_jobs(Error **errp)
>> BlockJobInfoList *head = NULL, **tail = &head;
>> BlockJob *job;
>>
>> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
>> + JOB_LOCK_GUARD();
>> +
>> + for (job = block_job_next_locked(NULL); job;
>> + job = block_job_next_locked(job)) {
>> BlockJobInfo *value;
>> AioContext *aio_context;
>
> More context:
>
> BlockJobInfo *value;
> AioContext *aio_context;
>
> if (block_job_is_internal(job)) {
> continue;
> }
> aio_context = block_job_get_aio_context(job);
> aio_context_acquire(aio_context);
> value = block_job_query(job, errp);
> aio_context_release(aio_context);
>
> This should become block_job_query_locked(). (You do that in patch 18,
> but it looks a bit out of place there - which is precisely because it
> really belongs in this one.)
Ok
>
>> diff --git a/blockjob.c b/blockjob.c
>> index 0d59aba439..96fb9d9f73 100644
>> --- a/blockjob.c
>> +++ b/blockjob.c
>> @@ -111,8 +111,10 @@ static bool child_job_drained_poll(BdrvChild *c)
>> /* An inactive or completed job doesn't have any pending requests. Jobs
>> * with !job->busy are either already paused or have a pause point after
>> * being reentered, so no job driver code will run before they pause. */
>> - if (!job->busy || job_is_completed(job)) {
>> - return false;
>> + WITH_JOB_LOCK_GUARD() {
>> + if (!job->busy || job_is_completed_locked(job)) {
>> + return false;
>> + }
>> }
>>
>> /* Otherwise, assume that it isn't fully stopped yet, but allow the job to
>
> Assuming that the job status can actually change, don't we need the
> locking for the rest of the function, too? Otherwise we might call
> drv->drained_poll() for a job that has already paused or completed.
>
> Of course, this goes against the assumption that all callbacks are
> called without holding the job lock. Maybe it's not a good assumption.
>
>> @@ -475,13 +477,15 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver,
>> job->ready_notifier.notify = block_job_event_ready;
>> job->idle_notifier.notify = block_job_on_idle;
>>
>> - notifier_list_add(&job->job.on_finalize_cancelled,
>> - &job->finalize_cancelled_notifier);
>> - notifier_list_add(&job->job.on_finalize_completed,
>> - &job->finalize_completed_notifier);
>> - notifier_list_add(&job->job.on_pending, &job->pending_notifier);
>> - notifier_list_add(&job->job.on_ready, &job->ready_notifier);
>> - notifier_list_add(&job->job.on_idle, &job->idle_notifier);
>> + WITH_JOB_LOCK_GUARD() {
>> + notifier_list_add(&job->job.on_finalize_cancelled,
>> + &job->finalize_cancelled_notifier);
>> + notifier_list_add(&job->job.on_finalize_completed,
>> + &job->finalize_completed_notifier);
>> + notifier_list_add(&job->job.on_pending, &job->pending_notifier);
>> + notifier_list_add(&job->job.on_ready, &job->ready_notifier);
>> + notifier_list_add(&job->job.on_idle, &job->idle_notifier);
>> + }
>>
>> error_setg(&job->blocker, "block device is in use by block job: %s",
>> job_type_str(&job->job));
>
> Why is this the right scope for the lock? It looks very arbitrary to
> lock only here, but not for the assignments above or the function calls
> below.
>
> Given that job_create() already puts the job in the job_list so it
> becomes visible for other code, should we not keep the job lock from the
> moment that we create the job until it is fully initialised?
I try to protect only what needs protection, nothing more. Otherwise
then it is not clear what are we protecting and why. According to the
split I made in job.h, things like job_type_str and whatever I did not
protect are not protected because they don't need the lock.
>
>> @@ -558,10 +562,15 @@ BlockErrorAction block_job_error_action(BlockJob *job, BlockdevOnError on_err,
>> action);
>> }
>> if (action == BLOCK_ERROR_ACTION_STOP) {
>> - if (!job->job.user_paused) {
>> - job_pause(&job->job);
>> - /* make the pause user visible, which will be resumed from QMP. */
>> - job->job.user_paused = true;
>> + WITH_JOB_LOCK_GUARD() {
>> + if (!job->job.user_paused) {
>> + job_pause_locked(&job->job);
>> + /*
>> + * make the pause user visible, which will be
>> + * resumed from QMP.
>> + */
>> + job->job.user_paused = true;
>> + }
>> }
>> block_job_iostatus_set_err(job, error);
>
> Why is this call not in the critical section? It accesses job->iostatus.
But the blockjob is not yet "classified". Comes after.
>
>> }
>> diff --git a/job-qmp.c b/job-qmp.c
>> index ac11a6c23c..cfaf34ffb7 100644
>> --- a/job-qmp.c
>> +++ b/job-qmp.c
>> @@ -194,7 +194,9 @@ JobInfoList *qmp_query_jobs(Error **errp)
>> JobInfoList *head = NULL, **tail = &head;
>> Job *job;
>>
>> - for (job = job_next(NULL); job; job = job_next(job)) {
>> + JOB_LOCK_GUARD();
>> +
>> + for (job = job_next_locked(NULL); job; job = job_next_locked(job)) {
>> JobInfo *value;
>> AioContext *aio_context;
>
> Should job_query_single() be renamed to job_query_single_locked()?
Makes sense
>
>> diff --git a/job.c b/job.c
>> index ebaa4e585b..b0729e2bb2 100644
>> --- a/job.c
>> +++ b/job.c
>> @@ -668,7 +668,7 @@ void coroutine_fn job_pause_point(Job *job)
>> job_pause_point_locked(job);
>> }
>>
>> -void job_yield_locked(Job *job)
>> +static void job_yield_locked(Job *job)
>> {
>> assert(job->busy);
>
> It was already unused outside of job.c before this patch. Should it have
> been static from the start?
>
>> @@ -1041,11 +1041,14 @@ static void job_completed_txn_abort_locked(Job *job)
>> /* Called with job_mutex held, but releases it temporarily */
>> static int job_prepare_locked(Job *job)
>> {
>> + int ret;
>> +
>> GLOBAL_STATE_CODE();
>> if (job->ret == 0 && job->driver->prepare) {
>> job_unlock();
>> - job->ret = job->driver->prepare(job);
>> + ret = job->driver->prepare(job);
>> job_lock();
>> + job->ret = ret;
>> job_update_rc_locked(job);
>> }
>> return job->ret;
>> diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
>> index 1ebb89f46c..1897ed7a13 100644
>> --- a/monitor/qmp-cmds.c
>> +++ b/monitor/qmp-cmds.c
>> @@ -133,8 +133,11 @@ void qmp_cont(Error **errp)
>> blk_iostatus_reset(blk);
>> }
>>
>> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
>> - block_job_iostatus_reset(job);
>> + WITH_JOB_LOCK_GUARD() {
>> + for (job = block_job_next_locked(NULL); job;
>> + job = block_job_next_locked(job)) {
>> + block_job_iostatus_reset_locked(job);
>> + }
>> }
>>
>> /* Continuing after completed migration. Images have been inactivated to
>> diff --git a/qemu-img.c b/qemu-img.c
>> index 4cf4d2423d..98c7662b0f 100644
>> --- a/qemu-img.c
>> +++ b/qemu-img.c
>> @@ -912,25 +912,30 @@ static void run_block_job(BlockJob *job, Error **errp)
>> int ret = 0;
>>
>> aio_context_acquire(aio_context);
>> - job_ref(&job->job);
>> - do {
>> - float progress = 0.0f;
>> - aio_poll(aio_context, true);
>> + WITH_JOB_LOCK_GUARD() {
>> + job_ref_locked(&job->job);
>> + do {
>> + float progress = 0.0f;
>> + job_unlock();
>
> This might be more a question of style, but mixing WITH_JOB_LOCK_GUARD()
> with manual job_unlock()/job_lock() feels dangerous. What if someone
> added a break between the unlock/lock pair? The lock guard would try to
> unlock a mutex that is already unlocked, which probably means an
> assertion failure.
>
> I feel we should just use manual job_lock()/job_unlock() for everything
> in this function.
As you wish, ok.
>
>> + aio_poll(aio_context, true);
>> +
>> + progress_get_snapshot(&job->job.progress, &progress_current,
>> + &progress_total);
>> + if (progress_total) {
>> + progress = (float)progress_current / progress_total * 100.f;
>> + }
>> + qemu_progress_print(progress, 0);
>> + job_lock();
>> + } while (!job_is_ready_locked(&job->job) &&
>> + !job_is_completed_locked(&job->job));
>>
>> - progress_get_snapshot(&job->job.progress, &progress_current,
>> - &progress_total);
>> - if (progress_total) {
>> - progress = (float)progress_current / progress_total * 100.f;
>> + if (!job_is_completed_locked(&job->job)) {
>> + ret = job_complete_sync_locked(&job->job, errp);
>> + } else {
>> + ret = job->job.ret;
>> }
>> - qemu_progress_print(progress, 0);
>> - } while (!job_is_ready(&job->job) && !job_is_completed(&job->job));
>> -
>> - if (!job_is_completed(&job->job)) {
>> - ret = job_complete_sync(&job->job, errp);
>> - } else {
>> - ret = job->job.ret;
>> + job_unref_locked(&job->job);
>> }
>> - job_unref(&job->job);
>> aio_context_release(aio_context);
>>
>> /* publish completion progress only when success */
>
Emanuele
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 12/21] commit and mirror: create new nodes using bdrv_get_aio_context, and not the job aiocontext
2022-08-05 8:14 ` Kevin Wolf
@ 2022-08-16 14:57 ` Emanuele Giuseppe Esposito
0 siblings, 0 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-16 14:57 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 05/08/2022 um 10:14 schrieb Kevin Wolf:
> Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
>> We are always using the given bs AioContext, so there is no need
>> to take the job ones (which is identical anyways).
>> This also reduces the point we need to check when protecting
>> job.aio_context field.
>>
>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
>
> I'm not sure against which scenario this is actually protecting. The
> only reason for not using job.aio_context seems to be if someone else
> can change the job AioContext in parallel. However, if that is the case
> (I don't think it is, but for the hypothetical case this patch seems to
> address), the AioContext is not identical any more and the change is
> wrong because we actually want things to run in the job AioContext -
> otherwise accessing the BlockBackend from the job coroutine wouldn't be
> possible.
>
> So I believe the current code expresses how we actually want to use the
> BlockBackend and the change doesn't only protect nothing, but is even
> misleading because it implies that the job can work with any AioContext,
> which is not true.
>
> Kevin
>
Ok, dropped
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 13/21] job: detect change of aiocontext within job coroutine
2022-08-05 8:37 ` Kevin Wolf
@ 2022-08-16 15:09 ` Emanuele Giuseppe Esposito
2022-08-17 8:34 ` Kevin Wolf
0 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-16 15:09 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 05/08/2022 um 10:37 schrieb Kevin Wolf:
> Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
>> From: Paolo Bonzini <pbonzini@redhat.com>
>>
>> We want to make sure access of job->aio_context is always done
>> under either BQL or job_mutex.
>
> Is this the goal of this series? If so, it would have been useful to
> state somewhere more obvious, because I had assumed that holding the BQL
> would not be considered enough, but everyone needs to hold the job_mutex.
It is the goal for this patch :)
The whole job API can't rely on BQL since there are coroutines running
in another aiocontext.
>
>> The problem is that using
>> aio_co_enter(job->aiocontext, job->co) in job_start and job_enter_cond
>> makes the coroutine immediately resume, so we can't hold the job lock.
>> And caching it is not safe either, as it might change.
>>
>> job_start is under BQL, so it can freely read job->aiocontext, but
>> job_enter_cond is not. In order to fix this, use aio_co_wake():
>> the advantage is that it won't use job->aiocontext, but the
>> main disadvantage is that it won't be able to detect a change of
>> job AioContext.
>>
>> Calling bdrv_try_set_aio_context() will issue the following calls
>> (simplified):
>> * in terms of bdrv callbacks:
>> .drained_begin -> .set_aio_context -> .drained_end
>> * in terms of child_job functions:
>> child_job_drained_begin -> child_job_set_aio_context -> child_job_drained_end
>> * in terms of job functions:
>> job_pause_locked -> job_set_aio_context -> job_resume_locked
>>
>> We can see that after setting the new aio_context, job_resume_locked
>> calls again job_enter_cond, which then invokes aio_co_wake(). But
>> while job->aiocontext has been set in job_set_aio_context,
>> job->co->ctx has not changed, so the coroutine would be entering in
>> the wrong aiocontext.
>>
>> Using aio_co_schedule in job_resume_locked() might seem as a valid
>> alternative, but the problem is that the bh resuming the coroutine
>> is not scheduled immediately, and if in the meanwhile another
>> bdrv_try_set_aio_context() is run (see test_propagate_mirror() in
>> test-block-iothread.c), we would have the first schedule in the
>> wrong aiocontext, and the second set of drains won't even manage
>> to schedule the coroutine, as job->busy would still be true from
>> the previous job_resume_locked().
>>
>> The solution is to stick with aio_co_wake(), but then detect every time
>> the coroutine resumes back from yielding if job->aio_context
>> has changed. If so, we can reschedule it to the new context.
>
> Hm, but with this in place, what does aio_co_wake() actually buy us
> compared to aio_co_enter()?
>
> I guess it's a bit simpler code because you don't have to explicitly
> specify the AioContext, but we're still going to enter the coroutine in
> the wrong AioContext occasionally and have to reschedule it, just like
> in the existing code (except that the rescheduling doesn't exist there
> yet).
>
> So while I don't disagree with the change, I don't think the
> justification in the commit message is right for this part.
What do you suggest to change?
>
>> Check for the aiocontext change in job_do_yield_locked because:
>> 1) aio_co_reschedule_self requires to be in the running coroutine
>> 2) since child_job_set_aio_context allows changing the aiocontext only
>> while the job is paused, this is the exact place where the coroutine
>> resumes, before running JobDriver's code.
>>
>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> ---
>> job.c | 21 +++++++++++++++++++--
>> 1 file changed, 19 insertions(+), 2 deletions(-)
>>
>> diff --git a/job.c b/job.c
>> index b0729e2bb2..ecec66b44e 100644
>> --- a/job.c
>> +++ b/job.c
>> @@ -585,7 +585,9 @@ void job_enter_cond_locked(Job *job, bool(*fn)(Job *job))
>> timer_del(&job->sleep_timer);
>> job->busy = true;
>> real_job_unlock();
>> - aio_co_enter(job->aio_context, job->co);
>> + job_unlock();
>> + aio_co_wake(job->co);
>> + job_lock();
>
> The addition of job_unlock/lock is unrelated, this was necessary even
> before this patch.
Ok
>
>> }
>>
>> void job_enter_cond(Job *job, bool(*fn)(Job *job))
>> @@ -611,6 +613,8 @@ void job_enter(Job *job)
>> */
>> static void coroutine_fn job_do_yield_locked(Job *job, uint64_t ns)
>> {
>> + AioContext *next_aio_context;
>> +
>> real_job_lock();
>> if (ns != -1) {
>> timer_mod(&job->sleep_timer, ns);
>> @@ -622,7 +626,20 @@ static void coroutine_fn job_do_yield_locked(Job *job, uint64_t ns)
>> qemu_coroutine_yield();
>> job_lock();
>>
>> - /* Set by job_enter_cond() before re-entering the coroutine. */
>> + next_aio_context = job->aio_context;
>> + /*
>> + * Coroutine has resumed, but in the meanwhile the job AioContext
>> + * might have changed via bdrv_try_set_aio_context(), so we need to move
>> + * the coroutine too in the new aiocontext.
>> + */
>> + while (qemu_get_current_aio_context() != next_aio_context) {
>> + job_unlock();
>> + aio_co_reschedule_self(next_aio_context);
>> + job_lock();
>> + next_aio_context = job->aio_context;
>> + }
>> +
>> + /* Set by job_enter_cond_locked() before re-entering the coroutine. */
>> assert(job->busy);
>> }
>
Emanuele
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 02/21] job.h: categorize fields in struct Job
2022-07-25 7:38 ` [PATCH v10 02/21] job.h: categorize fields in struct Job Emanuele Giuseppe Esposito
2022-07-29 12:30 ` Kevin Wolf
@ 2022-08-16 18:28 ` Stefan Hajnoczi
1 sibling, 0 replies; 71+ messages in thread
From: Stefan Hajnoczi @ 2022-08-16 18:28 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
[-- Attachment #1: Type: text/plain, Size: 533 bytes --]
On Mon, Jul 25, 2022 at 03:38:36AM -0400, Emanuele Giuseppe Esposito wrote:
> Categorize the fields in struct Job to understand which ones
> need to be protected by the job mutex and which don't.
>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> ---
> include/qemu/job.h | 61 +++++++++++++++++++++++++++-------------------
> 1 file changed, 36 insertions(+), 25 deletions(-)
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact
2022-07-25 7:38 ` [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact Emanuele Giuseppe Esposito
2022-07-27 10:45 ` Vladimir Sementsov-Ogievskiy
2022-07-29 13:33 ` Kevin Wolf
@ 2022-08-16 18:31 ` Stefan Hajnoczi
2 siblings, 0 replies; 71+ messages in thread
From: Stefan Hajnoczi @ 2022-08-16 18:31 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]
On Mon, Jul 25, 2022 at 03:38:39AM -0400, Emanuele Giuseppe Esposito wrote:
> With "intact" we mean that all job.h functions implicitly
> take the lock. Therefore API callers are unmodified.
>
> This means that:
> - many static functions that will be always called with job lock held
> become _locked, and call _locked functions
> - all public functions take the lock internally if needed, and call _locked
> functions
> - all public functions called internally by other functions in job.c will have a
> _locked counterpart (sometimes public), to avoid deadlocks (job lock already taken).
> These functions are not used for now.
> - some public functions called only from exernal files (not job.c) do not
> have _locked() counterpart and take the lock inside. Others won't need
> the lock at all because use fields only set at initialization and
> never modified.
>
> job_{lock/unlock} is independent from real_job_{lock/unlock}.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> ---
> include/qemu/job.h | 138 ++++++++++-
> job.c | 600 +++++++++++++++++++++++++++++++--------------
> 2 files changed, 553 insertions(+), 185 deletions(-)
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 06/21] job: move and update comments from blockjob.c
2022-07-25 7:38 ` [PATCH v10 06/21] job: move and update comments from blockjob.c Emanuele Giuseppe Esposito
2022-08-03 15:47 ` Kevin Wolf
@ 2022-08-16 18:32 ` Stefan Hajnoczi
1 sibling, 0 replies; 71+ messages in thread
From: Stefan Hajnoczi @ 2022-08-16 18:32 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
[-- Attachment #1: Type: text/plain, Size: 659 bytes --]
On Mon, Jul 25, 2022 at 03:38:40AM -0400, Emanuele Giuseppe Esposito wrote:
> This comment applies more on job, it was left in blockjob as in the past
> the whole job logic was implemented there.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> No functional change intended.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> ---
> blockjob.c | 20 --------------------
> job.c | 14 ++++++++++++++
> 2 files changed, 14 insertions(+), 20 deletions(-)
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 07/21] blockjob: introduce block_job _locked() APIs
2022-07-25 7:38 ` [PATCH v10 07/21] blockjob: introduce block_job _locked() APIs Emanuele Giuseppe Esposito
2022-08-03 15:52 ` Kevin Wolf
@ 2022-08-16 18:33 ` Stefan Hajnoczi
1 sibling, 0 replies; 71+ messages in thread
From: Stefan Hajnoczi @ 2022-08-16 18:33 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
[-- Attachment #1: Type: text/plain, Size: 762 bytes --]
On Mon, Jul 25, 2022 at 03:38:41AM -0400, Emanuele Giuseppe Esposito wrote:
> Just as done with job.h, create _locked() functions in blockjob.h
>
> These functions will be later useful when caller has already taken
> the lock. All blockjob _locked functions call job _locked functions.
>
> Note: at this stage, job_{lock/unlock} and job lock guard macros
> are *nop*.
>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> ---
> blockjob.c | 52 ++++++++++++++++++++++++++++++++--------
> include/block/blockjob.h | 18 ++++++++++++++
> 2 files changed, 60 insertions(+), 10 deletions(-)
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex
2022-08-05 9:12 ` Kevin Wolf
@ 2022-08-17 8:04 ` Emanuele Giuseppe Esposito
2022-08-17 13:10 ` Emanuele Giuseppe Esposito
0 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-17 8:04 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 05/08/2022 um 11:12 schrieb Kevin Wolf:
> Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
>> In order to make it thread safe, implement a "fake rwlock",
>> where we allow reads under BQL *or* job_mutex held, but
>> writes only under BQL *and* job_mutex.
>
> Oh, so the "or BQL" part is only for job.aio_context? Okay.
>
>> The only write we have is in child_job_set_aio_ctx, which always
>> happens under drain (so the job is paused).
>> For this reason, introduce job_set_aio_context and make sure that
>> the context is set under BQL, job_mutex and drain.
>> Also make sure all other places where the aiocontext is read
>> are protected.
>>
>> Note: at this stage, job_{lock/unlock} and job lock guard macros
>> are *nop*.
>>
>> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
>> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>> ---
>> block/replication.c | 6 ++++--
>> blockjob.c | 3 ++-
>> include/qemu/job.h | 19 ++++++++++++++++++-
>> job.c | 12 ++++++++++++
>> 4 files changed, 36 insertions(+), 4 deletions(-)
>>
>> diff --git a/block/replication.c b/block/replication.c
>> index 55c8f894aa..2189863df1 100644
>> --- a/block/replication.c
>> +++ b/block/replication.c
>> @@ -148,8 +148,10 @@ static void replication_close(BlockDriverState *bs)
>> }
>> if (s->stage == BLOCK_REPLICATION_FAILOVER) {
>> commit_job = &s->commit_job->job;
>> - assert(commit_job->aio_context == qemu_get_current_aio_context());
>> - job_cancel_sync(commit_job, false);
>> + WITH_JOB_LOCK_GUARD() {
>> + assert(commit_job->aio_context == qemu_get_current_aio_context());
>> + job_cancel_sync_locked(commit_job, false);
>> + }
>> }
>
> .bdrv_close runs under the BQL, so why is this needed? Maybe a
> GLOBAL_STATE_CODE() annotation would be helpful, though.
I think I left it because it would be confusing to leave a _locked
function without the job lock. I'll add the GLOBAL_STATE_CODE anyways.
>
>> if (s->mode == REPLICATION_MODE_SECONDARY) {
>> diff --git a/blockjob.c b/blockjob.c
>> index 96fb9d9f73..9ff2727025 100644
>> --- a/blockjob.c
>> +++ b/blockjob.c
>> @@ -162,12 +162,13 @@ static void child_job_set_aio_ctx(BdrvChild *c, AioContext *ctx,
>> bdrv_set_aio_context_ignore(sibling->bs, ctx, ignore);
>> }
>>
>> - job->job.aio_context = ctx;
>> + job_set_aio_context(&job->job, ctx);
>> }
>>
>> static AioContext *child_job_get_parent_aio_context(BdrvChild *c)
>> {
>> BlockJob *job = c->opaque;
>> + assert(qemu_in_main_thread());
>
> Any reason not to use GLOBAL_STATE_CODE()?
4 months ago GLOBAL_STATE_CODE did not exist yet, and I didn't think
about updating it :)
>
>> return job->job.aio_context;
>> }
>> + /* protect against read in job_do_yield_locked */
>> + JOB_LOCK_GUARD();
>> + /* ensure the coroutine is quiescent while the AioContext is changed */
>> + assert(job->pause_count > 0);
>
> job->pause_count only shows that pausing was requested. The coroutine is
> only really quiescent if job->busy == false, too.
>
> Or maybe job->paused is actually the one you want here.
I think job->paused works too.
Emanuele
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 13/21] job: detect change of aiocontext within job coroutine
2022-08-16 15:09 ` Emanuele Giuseppe Esposito
@ 2022-08-17 8:34 ` Kevin Wolf
2022-08-17 11:16 ` Emanuele Giuseppe Esposito
0 siblings, 1 reply; 71+ messages in thread
From: Kevin Wolf @ 2022-08-17 8:34 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 16.08.2022 um 17:09 hat Emanuele Giuseppe Esposito geschrieben:
>
>
> Am 05/08/2022 um 10:37 schrieb Kevin Wolf:
> > Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> >> From: Paolo Bonzini <pbonzini@redhat.com>
> >>
> >> We want to make sure access of job->aio_context is always done
> >> under either BQL or job_mutex.
> >
> > Is this the goal of this series? If so, it would have been useful to
> > state somewhere more obvious, because I had assumed that holding the BQL
> > would not be considered enough, but everyone needs to hold the job_mutex.
>
> It is the goal for this patch :)
> The whole job API can't rely on BQL since there are coroutines running
> in another aiocontext.
Yes, as I saw in patch 14, which describes the goal more clearly in the
commit message and also adds the corresponding documentation to
Job.aio_context. Maybe it would have been clearer if the documentation
were already in this patch.
> >> The problem is that using
> >> aio_co_enter(job->aiocontext, job->co) in job_start and job_enter_cond
> >> makes the coroutine immediately resume, so we can't hold the job lock.
> >> And caching it is not safe either, as it might change.
> >>
> >> job_start is under BQL, so it can freely read job->aiocontext, but
> >> job_enter_cond is not. In order to fix this, use aio_co_wake():
> >> the advantage is that it won't use job->aiocontext, but the
> >> main disadvantage is that it won't be able to detect a change of
> >> job AioContext.
> >>
> >> Calling bdrv_try_set_aio_context() will issue the following calls
> >> (simplified):
> >> * in terms of bdrv callbacks:
> >> .drained_begin -> .set_aio_context -> .drained_end
> >> * in terms of child_job functions:
> >> child_job_drained_begin -> child_job_set_aio_context -> child_job_drained_end
> >> * in terms of job functions:
> >> job_pause_locked -> job_set_aio_context -> job_resume_locked
> >>
> >> We can see that after setting the new aio_context, job_resume_locked
> >> calls again job_enter_cond, which then invokes aio_co_wake(). But
> >> while job->aiocontext has been set in job_set_aio_context,
> >> job->co->ctx has not changed, so the coroutine would be entering in
> >> the wrong aiocontext.
> >>
> >> Using aio_co_schedule in job_resume_locked() might seem as a valid
> >> alternative, but the problem is that the bh resuming the coroutine
> >> is not scheduled immediately, and if in the meanwhile another
> >> bdrv_try_set_aio_context() is run (see test_propagate_mirror() in
> >> test-block-iothread.c), we would have the first schedule in the
> >> wrong aiocontext, and the second set of drains won't even manage
> >> to schedule the coroutine, as job->busy would still be true from
> >> the previous job_resume_locked().
> >>
> >> The solution is to stick with aio_co_wake(), but then detect every time
> >> the coroutine resumes back from yielding if job->aio_context
> >> has changed. If so, we can reschedule it to the new context.
> >
> > Hm, but with this in place, what does aio_co_wake() actually buy us
> > compared to aio_co_enter()?
> >
> > I guess it's a bit simpler code because you don't have to explicitly
> > specify the AioContext, but we're still going to enter the coroutine in
> > the wrong AioContext occasionally and have to reschedule it, just like
> > in the existing code (except that the rescheduling doesn't exist there
> > yet).
> >
> > So while I don't disagree with the change, I don't think the
> > justification in the commit message is right for this part.
>
> What do you suggest to change?
The commit message shouldn't pretend that aio_co_wake() solves the
problem (it says "In order to fix this, use aio_co_wake"), even if
that's what you thought at first before you saw that the problem wasn't
fully fixed by it.
I would move the real solution up in the commit message ("In order to
fix this, detect every time..."), and then maybe mention why
aio_co_wake() doesn't solve the problem, but you're leaving it in anyway
because it's nicer than the previous sequence or something like that.
> >> Check for the aiocontext change in job_do_yield_locked because:
> >> 1) aio_co_reschedule_self requires to be in the running coroutine
> >> 2) since child_job_set_aio_context allows changing the aiocontext only
> >> while the job is paused, this is the exact place where the coroutine
> >> resumes, before running JobDriver's code.
> >>
> >> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> >> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> >> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 11/21] jobs: group together API calls under the same job lock
2022-08-16 14:54 ` Emanuele Giuseppe Esposito
@ 2022-08-17 8:46 ` Kevin Wolf
2022-08-17 9:35 ` Emanuele Giuseppe Esposito
0 siblings, 1 reply; 71+ messages in thread
From: Kevin Wolf @ 2022-08-17 8:46 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 16.08.2022 um 16:54 hat Emanuele Giuseppe Esposito geschrieben:
> Am 04/08/2022 um 19:10 schrieb Kevin Wolf:
> > Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
> >> Now that the API offers also _locked() functions, take advantage
> >> of it and give also the caller control to take the lock and call
> >> _locked functions.
> >>
> >> This makes sense especially when we have for loops, because it
> >> makes no sense to have:
> >>
> >> for(job = job_next(); ...)
> >>
> >> where each job_next() takes the lock internally.
> >> Instead we want
> >>
> >> JOB_LOCK_GUARD();
> >> for(job = job_next_locked(); ...)
> >>
> >> In addition, protect also direct field accesses, by either creating a
> >> new critical section or widening the existing ones.
> >
> > "In addition" sounds like it should be a separate patch. I was indeed
> > surprised when after a few for loops where you just pulled the existing
> > locking up a bit, I saw some hunks that add completely new locking.
>
> Would it be okay if we don't split it in two? There would be two
> microscopical patches.
If it would be just a hunk or two, fair enough.
> >> Note: at this stage, job_{lock/unlock} and job lock guard macros
> >> are *nop*.
> >>
> >> Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
> >> ---
> >> block.c | 17 ++++++++++-------
> >> blockdev.c | 12 +++++++++---
> >> blockjob.c | 35 ++++++++++++++++++++++-------------
> >> job-qmp.c | 4 +++-
> >> job.c | 7 +++++--
> >> monitor/qmp-cmds.c | 7 +++++--
> >> qemu-img.c | 37 +++++++++++++++++++++----------------
> >> 7 files changed, 75 insertions(+), 44 deletions(-)
> >>
> >> diff --git a/block.c b/block.c
> >> index 2c00dddd80..7559965dbc 100644
> >> --- a/block.c
> >> +++ b/block.c
> >> @@ -4978,8 +4978,8 @@ static void bdrv_close(BlockDriverState *bs)
> >>
> >> void bdrv_close_all(void)
> >> {
> >> - assert(job_next(NULL) == NULL);
> >> GLOBAL_STATE_CODE();
> >> + assert(job_next(NULL) == NULL);
> >>
> >> /* Drop references from requests still in flight, such as canceled block
> >> * jobs whose AIO context has not been polled yet */
> >> @@ -6165,13 +6165,16 @@ XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp)
> >> }
> >> }
> >>
> >> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> >> - GSList *el;
> >> + WITH_JOB_LOCK_GUARD() {
> >> + for (job = block_job_next_locked(NULL); job;
> >> + job = block_job_next_locked(job)) {
> >> + GSList *el;
> >>
> >> - xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
> >> - job->job.id);
> >> - for (el = job->nodes; el; el = el->next) {
> >> - xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
> >> + xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
> >> + job->job.id);
> >> + for (el = job->nodes; el; el = el->next) {
> >> + xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
> >> + }
> >> }
> >> }
> >>
> >> diff --git a/blockdev.c b/blockdev.c
> >> index 71f793c4ab..5b79093155 100644
> >> --- a/blockdev.c
> >> +++ b/blockdev.c
> >> @@ -150,12 +150,15 @@ void blockdev_mark_auto_del(BlockBackend *blk)
> >> return;
> >> }
> >>
> >> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> >> + JOB_LOCK_GUARD();
> >> +
> >> + for (job = block_job_next_locked(NULL); job;
> >> + job = block_job_next_locked(job)) {
> >> if (block_job_has_bdrv(job, blk_bs(blk))) {
> >
> > Should this be renamed to block_job_has_bdrv_locked() now?
> >
> > It looks to me like it does need the locking. (Which actually makes
> > this patch a fix and not just an optimisation as the commit message
> > suggests.)
>
> Nope, as GSList *nodes; is always read and written under BQL.
Ah, right. I wonder if we should later take the lock anyway even for
fields where it's not strictly necessary to simplify the locking rules.
Having to check the rules for each field separately is kind of hard. But
let's do only the necessary things in this series.
> >
> >> AioContext *aio_context = job->job.aio_context;
> >> aio_context_acquire(aio_context);
> >>
> >> - job_cancel(&job->job, false);
> >> + job_cancel_locked(&job->job, false);
> >>
> >> aio_context_release(aio_context);
> >> }
> >> @@ -3745,7 +3748,10 @@ BlockJobInfoList *qmp_query_block_jobs(Error **errp)
> >> BlockJobInfoList *head = NULL, **tail = &head;
> >> BlockJob *job;
> >>
> >> - for (job = block_job_next(NULL); job; job = block_job_next(job)) {
> >> + JOB_LOCK_GUARD();
> >> +
> >> + for (job = block_job_next_locked(NULL); job;
> >> + job = block_job_next_locked(job)) {
> >> BlockJobInfo *value;
> >> AioContext *aio_context;
> >
> > More context:
> >
> > BlockJobInfo *value;
> > AioContext *aio_context;
> >
> > if (block_job_is_internal(job)) {
> > continue;
> > }
> > aio_context = block_job_get_aio_context(job);
> > aio_context_acquire(aio_context);
> > value = block_job_query(job, errp);
> > aio_context_release(aio_context);
> >
> > This should become block_job_query_locked(). (You do that in patch 18,
> > but it looks a bit out of place there - which is precisely because it
> > really belongs in this one.)
>
> Ok
> >
> >> diff --git a/blockjob.c b/blockjob.c
> >> index 0d59aba439..96fb9d9f73 100644
> >> --- a/blockjob.c
> >> +++ b/blockjob.c
> >> @@ -111,8 +111,10 @@ static bool child_job_drained_poll(BdrvChild *c)
> >> /* An inactive or completed job doesn't have any pending requests. Jobs
> >> * with !job->busy are either already paused or have a pause point after
> >> * being reentered, so no job driver code will run before they pause. */
> >> - if (!job->busy || job_is_completed(job)) {
> >> - return false;
> >> + WITH_JOB_LOCK_GUARD() {
> >> + if (!job->busy || job_is_completed_locked(job)) {
> >> + return false;
> >> + }
> >> }
> >>
> >> /* Otherwise, assume that it isn't fully stopped yet, but allow the job to
> >
> > Assuming that the job status can actually change, don't we need the
> > locking for the rest of the function, too? Otherwise we might call
> > drv->drained_poll() for a job that has already paused or completed.
> >
> > Of course, this goes against the assumption that all callbacks are
> > called without holding the job lock. Maybe it's not a good assumption.
> >
> >> @@ -475,13 +477,15 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver,
> >> job->ready_notifier.notify = block_job_event_ready;
> >> job->idle_notifier.notify = block_job_on_idle;
> >>
> >> - notifier_list_add(&job->job.on_finalize_cancelled,
> >> - &job->finalize_cancelled_notifier);
> >> - notifier_list_add(&job->job.on_finalize_completed,
> >> - &job->finalize_completed_notifier);
> >> - notifier_list_add(&job->job.on_pending, &job->pending_notifier);
> >> - notifier_list_add(&job->job.on_ready, &job->ready_notifier);
> >> - notifier_list_add(&job->job.on_idle, &job->idle_notifier);
> >> + WITH_JOB_LOCK_GUARD() {
> >> + notifier_list_add(&job->job.on_finalize_cancelled,
> >> + &job->finalize_cancelled_notifier);
> >> + notifier_list_add(&job->job.on_finalize_completed,
> >> + &job->finalize_completed_notifier);
> >> + notifier_list_add(&job->job.on_pending, &job->pending_notifier);
> >> + notifier_list_add(&job->job.on_ready, &job->ready_notifier);
> >> + notifier_list_add(&job->job.on_idle, &job->idle_notifier);
> >> + }
> >>
> >> error_setg(&job->blocker, "block device is in use by block job: %s",
> >> job_type_str(&job->job));
> >
> > Why is this the right scope for the lock? It looks very arbitrary to
> > lock only here, but not for the assignments above or the function calls
> > below.
> >
> > Given that job_create() already puts the job in the job_list so it
> > becomes visible for other code, should we not keep the job lock from the
> > moment that we create the job until it is fully initialised?
>
> I try to protect only what needs protection, nothing more. Otherwise
> then it is not clear what are we protecting and why. According to the
> split I made in job.h, things like job_type_str and whatever I did not
> protect are not protected because they don't need the lock.
I think the last paragraph above explains what it would protect?
Having a half-initialised job in the job list without holding the lock
sounds dangerous to me. Maybe it's actually okay in practice because
this is GLOBAL_STATE_CODE() and we can assume that code accessing
the job list outside of the main thread probably skips over the
half-initialised job, but it's another case where relying on the BQL is
confusing when there would be a more specific lock for it.
> >
> >> @@ -558,10 +562,15 @@ BlockErrorAction block_job_error_action(BlockJob *job, BlockdevOnError on_err,
> >> action);
> >> }
> >> if (action == BLOCK_ERROR_ACTION_STOP) {
> >> - if (!job->job.user_paused) {
> >> - job_pause(&job->job);
> >> - /* make the pause user visible, which will be resumed from QMP. */
> >> - job->job.user_paused = true;
> >> + WITH_JOB_LOCK_GUARD() {
> >> + if (!job->job.user_paused) {
> >> + job_pause_locked(&job->job);
> >> + /*
> >> + * make the pause user visible, which will be
> >> + * resumed from QMP.
> >> + */
> >> + job->job.user_paused = true;
> >> + }
> >> }
> >> block_job_iostatus_set_err(job, error);
> >
> > Why is this call not in the critical section? It accesses job->iostatus.
>
> But the blockjob is not yet "classified". Comes after.
Ok.
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 11/21] jobs: group together API calls under the same job lock
2022-08-17 8:46 ` Kevin Wolf
@ 2022-08-17 9:35 ` Emanuele Giuseppe Esposito
2022-08-17 9:59 ` Kevin Wolf
0 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-17 9:35 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 17/08/2022 um 10:46 schrieb Kevin Wolf:
>>>> @@ -475,13 +477,15 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver,
>>>> job->ready_notifier.notify = block_job_event_ready;
>>>> job->idle_notifier.notify = block_job_on_idle;
>>>>
>>>> - notifier_list_add(&job->job.on_finalize_cancelled,
>>>> - &job->finalize_cancelled_notifier);
>>>> - notifier_list_add(&job->job.on_finalize_completed,
>>>> - &job->finalize_completed_notifier);
>>>> - notifier_list_add(&job->job.on_pending, &job->pending_notifier);
>>>> - notifier_list_add(&job->job.on_ready, &job->ready_notifier);
>>>> - notifier_list_add(&job->job.on_idle, &job->idle_notifier);
>>>> + WITH_JOB_LOCK_GUARD() {
>>>> + notifier_list_add(&job->job.on_finalize_cancelled,
>>>> + &job->finalize_cancelled_notifier);
>>>> + notifier_list_add(&job->job.on_finalize_completed,
>>>> + &job->finalize_completed_notifier);
>>>> + notifier_list_add(&job->job.on_pending, &job->pending_notifier);
>>>> + notifier_list_add(&job->job.on_ready, &job->ready_notifier);
>>>> + notifier_list_add(&job->job.on_idle, &job->idle_notifier);
>>>> + }
>>>>
>>>> error_setg(&job->blocker, "block device is in use by block job: %s",
>>>> job_type_str(&job->job));
>>> Why is this the right scope for the lock? It looks very arbitrary to
>>> lock only here, but not for the assignments above or the function calls
>>> below.
>>>
>>> Given that job_create() already puts the job in the job_list so it
>>> becomes visible for other code, should we not keep the job lock from the
>>> moment that we create the job until it is fully initialised?
>> I try to protect only what needs protection, nothing more. Otherwise
>> then it is not clear what are we protecting and why. According to the
>> split I made in job.h, things like job_type_str and whatever I did not
>> protect are not protected because they don't need the lock.
> I think the last paragraph above explains what it would protect?
>
> Having a half-initialised job in the job list without holding the lock
> sounds dangerous to me. Maybe it's actually okay in practice because
> this is GLOBAL_STATE_CODE() and we can assume that code accessing
> the job list outside of the main thread probably skips over the
> half-initialised job, but it's another case where relying on the BQL is
> confusing when there would be a more specific lock for it.
>
Ok, but this would imply:
1. create job_create_locked()
2. still drop the lock when calling block_job_add_bdrv(), since I am
pretty sure it can drain. So we still split the function in two (or
maybe three, if we need to reaqiure the lock after) parts.
Is that what you had in mind?
Emanuele
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 11/21] jobs: group together API calls under the same job lock
2022-08-17 9:35 ` Emanuele Giuseppe Esposito
@ 2022-08-17 9:59 ` Kevin Wolf
0 siblings, 0 replies; 71+ messages in thread
From: Kevin Wolf @ 2022-08-17 9:59 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 17.08.2022 um 11:35 hat Emanuele Giuseppe Esposito geschrieben:
>
>
> Am 17/08/2022 um 10:46 schrieb Kevin Wolf:
> >>>> @@ -475,13 +477,15 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver,
> >>>> job->ready_notifier.notify = block_job_event_ready;
> >>>> job->idle_notifier.notify = block_job_on_idle;
> >>>>
> >>>> - notifier_list_add(&job->job.on_finalize_cancelled,
> >>>> - &job->finalize_cancelled_notifier);
> >>>> - notifier_list_add(&job->job.on_finalize_completed,
> >>>> - &job->finalize_completed_notifier);
> >>>> - notifier_list_add(&job->job.on_pending, &job->pending_notifier);
> >>>> - notifier_list_add(&job->job.on_ready, &job->ready_notifier);
> >>>> - notifier_list_add(&job->job.on_idle, &job->idle_notifier);
> >>>> + WITH_JOB_LOCK_GUARD() {
> >>>> + notifier_list_add(&job->job.on_finalize_cancelled,
> >>>> + &job->finalize_cancelled_notifier);
> >>>> + notifier_list_add(&job->job.on_finalize_completed,
> >>>> + &job->finalize_completed_notifier);
> >>>> + notifier_list_add(&job->job.on_pending, &job->pending_notifier);
> >>>> + notifier_list_add(&job->job.on_ready, &job->ready_notifier);
> >>>> + notifier_list_add(&job->job.on_idle, &job->idle_notifier);
> >>>> + }
> >>>>
> >>>> error_setg(&job->blocker, "block device is in use by block job: %s",
> >>>> job_type_str(&job->job));
> >>> Why is this the right scope for the lock? It looks very arbitrary to
> >>> lock only here, but not for the assignments above or the function calls
> >>> below.
> >>>
> >>> Given that job_create() already puts the job in the job_list so it
> >>> becomes visible for other code, should we not keep the job lock from the
> >>> moment that we create the job until it is fully initialised?
> >> I try to protect only what needs protection, nothing more. Otherwise
> >> then it is not clear what are we protecting and why. According to the
> >> split I made in job.h, things like job_type_str and whatever I did not
> >> protect are not protected because they don't need the lock.
> > I think the last paragraph above explains what it would protect?
> >
> > Having a half-initialised job in the job list without holding the lock
> > sounds dangerous to me. Maybe it's actually okay in practice because
> > this is GLOBAL_STATE_CODE() and we can assume that code accessing
> > the job list outside of the main thread probably skips over the
> > half-initialised job, but it's another case where relying on the BQL is
> > confusing when there would be a more specific lock for it.
> >
>
> Ok, but this would imply:
> 1. create job_create_locked()
> 2. still drop the lock when calling block_job_add_bdrv(), since I am
> pretty sure it can drain. So we still split the function in two (or
> maybe three, if we need to reaqiure the lock after) parts.
Ah, crap. Temporarily dropping the lock makes it useless, the incomplete
state is then still visible to the outside. So the locked section would
have to end before it.
Maybe just leave it as it is then.
Kevin
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 13/21] job: detect change of aiocontext within job coroutine
2022-08-17 8:34 ` Kevin Wolf
@ 2022-08-17 11:16 ` Emanuele Giuseppe Esposito
0 siblings, 0 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-17 11:16 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel,
Vladimir Sementsov-Ogievskiy
Am 17/08/2022 um 10:34 schrieb Kevin Wolf:
> Am 16.08.2022 um 17:09 hat Emanuele Giuseppe Esposito geschrieben:
>>
>>
>> Am 05/08/2022 um 10:37 schrieb Kevin Wolf:
>>> Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
>>>> From: Paolo Bonzini <pbonzini@redhat.com>
>>>>
>>>> We want to make sure access of job->aio_context is always done
>>>> under either BQL or job_mutex.
>>>
>>> Is this the goal of this series? If so, it would have been useful to
>>> state somewhere more obvious, because I had assumed that holding the BQL
>>> would not be considered enough, but everyone needs to hold the job_mutex.
>>
>> It is the goal for this patch :)
>> The whole job API can't rely on BQL since there are coroutines running
>> in another aiocontext.
>
> Yes, as I saw in patch 14, which describes the goal more clearly in the
> commit message and also adds the corresponding documentation to
> Job.aio_context. Maybe it would have been clearer if the documentation
> were already in this patch.
>
I don't understand what you mean here, sorry. You mean
job_set_aiocontext documentation? I don't see what is confusing here,
could you please clarify?
>>>> The problem is that using
>>>> aio_co_enter(job->aiocontext, job->co) in job_start and job_enter_cond
>>>> makes the coroutine immediately resume, so we can't hold the job lock.
>>>> And caching it is not safe either, as it might change.
>>>>
>>>> job_start is under BQL, so it can freely read job->aiocontext, but
>>>> job_enter_cond is not. In order to fix this, use aio_co_wake():
>>>> the advantage is that it won't use job->aiocontext, but the
>>>> main disadvantage is that it won't be able to detect a change of
>>>> job AioContext.
>>>>
>>>> Calling bdrv_try_set_aio_context() will issue the following calls
>>>> (simplified):
>>>> * in terms of bdrv callbacks:
>>>> .drained_begin -> .set_aio_context -> .drained_end
>>>> * in terms of child_job functions:
>>>> child_job_drained_begin -> child_job_set_aio_context -> child_job_drained_end
>>>> * in terms of job functions:
>>>> job_pause_locked -> job_set_aio_context -> job_resume_locked
>>>>
>>>> We can see that after setting the new aio_context, job_resume_locked
>>>> calls again job_enter_cond, which then invokes aio_co_wake(). But
>>>> while job->aiocontext has been set in job_set_aio_context,
>>>> job->co->ctx has not changed, so the coroutine would be entering in
>>>> the wrong aiocontext.
>>>>
>>>> Using aio_co_schedule in job_resume_locked() might seem as a valid
>>>> alternative, but the problem is that the bh resuming the coroutine
>>>> is not scheduled immediately, and if in the meanwhile another
>>>> bdrv_try_set_aio_context() is run (see test_propagate_mirror() in
>>>> test-block-iothread.c), we would have the first schedule in the
>>>> wrong aiocontext, and the second set of drains won't even manage
>>>> to schedule the coroutine, as job->busy would still be true from
>>>> the previous job_resume_locked().
>>>>
>>>> The solution is to stick with aio_co_wake(), but then detect every time
>>>> the coroutine resumes back from yielding if job->aio_context
>>>> has changed. If so, we can reschedule it to the new context.
>>>
>>> Hm, but with this in place, what does aio_co_wake() actually buy us
>>> compared to aio_co_enter()?
>>>
>>> I guess it's a bit simpler code because you don't have to explicitly
>>> specify the AioContext, but we're still going to enter the coroutine in
>>> the wrong AioContext occasionally and have to reschedule it, just like
>>> in the existing code (except that the rescheduling doesn't exist there
>>> yet).
>>>
>>> So while I don't disagree with the change, I don't think the
>>> justification in the commit message is right for this part.
>>
>> What do you suggest to change?
>
> The commit message shouldn't pretend that aio_co_wake() solves the
> problem (it says "In order to fix this, use aio_co_wake"), even if
> that's what you thought at first before you saw that the problem wasn't
> fully fixed by it.
>
> I would move the real solution up in the commit message ("In order to
> fix this, detect every time..."), and then maybe mention why
> aio_co_wake() doesn't solve the problem, but you're leaving it in anyway
> because it's nicer than the previous sequence or something like that.
I think the issue is "In order to fix this", meaning what "this" refers
to. It is not the fix, it is the problem stated in the previous sentence.
Since job_enter_cond is not under BQL, we can't read job->aiocontext,
therefore use aio_co_wake to avoid doing that.
Would it be ok if I replace the paragraph with:
job_start is under BQL, so it can freely read job->aiocontext, but
job_enter_cond is not. In order to avoid reading the job aiocontext, use
aio_co_wake(), since it resorts so job->co->ctx. The
only disadvantage is that it won't be able to detect a change of
job AioContext, as explained below.
Emanuele
>
>>>> Check for the aiocontext change in job_do_yield_locked because:
>>>> 1) aio_co_reschedule_self requires to be in the running coroutine
>>>> 2) since child_job_set_aio_context allows changing the aiocontext only
>>>> while the job is paused, this is the exact place where the coroutine
>>>> resumes, before running JobDriver's code.
>>>>
>>>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>>>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>>>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>
> Kevin
>
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks
2022-08-05 13:01 ` Kevin Wolf
@ 2022-08-17 12:45 ` Emanuele Giuseppe Esposito
0 siblings, 0 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-17 12:45 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-block, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 05/08/2022 um 15:01 schrieb Kevin Wolf:
> Am 25.07.2022 um 09:38 hat Emanuele Giuseppe Esposito geschrieben:
>> Change the job_{lock/unlock} and macros to use job_mutex.
>>
>> Now that they are not nop anymore, remove the aiocontext
>> to avoid deadlocks.
>
> Okay, so this is the big bad patch where we need to verify the
> completeness of all changes made so far. Let's see...
>
>> Therefore:
>> - when possible, remove completely the aiocontext lock/unlock pair
>> - if it is used by some other function too, reduce the locking
>> section as much as possible, leaving the job API outside.
>> - change AIO_WAIT_WHILE in AIO_WAIT_WHILE_UNLOCKED, since we
>> are not using the aiocontext lock anymore
>
> Does this imply that there is a new rule that job_*() must not be called
> with the AioContext lock held? Or is it optional now?
>
> If it's a rule, it should be documented.
>
>
> (Coming back after reviewing more of the patch:)
>
>
> It doesn't seem to be a rule, or at least not a rule that is obeyed.
>
> Actually each function in job.c belongs in one of at most three
> categories: Must hold the AioContext (because it calls callbacks that
> need it), may hold the AioContext optionally, and must not hold it
> (everything that would cause the deadlocks you're alluding to, but not
> explaining, in the commit message).
>
> It is not obvious which function is in which category. So I maintain
> that we need some documentation for the assumptions made.
>
> All coroutine_fns (which are called from the job coroutine) should be in
> the category that they still run with the AioContext locked, but that
> covers only a small minority of functions.
>
> The driver callbacks look mostly correct at least with respect to
> AioContext locking, even if their status isn't documented. I suppose
> this is the most important part.
>
Well I would say the general case is that no function should need the
aiocontext lock taken. The only ones that need the aiocontext lock are
the functions currently taking it, so just the driver callbacks. But
they take it internally, so what do you propose to document here?
After looking at your feedbacks, I think we just need the driver
callbacks to be documented, right?
>> @@ -197,23 +186,14 @@ static int job_txn_apply_locked(Job *job, int fn(Job *))
>> * break AIO_WAIT_WHILE from within fn.
>> */
>> job_ref_locked(job);
>> - aio_context_release(job->aio_context);
>>
>> QLIST_FOREACH_SAFE(other_job, &txn->jobs, txn_list, next) {
>> - inner_ctx = other_job->aio_context;
>> - aio_context_acquire(inner_ctx);
>> rc = fn(other_job);
>> - aio_context_release(inner_ctx);
>
> Okay, so fn() is now called without the AioContext lock while it was
> called with it previously. This requires checking all callers.
>
> What isn't immediately clear, but seems to be true, is that all notifiers
> don't need the AioContext lock. Probably worth documenting in struct
> Job. (Needed because of job_transition_to_pending_locked(), which is
> passed as fn.)
I am not sure what you mean here, but all fn() passed take the
AioContext lock internally if they need to take it (ie just for the
drivers callback). Reading also below comment, I will add the comment
"Called with AioContext lock held, since many callback implementations
use bdrv_* functions that require to hold the lock." to the following
callbacks:
- prepare
- free
- commit
- abort
- clean
- cancel
Sounds good?
>
>> if (rc) {
>> break;
>> }
>> }
>>
>> - /*
>> - * Note that job->aio_context might have been changed by calling fn, so we
>> - * can't use a local variable to cache it.
>> - */
>> - aio_context_acquire(job->aio_context);
>> job_unref_locked(job);
>> return rc;
>> }
>> @@ -501,8 +481,12 @@ void job_unref_locked(Job *job)
>> assert(!job->txn);
>>
>> if (job->driver->free) {
>> + AioContext *aio_context = job->aio_context;
>> job_unlock();
>> + /* FIXME: aiocontext lock is required because cb calls blk_unref */
>> + aio_context_acquire(aio_context);
>> job->driver->free(job);
>> + aio_context_release(aio_context);
>
> The documentation of JobDriver doesn't specify which callbacks are
> called with the AioContext locked and which are called without it. It
> probably should.
>
> (A good part of the documentation clarifications I'm asking for in this
> review should probably done in a patch before this one, so that
> reviewing the documentation involves checking that the requirements of
> the callback match what we're documenting, and then the review of this
> patch can focus on that the documented contract is still obeyed.)
Makes sense, I'll do what I stated in the answer above.
>
>> job_lock();
>> }
>>
>> @@ -922,6 +900,7 @@ static void job_clean(Job *job)
>> static int job_finalize_single_locked(Job *job)
>> {
>> int job_ret;
>> + AioContext *ctx = job->aio_context;
>>
>> assert(job_is_completed_locked(job));
>>
>> @@ -929,6 +908,7 @@ static int job_finalize_single_locked(Job *job)
>> job_update_rc_locked(job);
>>
>> job_unlock();
>> + aio_context_acquire(ctx);
>>
>> if (!job->ret) {
>> job_commit(job);
>> @@ -937,6 +917,7 @@ static int job_finalize_single_locked(Job *job)
>> }
>> job_clean(job);
>>
>> + aio_context_release(ctx);
>> job_lock();
>
> Let's add comments to job_commit(), job_abort() and job_clean() that
> they are called with the AioContext lock held.
>
> A few lines below we are now calling job->cb() without the AioContext
> lock even though previously it was called with it. Which way is right?
> The intended behaviour should be documented in struct Job.
I think we unfortunately need the aiocontext lock here too.
backup_job_create -> passes cb to block_job_create -> passes it to
job_create
cb is
backup_job_completed -> calls backup_job_cleanup -> calls
reopen_backing_file
Which has subtree drains and so on. So yes, I will wrap cb under
aiocontext lock, merging it in the same aiocontext lock section of
job_commit and friends.
>
>> if (job->cb) {
>> @@ -1002,7 +983,6 @@ static void job_cancel_async_locked(Job *job, bool force)
>> /* Called with job_mutex held, but releases it temporarily. */
>> static void job_completed_txn_abort_locked(Job *job)
>> {
>> - AioContext *ctx;
>> JobTxn *txn = job->txn;
>> Job *other_job;
>>
>> @@ -1015,54 +995,31 @@ static void job_completed_txn_abort_locked(Job *job)
>> txn->aborting = true;
>> job_txn_ref_locked(txn);
>>
>> - /*
>> - * We can only hold the single job's AioContext lock while calling
>> - * job_finalize_single() because the finalization callbacks can involve
>> - * calls of AIO_WAIT_WHILE(), which could deadlock otherwise.
>> - * Note that the job's AioContext may change when it is finalized.
>> - */
>> job_ref_locked(job);
>> - aio_context_release(job->aio_context);
>>
>> /* Other jobs are effectively cancelled by us, set the status for
>> * them; this job, however, may or may not be cancelled, depending
>> * on the caller, so leave it. */
>> QLIST_FOREACH(other_job, &txn->jobs, txn_list) {
>> if (other_job != job) {
>> - ctx = other_job->aio_context;
>> - aio_context_acquire(ctx);
>> /*
>> * This is a transaction: If one job failed, no result will matter.
>> * Therefore, pass force=true to terminate all other jobs as quickly
>> * as possible.
>> */
>> job_cancel_async_locked(other_job, true);
>
> job_cancel_async_locked() calls job->driver->cancel() without the
> AioContext lock now. Some implementations call bdrv_cancel_in_flight().
> Generally bdrv_*() are called with the AioContext lock held.
>
> If we want to make bdrv_cancel_in_flight() an exception, at the very
> least this need to be documented.
>
> The more obvious solution would be to acquire the AioContext lock in
> job_cancel_async_locked() around the callback.
>
Added to the above list of callbacks to document.
>> diff --git a/tests/unit/test-blockjob.c b/tests/unit/test-blockjob.c
>> index b0cd06c529..8a9350078f 100644
>> --- a/tests/unit/test-blockjob.c
>> +++ b/tests/unit/test-blockjob.c
>> @@ -228,10 +228,6 @@ static void cancel_common(CancelJob *s)
>> BlockJob *job = &s->common;
>> BlockBackend *blk = s->blk;
>> JobStatus sts = job->job.status;
>> - AioContext *ctx;
>> -
>> - ctx = job->job.aio_context;
>> - aio_context_acquire(ctx);
>>
>> job_cancel_sync(&job->job, true);
>> WITH_JOB_LOCK_GUARD() {
>> @@ -244,7 +240,6 @@ static void cancel_common(CancelJob *s)
>> }
>> destroy_blk(blk);
>>
>> - aio_context_release(ctx);
>
> destroy_blk() requires the AioContext to be locked.
Makes sense.
>
>> }
>>
>> static void test_cancel_created(void)
>> @@ -384,12 +379,10 @@ static void test_cancel_concluded(void)
>> aio_poll(qemu_get_aio_context(), true);
>> assert_job_status_is(job, JOB_STATUS_PENDING);
>>
>> - aio_context_acquire(job->aio_context);
>> WITH_JOB_LOCK_GUARD() {
>> job_finalize_locked(job, &error_abort);
>> + assert(job->status == JOB_STATUS_CONCLUDED);
>> }
>> - aio_context_release(job->aio_context);
>> - assert_job_status_is(job, JOB_STATUS_CONCLUDED);
>>
>> cancel_common(s);
>> }
>> @@ -482,13 +475,11 @@ static void test_complete_in_standby(void)
>>
>> /* Wait for the job to become READY */
>> job_start(job);
>> - aio_context_acquire(ctx);
>> /*
>> * Here we are waiting for the status to change, so don't bother
>> * protecting the read every time.
>> */
>> - AIO_WAIT_WHILE(ctx, job->status != JOB_STATUS_READY);
>> - aio_context_release(ctx);
>> + AIO_WAIT_WHILE_UNLOCKED(ctx, job->status != JOB_STATUS_READY);
>>
>> /* Begin the drained section, pausing the job */
>> bdrv_drain_all_begin();
>> @@ -507,6 +498,7 @@ static void test_complete_in_standby(void)
>> job_complete_locked(job, &error_abort);
>>
>> /* The test is done now, clean up. */
>> + aio_context_release(ctx);
>
> job_complete_locked() is not supposed to be called with the AioContext
> locked (otherwise blockdev.c would be wrong).
>
Not sure what it has to do with blockdev.c, but you're right, the lock
is not needed there. Test is not affected by the lock position, I will
move it right after bdrv_drain_all_begin.
Emanuele
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex
2022-08-17 8:04 ` Emanuele Giuseppe Esposito
@ 2022-08-17 13:10 ` Emanuele Giuseppe Esposito
2022-08-18 8:48 ` Emanuele Giuseppe Esposito
0 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-17 13:10 UTC (permalink / raw)
To: Kevin Wolf, Paolo Bonzini
Cc: qemu-block, Hanna Reitz, John Snow, Vladimir Sementsov-Ogievskiy,
Wen Congyang, Xie Changlong, Markus Armbruster, Stefan Hajnoczi,
Fam Zheng, qemu-devel
Am 17/08/2022 um 10:04 schrieb Emanuele Giuseppe Esposito:
>>> + /* protect against read in job_do_yield_locked */
>>> + JOB_LOCK_GUARD();
>>> + /* ensure the coroutine is quiescent while the AioContext is changed */
>>> + assert(job->pause_count > 0);
>> job->pause_count only shows that pausing was requested. The coroutine is
>> only really quiescent if job->busy == false, too.
>>
>> Or maybe job->paused is actually the one you want here.
> I think job->paused works too.
>
Actually it doesn't. At least test-block-iothread
test_propagate_mirror() fails, for both job->paused and !job->busy. I
think the reason is that if we wait for the flag to be set, we need to
actually wait that the job gets to the next pause point, because
job_pause() doesn't really pause the job, it just kind of "schedules"
the pause on next pause point.
So, either we leave pause_count > 0, or somehow wait (I was thinking
AIO_WAIT_WHILE(job->paused) but that's probably a very bad idea).
Do you have any suggestion for that?
Maybe Paolo has a better idea on how to do it?
Emanuele
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks
2022-08-16 12:52 ` Emanuele Giuseppe Esposito
@ 2022-08-17 18:54 ` Vladimir Sementsov-Ogievskiy
2022-08-18 7:46 ` Emanuele Giuseppe Esposito
0 siblings, 1 reply; 71+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2022-08-17 18:54 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
On 8/16/22 15:52, Emanuele Giuseppe Esposito wrote:
>>> }
>>> @@ -501,8 +481,12 @@ void job_unref_locked(Job *job)>
>>> assert(!job->txn);
>>> if (job->driver->free) {
>>> + AioContext *aio_context = job->aio_context;
>>> job_unlock();
>>> + /* FIXME: aiocontext lock is required because cb calls
>>> blk_unref */
>>> + aio_context_acquire(aio_context);
>>> job->driver->free(job);
>>> + aio_context_release(aio_context);
>> So, job_unref_locked() must be called with aio_context not locked,
>> otherwise we dead-lock here? That should be documented in function
>> declaration comment.
>>
>> For example in qemu-img.c in run_block_job() we do exactly that: call
>> job_unref_locked() inside aio-context lock critical seaction..
> No, job_unref_locked has also status and refcnt and all the other fields
> that need to be protectd. Only free must be called without job lock held.
>
I mean, don't we try here to acquire aio_context twice, when we call job_unref_locked() with aio_context _already_ acquired?
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks
2022-08-17 18:54 ` Vladimir Sementsov-Ogievskiy
@ 2022-08-18 7:46 ` Emanuele Giuseppe Esposito
2022-08-19 15:49 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-18 7:46 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
Am 17/08/2022 um 20:54 schrieb Vladimir Sementsov-Ogievskiy:
> On 8/16/22 15:52, Emanuele Giuseppe Esposito wrote:
>>>> }
>>>> @@ -501,8 +481,12 @@ void job_unref_locked(Job *job)>
>>>> assert(!job->txn);
>>>> if (job->driver->free) {
>>>> + AioContext *aio_context = job->aio_context;
>>>> job_unlock();
>>>> + /* FIXME: aiocontext lock is required because cb calls
>>>> blk_unref */
>>>> + aio_context_acquire(aio_context);
>>>> job->driver->free(job);
>>>> + aio_context_release(aio_context);
>>> So, job_unref_locked() must be called with aio_context not locked,
>>> otherwise we dead-lock here? That should be documented in function
>>> declaration comment.
>>>
>>> For example in qemu-img.c in run_block_job() we do exactly that: call
>>> job_unref_locked() inside aio-context lock critical seaction..
>> No, job_unref_locked has also status and refcnt and all the other fields
>> that need to be protectd. Only free must be called without job lock held.
>>
>
> I mean, don't we try here to acquire aio_context twice, when we call
> job_unref_locked() with aio_context _already_ acquired?
>
You're right, so why do we need the AioContext lock in qemu-img then?
All jobs API don't need it, aio_poll seems not to need it either, and
progress API has its own lock.
I guess I could remove it?
Emanuele
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex
2022-08-17 13:10 ` Emanuele Giuseppe Esposito
@ 2022-08-18 8:48 ` Emanuele Giuseppe Esposito
0 siblings, 0 replies; 71+ messages in thread
From: Emanuele Giuseppe Esposito @ 2022-08-18 8:48 UTC (permalink / raw)
To: Kevin Wolf, Paolo Bonzini
Cc: qemu-block, Hanna Reitz, John Snow, Vladimir Sementsov-Ogievskiy,
Wen Congyang, Xie Changlong, Markus Armbruster, Stefan Hajnoczi,
Fam Zheng, qemu-devel
Am 17/08/2022 um 15:10 schrieb Emanuele Giuseppe Esposito:
>
>
> Am 17/08/2022 um 10:04 schrieb Emanuele Giuseppe Esposito:
>>>> + /* protect against read in job_do_yield_locked */
>>>> + JOB_LOCK_GUARD();
>>>> + /* ensure the coroutine is quiescent while the AioContext is changed */
>>>> + assert(job->pause_count > 0);
>>> job->pause_count only shows that pausing was requested. The coroutine is
>>> only really quiescent if job->busy == false, too.
>>>
>>> Or maybe job->paused is actually the one you want here.
>> I think job->paused works too.
>>
>
> Actually it doesn't. At least test-block-iothread
> test_propagate_mirror() fails, for both job->paused and !job->busy. I
> think the reason is that if we wait for the flag to be set, we need to
> actually wait that the job gets to the next pause point, because
> job_pause() doesn't really pause the job, it just kind of "schedules"
> the pause on next pause point.
>
> So, either we leave pause_count > 0, or somehow wait (I was thinking
> AIO_WAIT_WHILE(job->paused) but that's probably a very bad idea).
>
> Do you have any suggestion for that?
> Maybe Paolo has a better idea on how to do it?
>
Additional update: after debugging it a little bit, I figured that there
are places where busy=false and paused=false too. This is clear in
job_do_dismiss and not-so-clear in job_exit (at least for me).
I don't know if this is expected or a bug, because it just means that if
we are draining and the job is completing, there is no way to stop it.
Is it a bug? Or am I missing something?
Emanuele
^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks
2022-08-18 7:46 ` Emanuele Giuseppe Esposito
@ 2022-08-19 15:49 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 71+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2022-08-19 15:49 UTC (permalink / raw)
To: Emanuele Giuseppe Esposito, qemu-block
Cc: Kevin Wolf, Hanna Reitz, Paolo Bonzini, John Snow,
Vladimir Sementsov-Ogievskiy, Wen Congyang, Xie Changlong,
Markus Armbruster, Stefan Hajnoczi, Fam Zheng, qemu-devel
On 8/18/22 10:46, Emanuele Giuseppe Esposito wrote:
>
>
> Am 17/08/2022 um 20:54 schrieb Vladimir Sementsov-Ogievskiy:
>> On 8/16/22 15:52, Emanuele Giuseppe Esposito wrote:
>>>>> }
>>>>> @@ -501,8 +481,12 @@ void job_unref_locked(Job *job)>
>>>>> assert(!job->txn);
>>>>> if (job->driver->free) {
>>>>> + AioContext *aio_context = job->aio_context;
>>>>> job_unlock();
>>>>> + /* FIXME: aiocontext lock is required because cb calls
>>>>> blk_unref */
>>>>> + aio_context_acquire(aio_context);
>>>>> job->driver->free(job);
>>>>> + aio_context_release(aio_context);
>>>> So, job_unref_locked() must be called with aio_context not locked,
>>>> otherwise we dead-lock here? That should be documented in function
>>>> declaration comment.
>>>>
>>>> For example in qemu-img.c in run_block_job() we do exactly that: call
>>>> job_unref_locked() inside aio-context lock critical seaction..
>>> No, job_unref_locked has also status and refcnt and all the other fields
>>> that need to be protectd. Only free must be called without job lock held.
>>>
>>
>> I mean, don't we try here to acquire aio_context twice, when we call
>> job_unref_locked() with aio_context _already_ acquired?
>>
> You're right, so why do we need the AioContext lock in qemu-img then?
> All jobs API don't need it, aio_poll seems not to need it either, and
> progress API has its own lock.
>
> I guess I could remove it?
>
Not sure, but hope that yes.
Anyway, trying to lock it twice will dead-lock, as I understand.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 71+ messages in thread
end of thread, other threads:[~2022-08-19 15:54 UTC | newest]
Thread overview: 71+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-25 7:38 [PATCH v10 00/21] job: replace AioContext lock with job_mutex Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 01/21] job.c: make job_mutex and job_lock/unlock() public Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 02/21] job.h: categorize fields in struct Job Emanuele Giuseppe Esposito
2022-07-29 12:30 ` Kevin Wolf
2022-08-16 18:28 ` Stefan Hajnoczi
2022-07-25 7:38 ` [PATCH v10 03/21] job.c: API functions not used outside should be static Emanuele Giuseppe Esposito
2022-07-29 12:30 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 04/21] aio-wait.h: introduce AIO_WAIT_WHILE_UNLOCKED Emanuele Giuseppe Esposito
2022-07-29 12:33 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 05/21] job.c: add job_lock/unlock while keeping job.h intact Emanuele Giuseppe Esposito
2022-07-27 10:45 ` Vladimir Sementsov-Ogievskiy
2022-07-29 13:33 ` Kevin Wolf
2022-08-16 18:31 ` Stefan Hajnoczi
2022-07-25 7:38 ` [PATCH v10 06/21] job: move and update comments from blockjob.c Emanuele Giuseppe Esposito
2022-08-03 15:47 ` Kevin Wolf
2022-08-16 18:32 ` Stefan Hajnoczi
2022-07-25 7:38 ` [PATCH v10 07/21] blockjob: introduce block_job _locked() APIs Emanuele Giuseppe Esposito
2022-08-03 15:52 ` Kevin Wolf
2022-08-16 18:33 ` Stefan Hajnoczi
2022-07-25 7:38 ` [PATCH v10 08/21] jobs: add job lock in find_* functions Emanuele Giuseppe Esposito
2022-08-04 11:47 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 09/21] jobs: use job locks also in the unit tests Emanuele Giuseppe Esposito
2022-07-27 14:29 ` Vladimir Sementsov-Ogievskiy
2022-08-04 11:56 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 10/21] block/mirror.c: use of job helpers in drivers to avoid TOC/TOU Emanuele Giuseppe Esposito
2022-08-04 16:35 ` Kevin Wolf
2022-08-16 14:23 ` Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 11/21] jobs: group together API calls under the same job lock Emanuele Giuseppe Esposito
2022-07-27 14:50 ` Vladimir Sementsov-Ogievskiy
2022-08-04 17:10 ` Kevin Wolf
2022-08-16 14:54 ` Emanuele Giuseppe Esposito
2022-08-17 8:46 ` Kevin Wolf
2022-08-17 9:35 ` Emanuele Giuseppe Esposito
2022-08-17 9:59 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 12/21] commit and mirror: create new nodes using bdrv_get_aio_context, and not the job aiocontext Emanuele Giuseppe Esposito
2022-08-05 8:14 ` Kevin Wolf
2022-08-16 14:57 ` Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 13/21] job: detect change of aiocontext within job coroutine Emanuele Giuseppe Esposito
2022-08-05 8:37 ` Kevin Wolf
2022-08-16 15:09 ` Emanuele Giuseppe Esposito
2022-08-17 8:34 ` Kevin Wolf
2022-08-17 11:16 ` Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 14/21] jobs: protect job.aio_context with BQL and job_mutex Emanuele Giuseppe Esposito
2022-07-27 15:22 ` Vladimir Sementsov-Ogievskiy
2022-08-05 9:12 ` Kevin Wolf
2022-08-17 8:04 ` Emanuele Giuseppe Esposito
2022-08-17 13:10 ` Emanuele Giuseppe Esposito
2022-08-18 8:48 ` Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 15/21] blockjob.h: categorize fields in struct BlockJob Emanuele Giuseppe Esposito
2022-08-05 9:21 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 16/21] blockjob: rename notifier callbacks as _locked Emanuele Giuseppe Esposito
2022-08-05 9:25 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 17/21] blockjob: protect iostatus field in BlockJob struct Emanuele Giuseppe Esposito
2022-07-27 15:29 ` Vladimir Sementsov-Ogievskiy
2022-08-16 12:39 ` Emanuele Giuseppe Esposito
2022-08-05 10:55 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 18/21] job.c: enable job lock/unlock and remove Aiocontext locks Emanuele Giuseppe Esposito
2022-07-27 15:53 ` Vladimir Sementsov-Ogievskiy
2022-08-16 12:52 ` Emanuele Giuseppe Esposito
2022-08-17 18:54 ` Vladimir Sementsov-Ogievskiy
2022-08-18 7:46 ` Emanuele Giuseppe Esposito
2022-08-19 15:49 ` Vladimir Sementsov-Ogievskiy
2022-08-16 12:53 ` Emanuele Giuseppe Esposito
2022-08-05 13:01 ` Kevin Wolf
2022-08-17 12:45 ` Emanuele Giuseppe Esposito
2022-07-25 7:38 ` [PATCH v10 19/21] block_job_query: remove atomic read Emanuele Giuseppe Esposito
2022-08-05 13:01 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 20/21] blockjob: remove unused functions Emanuele Giuseppe Esposito
2022-08-05 13:05 ` Kevin Wolf
2022-07-25 7:38 ` [PATCH v10 21/21] job: " Emanuele Giuseppe Esposito
2022-08-05 13:09 ` Kevin Wolf
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.