* [PATCH for-7.2 1/5] block/mirror: Do not wait for active writes
2022-11-09 16:54 [PATCH for-7.2 0/5] block/mirror: Do not wait for active writes Hanna Reitz
@ 2022-11-09 16:54 ` Hanna Reitz
2022-11-10 12:04 ` Kevin Wolf
2022-11-09 16:54 ` [PATCH for-7.2 2/5] block/mirror: Drop mirror_wait_for_any_operation() Hanna Reitz
` (3 subsequent siblings)
4 siblings, 1 reply; 12+ messages in thread
From: Hanna Reitz @ 2022-11-09 16:54 UTC (permalink / raw)
To: qemu-block
Cc: qemu-devel, Hanna Reitz, Kevin Wolf, John Snow,
Vladimir Sementsov-Ogievskiy
Waiting for all active writes to settle before daring to create a
background copying operation means that we will never do background
operations while the guest does anything (in write-blocking mode), and
therefore cannot converge. Yes, we also will not diverge, but actually
converging would be even nicer.
It is unclear why we did decide to wait for all active writes to settle
before creating a background operation, but it just does not seem
necessary. Active writes will put themselves into the in_flight bitmap
and thus properly block actually conflicting background requests.
It is important for active requests to wait on overlapping background
requests, which we do in active_write_prepare(). However, so far it was
not documented why it is important. Add such documentation now, and
also to the other call of mirror_wait_on_conflicts(), so that it becomes
more clear why and when requests need to actively wait for other
requests to settle.
Another thing to note is that of course we need to ensure that there are
no active requests when the job completes, but that is done by virtue of
the BDS being drained anyway, so there cannot be any active requests at
that point.
With this change, we will need to explicitly keep track of how many
bytes are in flight in active requests so that
job_progress_set_remaining() in mirror_run() can set the correct number
of remaining bytes.
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2123297
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
block/mirror.c | 37 ++++++++++++++++++++++++++++++-------
1 file changed, 30 insertions(+), 7 deletions(-)
diff --git a/block/mirror.c b/block/mirror.c
index 1a75a47cc3..e5467b0053 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -82,6 +82,7 @@ typedef struct MirrorBlockJob {
int max_iov;
bool initial_zeroing_ongoing;
int in_active_write_counter;
+ int64_t active_write_bytes_in_flight;
bool prepared;
bool in_drain;
} MirrorBlockJob;
@@ -494,6 +495,13 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
}
bdrv_dirty_bitmap_unlock(s->dirty_bitmap);
+ /*
+ * Wait for concurrent requests to @offset. The next loop will limit the
+ * copied area based on in_flight_bitmap so we only copy an area that does
+ * not overlap with concurrent in-flight requests. Still, we would like to
+ * copy something, so wait until there are at least no more requests to the
+ * very beginning of the area.
+ */
mirror_wait_on_conflicts(NULL, s, offset, 1);
job_pause_point(&s->common.job);
@@ -988,12 +996,6 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
int64_t cnt, delta;
bool should_complete;
- /* Do not start passive operations while there are active
- * writes in progress */
- while (s->in_active_write_counter) {
- mirror_wait_for_any_operation(s, true);
- }
-
if (s->ret < 0) {
ret = s->ret;
goto immediate_exit;
@@ -1010,7 +1012,9 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
/* cnt is the number of dirty bytes remaining and s->bytes_in_flight is
* the number of bytes currently being processed; together those are
* the current remaining operation length */
- job_progress_set_remaining(&s->common.job, s->bytes_in_flight + cnt);
+ job_progress_set_remaining(&s->common.job,
+ s->bytes_in_flight + cnt +
+ s->active_write_bytes_in_flight);
/* Note that even when no rate limit is applied we need to yield
* periodically with no pending I/O so that bdrv_drain_all() returns.
@@ -1071,6 +1075,10 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
s->in_drain = true;
bdrv_drained_begin(bs);
+
+ /* Must be zero because we are drained */
+ assert(s->in_active_write_counter == 0);
+
cnt = bdrv_get_dirty_count(s->dirty_bitmap);
if (cnt > 0 || mirror_flush(s) < 0) {
bdrv_drained_end(bs);
@@ -1306,6 +1314,7 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
}
job_progress_increase_remaining(&job->common.job, bytes);
+ job->active_write_bytes_in_flight += bytes;
switch (method) {
case MIRROR_METHOD_COPY:
@@ -1327,6 +1336,7 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
abort();
}
+ job->active_write_bytes_in_flight -= bytes;
if (ret >= 0) {
job_progress_update(&job->common.job, bytes);
} else {
@@ -1375,6 +1385,19 @@ static MirrorOp *coroutine_fn active_write_prepare(MirrorBlockJob *s,
s->in_active_write_counter++;
+ /*
+ * Wait for concurrent requests affecting the area. If there are already
+ * running requests that are copying off now-to-be stale data in the area,
+ * we must wait for them to finish before we begin writing fresh data to the
+ * target so that the write operations appear in the correct order.
+ * Note that background requests (see mirror_iteration()) in contrast only
+ * wait for conflicting requests at the start of the dirty area, and then
+ * (based on the in_flight_bitmap) truncate the area to copy so it will not
+ * conflict with any requests beyond that. For active writes, however, we
+ * cannot truncate that area. The request from our parent must be blocked
+ * until the area is copied in full. Therefore, we must wait for the whole
+ * area to become free of concurrent requests.
+ */
mirror_wait_on_conflicts(op, s, offset, bytes);
bitmap_set(s->in_flight_bitmap, start_chunk, end_chunk - start_chunk);
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH for-7.2 1/5] block/mirror: Do not wait for active writes
2022-11-09 16:54 ` [PATCH for-7.2 1/5] " Hanna Reitz
@ 2022-11-10 12:04 ` Kevin Wolf
0 siblings, 0 replies; 12+ messages in thread
From: Kevin Wolf @ 2022-11-10 12:04 UTC (permalink / raw)
To: Hanna Reitz
Cc: qemu-block, qemu-devel, John Snow, Vladimir Sementsov-Ogievskiy
Am 09.11.2022 um 17:54 hat Hanna Reitz geschrieben:
> Waiting for all active writes to settle before daring to create a
> background copying operation means that we will never do background
> operations while the guest does anything (in write-blocking mode), and
> therefore cannot converge. Yes, we also will not diverge, but actually
> converging would be even nicer.
>
> It is unclear why we did decide to wait for all active writes to settle
> before creating a background operation, but it just does not seem
> necessary. Active writes will put themselves into the in_flight bitmap
> and thus properly block actually conflicting background requests.
>
> It is important for active requests to wait on overlapping background
> requests, which we do in active_write_prepare(). However, so far it was
> not documented why it is important. Add such documentation now, and
> also to the other call of mirror_wait_on_conflicts(), so that it becomes
> more clear why and when requests need to actively wait for other
> requests to settle.
>
> Another thing to note is that of course we need to ensure that there are
> no active requests when the job completes, but that is done by virtue of
> the BDS being drained anyway, so there cannot be any active requests at
> that point.
>
> With this change, we will need to explicitly keep track of how many
> bytes are in flight in active requests so that
> job_progress_set_remaining() in mirror_run() can set the correct number
> of remaining bytes.
>
> Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2123297
> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
> @@ -1327,6 +1336,7 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
> abort();
> }
>
> + job->active_write_bytes_in_flight -= bytes;
> if (ret >= 0) {
> job_progress_update(&job->common.job, bytes);
> } else {
Preexisting, but it happens to be in the context: Don't we need to call
job_progress_update() unconditionally? Otherwise, the total length will
go backwards in the case of an error.
Kevin
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH for-7.2 2/5] block/mirror: Drop mirror_wait_for_any_operation()
2022-11-09 16:54 [PATCH for-7.2 0/5] block/mirror: Do not wait for active writes Hanna Reitz
2022-11-09 16:54 ` [PATCH for-7.2 1/5] " Hanna Reitz
@ 2022-11-09 16:54 ` Hanna Reitz
2022-11-10 12:05 ` Kevin Wolf
2022-11-09 16:54 ` [PATCH for-7.2 3/5] block/mirror: Fix NULL s->job in active writes Hanna Reitz
` (2 subsequent siblings)
4 siblings, 1 reply; 12+ messages in thread
From: Hanna Reitz @ 2022-11-09 16:54 UTC (permalink / raw)
To: qemu-block
Cc: qemu-devel, Hanna Reitz, Kevin Wolf, John Snow,
Vladimir Sementsov-Ogievskiy
mirror_wait_for_free_in_flight_slot() is the only remaining user of
mirror_wait_for_any_operation(), so inline the latter into the former.
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
block/mirror.c | 21 ++++++++-------------
1 file changed, 8 insertions(+), 13 deletions(-)
diff --git a/block/mirror.c b/block/mirror.c
index e5467b0053..5b6f42392c 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -305,19 +305,21 @@ static int mirror_cow_align(MirrorBlockJob *s, int64_t *offset,
}
static inline void coroutine_fn
-mirror_wait_for_any_operation(MirrorBlockJob *s, bool active)
+mirror_wait_for_free_in_flight_slot(MirrorBlockJob *s)
{
MirrorOp *op;
QTAILQ_FOREACH(op, &s->ops_in_flight, next) {
- /* Do not wait on pseudo ops, because it may in turn wait on
+ /*
+ * Do not wait on pseudo ops, because it may in turn wait on
* some other operation to start, which may in fact be the
* caller of this function. Since there is only one pseudo op
* at any given time, we will always find some real operation
- * to wait on. */
- if (!op->is_pseudo_op && op->is_in_flight &&
- op->is_active_write == active)
- {
+ * to wait on.
+ * Also, do not wait on active operations, because they do not
+ * use up in-flight slots.
+ */
+ if (!op->is_pseudo_op && op->is_in_flight && !op->is_active_write) {
qemu_co_queue_wait(&op->waiting_requests, NULL);
return;
}
@@ -325,13 +327,6 @@ mirror_wait_for_any_operation(MirrorBlockJob *s, bool active)
abort();
}
-static inline void coroutine_fn
-mirror_wait_for_free_in_flight_slot(MirrorBlockJob *s)
-{
- /* Only non-active operations use up in-flight slots */
- mirror_wait_for_any_operation(s, false);
-}
-
/* Perform a mirror copy operation.
*
* *op->bytes_handled is set to the number of bytes copied after and
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-7.2 3/5] block/mirror: Fix NULL s->job in active writes
2022-11-09 16:54 [PATCH for-7.2 0/5] block/mirror: Do not wait for active writes Hanna Reitz
2022-11-09 16:54 ` [PATCH for-7.2 1/5] " Hanna Reitz
2022-11-09 16:54 ` [PATCH for-7.2 2/5] block/mirror: Drop mirror_wait_for_any_operation() Hanna Reitz
@ 2022-11-09 16:54 ` Hanna Reitz
2022-11-10 12:10 ` Kevin Wolf
2022-11-09 16:54 ` [PATCH for-7.2 4/5] iotests/151: Test that active mirror progresses Hanna Reitz
2022-11-09 16:54 ` [PATCH for-7.2 5/5] iotests/151: Test active requests on mirror start Hanna Reitz
4 siblings, 1 reply; 12+ messages in thread
From: Hanna Reitz @ 2022-11-09 16:54 UTC (permalink / raw)
To: qemu-block
Cc: qemu-devel, Hanna Reitz, Kevin Wolf, John Snow,
Vladimir Sementsov-Ogievskiy
There is a small gap in mirror_start_job() before putting the mirror
filter node into the block graph (bdrv_append() call) and the actual job
being created. Before the job is created, MirrorBDSOpaque.job is NULL.
It is possible that requests come in when bdrv_drained_end() is called,
and those requests would see MirrorBDSOpaque.job == NULL. Have our
filter node handle that case gracefully.
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
block/mirror.c | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)
diff --git a/block/mirror.c b/block/mirror.c
index 5b6f42392c..251adc5ae0 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -1438,11 +1438,13 @@ static int coroutine_fn bdrv_mirror_top_do_write(BlockDriverState *bs,
MirrorOp *op = NULL;
MirrorBDSOpaque *s = bs->opaque;
int ret = 0;
- bool copy_to_target;
+ bool copy_to_target = false;
- copy_to_target = s->job->ret >= 0 &&
- !job_is_cancelled(&s->job->common.job) &&
- s->job->copy_mode == MIRROR_COPY_MODE_WRITE_BLOCKING;
+ if (s->job) {
+ copy_to_target = s->job->ret >= 0 &&
+ !job_is_cancelled(&s->job->common.job) &&
+ s->job->copy_mode == MIRROR_COPY_MODE_WRITE_BLOCKING;
+ }
if (copy_to_target) {
op = active_write_prepare(s->job, offset, bytes);
@@ -1487,11 +1489,13 @@ static int coroutine_fn bdrv_mirror_top_pwritev(BlockDriverState *bs,
QEMUIOVector bounce_qiov;
void *bounce_buf;
int ret = 0;
- bool copy_to_target;
+ bool copy_to_target = false;
- copy_to_target = s->job->ret >= 0 &&
- !job_is_cancelled(&s->job->common.job) &&
- s->job->copy_mode == MIRROR_COPY_MODE_WRITE_BLOCKING;
+ if (s->job) {
+ copy_to_target = s->job->ret >= 0 &&
+ !job_is_cancelled(&s->job->common.job) &&
+ s->job->copy_mode == MIRROR_COPY_MODE_WRITE_BLOCKING;
+ }
if (copy_to_target) {
/* The guest might concurrently modify the data to write; but
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH for-7.2 3/5] block/mirror: Fix NULL s->job in active writes
2022-11-09 16:54 ` [PATCH for-7.2 3/5] block/mirror: Fix NULL s->job in active writes Hanna Reitz
@ 2022-11-10 12:10 ` Kevin Wolf
2022-11-10 12:40 ` Hanna Reitz
0 siblings, 1 reply; 12+ messages in thread
From: Kevin Wolf @ 2022-11-10 12:10 UTC (permalink / raw)
To: Hanna Reitz
Cc: qemu-block, qemu-devel, John Snow, Vladimir Sementsov-Ogievskiy
Am 09.11.2022 um 17:54 hat Hanna Reitz geschrieben:
> There is a small gap in mirror_start_job() before putting the mirror
> filter node into the block graph (bdrv_append() call) and the actual job
> being created. Before the job is created, MirrorBDSOpaque.job is NULL.
>
> It is possible that requests come in when bdrv_drained_end() is called,
> and those requests would see MirrorBDSOpaque.job == NULL. Have our
> filter node handle that case gracefully.
>
> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
This can only happen because bdrv_drained_end() polls, right? So after
changing that it won't be necessary any more, but this series is for 7.2
and the drain one isn't, so this is the right thing to do for now.
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH for-7.2 3/5] block/mirror: Fix NULL s->job in active writes
2022-11-10 12:10 ` Kevin Wolf
@ 2022-11-10 12:40 ` Hanna Reitz
0 siblings, 0 replies; 12+ messages in thread
From: Hanna Reitz @ 2022-11-10 12:40 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-block, qemu-devel, John Snow, Vladimir Sementsov-Ogievskiy
On 10.11.22 13:10, Kevin Wolf wrote:
> Am 09.11.2022 um 17:54 hat Hanna Reitz geschrieben:
>> There is a small gap in mirror_start_job() before putting the mirror
>> filter node into the block graph (bdrv_append() call) and the actual job
>> being created. Before the job is created, MirrorBDSOpaque.job is NULL.
>>
>> It is possible that requests come in when bdrv_drained_end() is called,
>> and those requests would see MirrorBDSOpaque.job == NULL. Have our
>> filter node handle that case gracefully.
>>
>> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
> This can only happen because bdrv_drained_end() polls, right? So after
> changing that it won't be necessary any more, but this series is for 7.2
> and the drain one isn't, so this is the right thing to do for now.
I was wondering the same, but I haven’t tested it yet.
Hanna
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH for-7.2 4/5] iotests/151: Test that active mirror progresses
2022-11-09 16:54 [PATCH for-7.2 0/5] block/mirror: Do not wait for active writes Hanna Reitz
` (2 preceding siblings ...)
2022-11-09 16:54 ` [PATCH for-7.2 3/5] block/mirror: Fix NULL s->job in active writes Hanna Reitz
@ 2022-11-09 16:54 ` Hanna Reitz
2022-11-10 12:33 ` Kevin Wolf
2022-11-09 16:54 ` [PATCH for-7.2 5/5] iotests/151: Test active requests on mirror start Hanna Reitz
4 siblings, 1 reply; 12+ messages in thread
From: Hanna Reitz @ 2022-11-09 16:54 UTC (permalink / raw)
To: qemu-block
Cc: qemu-devel, Hanna Reitz, Kevin Wolf, John Snow,
Vladimir Sementsov-Ogievskiy
Before this series, a mirror job in write-blocking mode would pause
issuing background requests while active requests are in flight. Thus,
if the source is constantly in use by active requests, no actual
progress can be made.
This series should have fixed that, making the mirror job issue
background requests even while active requests are in flight.
Have a new test case in 151 verify this.
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
tests/qemu-iotests/151 | 180 ++++++++++++++++++++++++++++++++++++-
tests/qemu-iotests/151.out | 4 +-
2 files changed, 181 insertions(+), 3 deletions(-)
diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
index 93d14193d0..0a052e5050 100755
--- a/tests/qemu-iotests/151
+++ b/tests/qemu-iotests/151
@@ -19,7 +19,10 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
+import math
import os
+import subprocess
+from typing import List
import iotests
from iotests import qemu_img
@@ -50,7 +53,7 @@ class TestActiveMirror(iotests.QMPTestCase):
self.vm = iotests.VM()
self.vm.add_drive_raw(self.vm.qmp_to_opts(blk_source))
self.vm.add_blockdev(self.vm.qmp_to_opts(blk_target))
- self.vm.add_device('virtio-blk,drive=source')
+ self.vm.add_device('virtio-blk,id=vblk,drive=source')
self.vm.launch()
def tearDown(self):
@@ -192,6 +195,181 @@ class TestActiveMirror(iotests.QMPTestCase):
self.potential_writes_in_flight = False
+class TestThrottledWithNbdExport(iotests.QMPTestCase):
+ image_len = 128 * 1024 * 1024 # MB
+ iops = 16
+ background_processes: List['subprocess.Popen[str]'] = []
+
+ def setUp(self):
+ qemu_img('create', '-f', iotests.imgfmt, source_img, '128M')
+ qemu_img('create', '-f', iotests.imgfmt, target_img, '128M')
+
+ self.vm = iotests.VM()
+ self.vm.launch()
+
+ result = self.vm.qmp('object-add', **{
+ 'qom-type': 'throttle-group',
+ 'id': 'thrgr',
+ 'limits': {
+ 'iops-total': self.iops,
+ 'iops-total-max': self.iops
+ }
+ })
+ self.assert_qmp(result, 'return', {})
+
+ result = self.vm.qmp('blockdev-add', **{
+ 'node-name': 'source-node',
+ 'driver': 'throttle',
+ 'throttle-group': 'thrgr',
+ 'file': {
+ 'driver': iotests.imgfmt,
+ 'file': {
+ 'driver': 'file',
+ 'filename': source_img
+ }
+ }
+ })
+ self.assert_qmp(result, 'return', {})
+
+ result = self.vm.qmp('blockdev-add', **{
+ 'node-name': 'target-node',
+ 'driver': iotests.imgfmt,
+ 'file': {
+ 'driver': 'file',
+ 'filename': target_img
+ }
+ })
+ self.assert_qmp(result, 'return', {})
+
+ self.nbd_sock = iotests.file_path('nbd.sock',
+ base_dir=iotests.sock_dir)
+ self.nbd_url = f'nbd+unix:///source-node?socket={self.nbd_sock}'
+
+ result = self.vm.qmp('nbd-server-start', addr={
+ 'type': 'unix',
+ 'data': {
+ 'path': self.nbd_sock
+ }
+ })
+ self.assert_qmp(result, 'return', {})
+
+ result = self.vm.qmp('block-export-add', id='exp0', type='nbd',
+ node_name='source-node', writable=True)
+ self.assert_qmp(result, 'return', {})
+
+ def tearDown(self):
+ # Wait for background requests to settle
+ try:
+ while True:
+ p = self.background_processes.pop()
+ while True:
+ try:
+ p.wait(timeout=0.0)
+ break
+ except subprocess.TimeoutExpired:
+ self.vm.qtest(f'clock_step {1 * 1000 * 1000 * 1000}')
+ except IndexError:
+ pass
+
+ # Cancel ongoing block jobs
+ for job in self.vm.qmp('query-jobs')['return']:
+ self.vm.qmp('block-job-cancel', device=job['id'], force=True)
+
+ while True:
+ self.vm.qtest(f'clock_step {1 * 1000 * 1000 * 1000}')
+ if len(self.vm.qmp('query-jobs')['return']) == 0:
+ break
+
+ self.vm.shutdown()
+ os.remove(source_img)
+ os.remove(target_img)
+
+ def testUnderLoad(self):
+ '''
+ Throttle the source node, then issue a whole bunch of external requests
+ while the mirror job (in write-blocking mode) is running. We want to
+ see background requests being issued even while the source is under
+ full load by active writes, so that progress can be made towards READY.
+ '''
+
+ # Fill the first half of the source image; do not fill the second half,
+ # that is where we will have active requests occur. This ensures that
+ # active mirroring itself will not directly contribute to the job's
+ # progress (because when the job was started, those areas were not
+ # intended to be copied, so active mirroring will only lead to not
+ # losing progress, but also not making any).
+ self.vm.hmp_qemu_io('source-node',
+ f'aio_write -P 1 0 {self.image_len // 2}')
+ self.vm.qtest(f'clock_step {1 * 1000 * 1000 * 1000}')
+
+ # Launch the mirror job
+ mirror_buf_size = 65536
+ result = self.vm.qmp('blockdev-mirror',
+ job_id='mirror',
+ filter_node_name='mirror-node',
+ device='source-node',
+ target='target-node',
+ sync='full',
+ copy_mode='write-blocking',
+ buf_size=mirror_buf_size)
+ self.assert_qmp(result, 'return', {})
+
+ # We create the external requests via qemu-io processes on the NBD
+ # server. Have their offset start in the middle of the image so they
+ # do not overlap with the background requests (which start from the
+ # beginning).
+ active_request_offset = self.image_len // 2
+ active_request_len = 4096
+
+ # Create enough requests to saturate the node for 5 seconds
+ for _ in range(0, 5 * self.iops):
+ req = f'write -P 42 {active_request_offset} {active_request_len}'
+ active_request_offset += active_request_len
+ p = iotests.qemu_io_popen('-f', 'nbd', self.nbd_url, '-c', req)
+ self.background_processes += [p]
+
+ # Now advance the clock one I/O operation at a time by the 4 seconds
+ # (i.e. one less than 5). We expect the mirror job to issue background
+ # operations here, even though active requests are still in flight.
+ # The active requests will take precedence, however, because they have
+ # been issued earlier than mirror's background requests.
+ # Once the active requests we have started above are done (i.e. after 5
+ # virtual seconds), we expect those background requests to be worked
+ # on. We only advance 4 seconds here to avoid race conditions.
+ for _ in range(0, 4 * self.iops):
+ step = math.ceil(1 * 1000 * 1000 * 1000 / self.iops)
+ self.vm.qtest(f'clock_step {step}')
+
+ # Note how much remains to be done until the mirror job is finished
+ job_status = self.vm.qmp('query-jobs')['return'][0]
+ start_remaining = job_status['total-progress'] - \
+ job_status['current-progress']
+
+ # Create a whole bunch of more active requests
+ for _ in range(0, 10 * self.iops):
+ req = f'write -P 42 {active_request_offset} {active_request_len}'
+ active_request_offset += active_request_len
+ p = iotests.qemu_io_popen('-f', 'nbd', self.nbd_url, '-c', req)
+ self.background_processes += [p]
+
+ # Let the clock advance more. After 1 second, as noted above, we
+ # expect the background requests to be worked on. Give them a couple
+ # of seconds (specifically 4) to see their impact.
+ for _ in range(0, 5 * self.iops):
+ step = math.ceil(1 * 1000 * 1000 * 1000 / self.iops)
+ self.vm.qtest(f'clock_step {step}')
+
+ # Note how much remains to be done now. We expect this number to be
+ # reduced thanks to those background requests.
+ job_status = self.vm.qmp('query-jobs')['return'][0]
+ end_remaining = job_status['total-progress'] - \
+ job_status['current-progress']
+
+ # See that indeed progress was being made on the job, even while the
+ # node was saturated with active requests
+ self.assertGreater(start_remaining - end_remaining, 0)
+
+
if __name__ == '__main__':
iotests.main(supported_fmts=['qcow2', 'raw'],
supported_protocols=['file'])
diff --git a/tests/qemu-iotests/151.out b/tests/qemu-iotests/151.out
index 89968f35d7..914e3737bd 100644
--- a/tests/qemu-iotests/151.out
+++ b/tests/qemu-iotests/151.out
@@ -1,5 +1,5 @@
-....
+.....
----------------------------------------------------------------------
-Ran 4 tests
+Ran 5 tests
OK
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH for-7.2 4/5] iotests/151: Test that active mirror progresses
2022-11-09 16:54 ` [PATCH for-7.2 4/5] iotests/151: Test that active mirror progresses Hanna Reitz
@ 2022-11-10 12:33 ` Kevin Wolf
0 siblings, 0 replies; 12+ messages in thread
From: Kevin Wolf @ 2022-11-10 12:33 UTC (permalink / raw)
To: Hanna Reitz
Cc: qemu-block, qemu-devel, John Snow, Vladimir Sementsov-Ogievskiy
Am 09.11.2022 um 17:54 hat Hanna Reitz geschrieben:
> Before this series, a mirror job in write-blocking mode would pause
> issuing background requests while active requests are in flight. Thus,
> if the source is constantly in use by active requests, no actual
> progress can be made.
>
> This series should have fixed that, making the mirror job issue
> background requests even while active requests are in flight.
>
> Have a new test case in 151 verify this.
>
> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH for-7.2 5/5] iotests/151: Test active requests on mirror start
2022-11-09 16:54 [PATCH for-7.2 0/5] block/mirror: Do not wait for active writes Hanna Reitz
` (3 preceding siblings ...)
2022-11-09 16:54 ` [PATCH for-7.2 4/5] iotests/151: Test that active mirror progresses Hanna Reitz
@ 2022-11-09 16:54 ` Hanna Reitz
2022-11-10 12:33 ` Kevin Wolf
4 siblings, 1 reply; 12+ messages in thread
From: Hanna Reitz @ 2022-11-09 16:54 UTC (permalink / raw)
To: qemu-block
Cc: qemu-devel, Hanna Reitz, Kevin Wolf, John Snow,
Vladimir Sementsov-Ogievskiy
Have write requests happen to the source node right when we start a
mirror job. The mirror filter node may encounter MirrorBDSOpaque.job
being NULL, but this should not cause a segfault.
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
tests/qemu-iotests/151 | 53 +++++++++++++++++++++++++++++++++++---
tests/qemu-iotests/151.out | 4 +--
2 files changed, 52 insertions(+), 5 deletions(-)
diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
index 0a052e5050..b4d1bc2553 100755
--- a/tests/qemu-iotests/151
+++ b/tests/qemu-iotests/151
@@ -22,7 +22,8 @@
import math
import os
import subprocess
-from typing import List
+import time
+from typing import List, Optional
import iotests
from iotests import qemu_img
@@ -195,12 +196,15 @@ class TestActiveMirror(iotests.QMPTestCase):
self.potential_writes_in_flight = False
-class TestThrottledWithNbdExport(iotests.QMPTestCase):
+class TestThrottledWithNbdExportBase(iotests.QMPTestCase):
image_len = 128 * 1024 * 1024 # MB
- iops = 16
+ iops: Optional[int] = None
background_processes: List['subprocess.Popen[str]'] = []
def setUp(self):
+ # Must be set by subclasses
+ self.assertIsNotNone(self.iops)
+
qemu_img('create', '-f', iotests.imgfmt, source_img, '128M')
qemu_img('create', '-f', iotests.imgfmt, target_img, '128M')
@@ -284,6 +288,10 @@ class TestThrottledWithNbdExport(iotests.QMPTestCase):
os.remove(source_img)
os.remove(target_img)
+
+class TestLowThrottledWithNbdExport(TestThrottledWithNbdExportBase):
+ iops = 16
+
def testUnderLoad(self):
'''
Throttle the source node, then issue a whole bunch of external requests
@@ -370,6 +378,45 @@ class TestThrottledWithNbdExport(iotests.QMPTestCase):
self.assertGreater(start_remaining - end_remaining, 0)
+class TestHighThrottledWithNbdExport(TestThrottledWithNbdExportBase):
+ iops = 1024
+
+ def testActiveOnCreation(self):
+ '''
+ Issue requests on the mirror source node right as the mirror is
+ instated. It's possible that requests occur before the actual job is
+ created, but after the node has been put into the graph. Write
+ requests across the node must in that case be forwarded to the source
+ node without attempting to mirror them (there is no job object yet, so
+ attempting to access it would cause a segfault).
+ We do this with a lightly throttled node (i.e. quite high IOPS limit).
+ Using throttling seems to increase reproductivity, but if the limit is
+ too low, all requests allowed per second will be submitted before
+ mirror_start_job() gets to the problematic point.
+ '''
+
+ # Let qemu-img bench create write requests (enough for two seconds on
+ # the virtual clock)
+ bench_args = ['bench', '-w', '-d', '1024', '-f', 'nbd',
+ '-c', str(self.iops * 2), self.nbd_url]
+ p = iotests.qemu_tool_popen(iotests.qemu_img_args + bench_args)
+ self.background_processes += [p]
+
+ # Give qemu-img bench time to start up and issue requests
+ time.sleep(1.0)
+ # Flush the request queue, so new requests can come in right as we
+ # start blockdev-mirror
+ self.vm.qtest(f'clock_step {1 * 1000 * 1000 * 1000}')
+
+ result = self.vm.qmp('blockdev-mirror',
+ job_id='mirror',
+ device='source-node',
+ target='target-node',
+ sync='full',
+ copy_mode='write-blocking')
+ self.assert_qmp(result, 'return', {})
+
+
if __name__ == '__main__':
iotests.main(supported_fmts=['qcow2', 'raw'],
supported_protocols=['file'])
diff --git a/tests/qemu-iotests/151.out b/tests/qemu-iotests/151.out
index 914e3737bd..3f8a935a08 100644
--- a/tests/qemu-iotests/151.out
+++ b/tests/qemu-iotests/151.out
@@ -1,5 +1,5 @@
-.....
+......
----------------------------------------------------------------------
-Ran 5 tests
+Ran 6 tests
OK
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread