* [Qemu-devel] [PATCH v4] throttle-groups: drain before detaching ThrottleState
@ 2017-11-10 15:19 Stefan Hajnoczi
2017-11-13 10:00 ` Alberto Garcia
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Stefan Hajnoczi @ 2017-11-10 15:19 UTC (permalink / raw)
To: qemu-devel
Cc: Max Reitz, Kevin Wolf, qemu-block, Manos Pitsidianakis,
Alberto Garcia, Stefan Hajnoczi
I/O requests hang after stop/cont commands at least since QEMU 2.10.0
with -drive iops=100:
(guest)$ dd if=/dev/zero of=/dev/vdb oflag=direct count=1000
(qemu) stop
(qemu) cont
...I/O is stuck...
This happens because blk_set_aio_context() detaches the ThrottleState
while requests may still be in flight:
if (tgm->throttle_state) {
throttle_group_detach_aio_context(tgm);
throttle_group_attach_aio_context(tgm, new_context);
}
This patch encloses the detach/attach calls in a drained region so no
I/O request is left hanging. Also add assertions so we don't make the
same mistake again in the future.
Reported-by: Yongxue Hong <yhong@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
v4:
* Simplified patch in response to Berto's review
---
block/block-backend.c | 2 ++
block/throttle-groups.c | 6 ++++++
2 files changed, 8 insertions(+)
diff --git a/block/block-backend.c b/block/block-backend.c
index 45d9101be3..da2f6c0f8a 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -1748,8 +1748,10 @@ void blk_set_aio_context(BlockBackend *blk, AioContext *new_context)
if (bs) {
if (tgm->throttle_state) {
+ bdrv_drained_begin(bs);
throttle_group_detach_aio_context(tgm);
throttle_group_attach_aio_context(tgm, new_context);
+ bdrv_drained_end(bs);
}
bdrv_set_aio_context(bs, new_context);
}
diff --git a/block/throttle-groups.c b/block/throttle-groups.c
index b291a88481..2587f19ca3 100644
--- a/block/throttle-groups.c
+++ b/block/throttle-groups.c
@@ -594,6 +594,12 @@ void throttle_group_attach_aio_context(ThrottleGroupMember *tgm,
void throttle_group_detach_aio_context(ThrottleGroupMember *tgm)
{
ThrottleTimers *tt = &tgm->throttle_timers;
+
+ /* Requests must have been drained */
+ assert(tgm->pending_reqs[0] == 0 && tgm->pending_reqs[1] == 0);
+ assert(qemu_co_queue_empty(&tgm->throttled_reqs[0]));
+ assert(qemu_co_queue_empty(&tgm->throttled_reqs[1]));
+
throttle_timers_detach_aio_context(tt);
tgm->aio_context = NULL;
}
--
2.13.6
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] [PATCH v4] throttle-groups: drain before detaching ThrottleState
2017-11-10 15:19 [Qemu-devel] [PATCH v4] throttle-groups: drain before detaching ThrottleState Stefan Hajnoczi
@ 2017-11-13 10:00 ` Alberto Garcia
2017-11-13 14:12 ` Stefan Hajnoczi
2017-11-13 14:29 ` Alberto Garcia
2 siblings, 0 replies; 5+ messages in thread
From: Alberto Garcia @ 2017-11-13 10:00 UTC (permalink / raw)
To: Stefan Hajnoczi, qemu-devel
Cc: Max Reitz, Kevin Wolf, qemu-block, Manos Pitsidianakis
On Fri 10 Nov 2017 04:19:34 PM CET, Stefan Hajnoczi wrote:
> I/O requests hang after stop/cont commands at least since QEMU 2.10.0
> with -drive iops=100:
>
> (guest)$ dd if=/dev/zero of=/dev/vdb oflag=direct count=1000
> (qemu) stop
> (qemu) cont
> ...I/O is stuck...
>
> This happens because blk_set_aio_context() detaches the ThrottleState
> while requests may still be in flight:
>
> if (tgm->throttle_state) {
> throttle_group_detach_aio_context(tgm);
> throttle_group_attach_aio_context(tgm, new_context);
> }
>
> This patch encloses the detach/attach calls in a drained region so no
> I/O request is left hanging. Also add assertions so we don't make the
> same mistake again in the future.
>
> Reported-by: Yongxue Hong <yhong@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Berto
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] [PATCH v4] throttle-groups: drain before detaching ThrottleState
2017-11-10 15:19 [Qemu-devel] [PATCH v4] throttle-groups: drain before detaching ThrottleState Stefan Hajnoczi
2017-11-13 10:00 ` Alberto Garcia
@ 2017-11-13 14:12 ` Stefan Hajnoczi
2017-11-13 14:29 ` Alberto Garcia
2 siblings, 0 replies; 5+ messages in thread
From: Stefan Hajnoczi @ 2017-11-13 14:12 UTC (permalink / raw)
To: qemu-devel
Cc: Max Reitz, Kevin Wolf, qemu-block, Manos Pitsidianakis, Alberto Garcia
[-- Attachment #1: Type: text/plain, Size: 1141 bytes --]
On Fri, Nov 10, 2017 at 03:19:34PM +0000, Stefan Hajnoczi wrote:
> I/O requests hang after stop/cont commands at least since QEMU 2.10.0
> with -drive iops=100:
>
> (guest)$ dd if=/dev/zero of=/dev/vdb oflag=direct count=1000
> (qemu) stop
> (qemu) cont
> ...I/O is stuck...
>
> This happens because blk_set_aio_context() detaches the ThrottleState
> while requests may still be in flight:
>
> if (tgm->throttle_state) {
> throttle_group_detach_aio_context(tgm);
> throttle_group_attach_aio_context(tgm, new_context);
> }
>
> This patch encloses the detach/attach calls in a drained region so no
> I/O request is left hanging. Also add assertions so we don't make the
> same mistake again in the future.
>
> Reported-by: Yongxue Hong <yhong@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> v4:
> * Simplified patch in response to Berto's review
> ---
> block/block-backend.c | 2 ++
> block/throttle-groups.c | 6 ++++++
> 2 files changed, 8 insertions(+)
Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] [PATCH v4] throttle-groups: drain before detaching ThrottleState
2017-11-10 15:19 [Qemu-devel] [PATCH v4] throttle-groups: drain before detaching ThrottleState Stefan Hajnoczi
2017-11-13 10:00 ` Alberto Garcia
2017-11-13 14:12 ` Stefan Hajnoczi
@ 2017-11-13 14:29 ` Alberto Garcia
2017-11-16 11:25 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2 siblings, 1 reply; 5+ messages in thread
From: Alberto Garcia @ 2017-11-13 14:29 UTC (permalink / raw)
To: Stefan Hajnoczi, qemu-devel
Cc: Max Reitz, Kevin Wolf, qemu-block, Manos Pitsidianakis
On Fri 10 Nov 2017 04:19:34 PM CET, Stefan Hajnoczi wrote:
> I/O requests hang after stop/cont commands at least since QEMU 2.10.0
> with -drive iops=100:
>
> (guest)$ dd if=/dev/zero of=/dev/vdb oflag=direct count=1000
> (qemu) stop
> (qemu) cont
> ...I/O is stuck...
>
> This happens because blk_set_aio_context() detaches the ThrottleState
> while requests may still be in flight:
>
> if (tgm->throttle_state) {
> throttle_group_detach_aio_context(tgm);
> throttle_group_attach_aio_context(tgm, new_context);
> }
>
> This patch encloses the detach/attach calls in a drained region so no
> I/O request is left hanging. Also add assertions so we don't make the
> same mistake again in the future.
I'm wondering about the implications of this change... is it possible
now to bypass the I/O limits simply by stopping and quickly resuming the
VM? And is that a problem?
Berto
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH v4] throttle-groups: drain before detaching ThrottleState
2017-11-13 14:29 ` Alberto Garcia
@ 2017-11-16 11:25 ` Stefan Hajnoczi
0 siblings, 0 replies; 5+ messages in thread
From: Stefan Hajnoczi @ 2017-11-16 11:25 UTC (permalink / raw)
To: Alberto Garcia
Cc: Stefan Hajnoczi, qemu-devel, Kevin Wolf, qemu block, Max Reitz
On Mon, Nov 13, 2017 at 2:29 PM, Alberto Garcia <berto@igalia.com> wrote:
> On Fri 10 Nov 2017 04:19:34 PM CET, Stefan Hajnoczi wrote:
>> I/O requests hang after stop/cont commands at least since QEMU 2.10.0
>> with -drive iops=100:
>>
>> (guest)$ dd if=/dev/zero of=/dev/vdb oflag=direct count=1000
>> (qemu) stop
>> (qemu) cont
>> ...I/O is stuck...
>>
>> This happens because blk_set_aio_context() detaches the ThrottleState
>> while requests may still be in flight:
>>
>> if (tgm->throttle_state) {
>> throttle_group_detach_aio_context(tgm);
>> throttle_group_attach_aio_context(tgm, new_context);
>> }
>>
>> This patch encloses the detach/attach calls in a drained region so no
>> I/O request is left hanging. Also add assertions so we don't make the
>> same mistake again in the future.
>
> I'm wondering about the implications of this change... is it possible
> now to bypass the I/O limits simply by stopping and quickly resuming the
> VM? And is that a problem?
bdrv_set_aio_context() already drains so this patch doesn't change
existing behavior with respect to bypassing throttling.
It's not ideal that certain VM lifecycle operations temporarily
disable throttling, but it's a trade-off since synchronous drain is
usually performance sensitive and should not take a long time.
Perhaps there are ways to improve the situation, I haven't studied it
in detail.
Stefan
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-11-16 11:25 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-11-10 15:19 [Qemu-devel] [PATCH v4] throttle-groups: drain before detaching ThrottleState Stefan Hajnoczi
2017-11-13 10:00 ` Alberto Garcia
2017-11-13 14:12 ` Stefan Hajnoczi
2017-11-13 14:29 ` Alberto Garcia
2017-11-16 11:25 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.