From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 48F9EC4167D for ; Mon, 13 Nov 2023 14:40:19 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r2Y5O-0005xX-F2; Mon, 13 Nov 2023 09:38:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r2Y5H-0005wT-PG; Mon, 13 Nov 2023 09:38:42 -0500 Received: from proxmox-new.maurer-it.com ([94.136.29.106]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r2Y5E-0001WS-Ob; Mon, 13 Nov 2023 09:38:39 -0500 Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id B052441D78; Mon, 13 Nov 2023 15:38:25 +0100 (CET) Message-ID: Date: Mon, 13 Nov 2023 15:38:24 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PULL 29/32] virtio-blk: implement BlockDevOps->drained_begin() From: Fiona Ebner To: qemu-block@nongnu.org, Kevin Wolf , Stefan Hajnoczi Cc: richard.henderson@linaro.org, qemu-devel@nongnu.org, Thomas Lamprecht , Hanna Reitz , Paolo Bonzini , Fam Zheng , "Michael S . Tsirkin" References: <20230530163239.576632-1-kwolf@redhat.com> <20230530163239.576632-30-kwolf@redhat.com> Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=94.136.29.106; envelope-from=f.ebner@proxmox.com; helo=proxmox-new.maurer-it.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Am 03.11.23 um 14:12 schrieb Fiona Ebner: > Hi, > > Am 30.05.23 um 18:32 schrieb Kevin Wolf: >> From: Stefan Hajnoczi >> >> Detach ioeventfds during drained sections to stop I/O submission from >> the guest. virtio-blk is no longer reliant on aio_disable_external() >> after this patch. This will allow us to remove the >> aio_disable_external() API once all other code that relies on it is >> converted. >> >> Take extra care to avoid attaching/detaching ioeventfds if the data >> plane is started/stopped during a drained section. This should be rare, >> but maybe the mirror block job can trigger it. >> >> Signed-off-by: Stefan Hajnoczi >> Message-Id: <20230516190238.8401-18-stefanha@redhat.com> >> Signed-off-by: Kevin Wolf > > I ran into a strange issue where guest IO would get completely stuck > during certain block jobs a while ago and finally managed to find a > small reproducer [0]. I'm using a VM with virtio-blk-pci (or > virtio-scsi-pci) with an iothread and running > > fio --name=file --size=100M --direct=1 --rw=randwrite --bs=4k > --ioengine=psync --numjobs=5 --runtime=1200 --time_based > > in the guest. Then I'm issuing the QMP command with the reproducer in a > loop. Usually, the guest IO will get stuck after about 1-3 minutes, > sometimes fio can manage to continue with a lower speed for a while (but > trying to Ctrl+C it or doing other IO in the guest will already be > broken), which I guess could be a hint that it's an issue with notifiers? > > Bisecting (to declare a commit good, I waited 10 minutes) led me to this > patch, i.e. commit 1665d9326f ("virtio-blk: implement > BlockDevOps->drained_begin()") and for SCSI, I verified that the issue > similarly starts happening after 766aa2de0f ("virtio-scsi: implement > BlockDevOps->drained_begin()"). > > Both issues are still present on current master (i.e. 1c98a821a2 > ("tests/qtest: Introduce tests for AMD/Xilinx Versal TRNG device")) > > Happy to provide more information and hints about how to debug the issue > further. > Of course, I meant "and for hints" ;) I should also mention that when IO is stuck, for the two BlockDriverStates (i.e. bdrv_raw and bdrv_file) and BlockBackend, in_flight and quiesce_counter are 0, tracked_requests, respectively queued_requests, are empty and quiesced_parent is false for the parents. Two observations: 1. I found that using QMP 'stop' and 'cont' will allow guest IO to get unstuck. I'm pretty sure, it's the virtio_blk_data_plane_stop/start calls it triggers. 2. While experimenting, I found that after the below change [1] in aio_poll(), I wasn't able to trigger the issue anymore (letting my reproducer run for 40 minutes). Best Regards, Fiona [1]: > diff --git a/util/aio-posix.c b/util/aio-posix.c > index 7f2c99729d..dff9ad4148 100644 > --- a/util/aio-posix.c > +++ b/util/aio-posix.c > @@ -655,7 +655,7 @@ bool aio_poll(AioContext *ctx, bool blocking) > /* If polling is allowed, non-blocking aio_poll does not need the > * system call---a single round of run_poll_handlers_once suffices. > */ > - if (timeout || ctx->fdmon_ops->need_wait(ctx)) { > + if (1) { //timeout || ctx->fdmon_ops->need_wait(ctx)) { > /* > * Disable poll mode. poll mode should be disabled before the call > * of ctx->fdmon_ops->wait() so that guest's notification can wake > [0]: > >> diff --git a/blockdev.c b/blockdev.c >> index db2725fe74..bf2e0fc22c 100644 >> --- a/blockdev.c >> +++ b/blockdev.c >> @@ -2986,6 +2986,11 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) >> bool zero_target; >> int ret; >> >> + bdrv_drain_all_begin(); >> + bdrv_drain_all_end(); >> + return; >> + >> + >> bs = qmp_get_root_bs(arg->device, errp); >> if (!bs) { >> return; > > >