From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79B1EC11F64 for ; Thu, 1 Jul 2021 18:58:43 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B558261410 for ; Thu, 1 Jul 2021 18:58:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B558261410 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=canonical.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:45114 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lz1tZ-0005ul-JA for qemu-devel@archiver.kernel.org; Thu, 01 Jul 2021 14:58:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59406) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lz0q9-0002zU-PX for qemu-devel@nongnu.org; Thu, 01 Jul 2021 13:51:05 -0400 Received: from youngberry.canonical.com ([91.189.89.112]:53036) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.90_1) (envelope-from ) id 1lz0q6-00034m-Ar for qemu-devel@nongnu.org; Thu, 01 Jul 2021 13:51:05 -0400 Received: from mail-oi1-f197.google.com ([209.85.167.197]) by youngberry.canonical.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1lz0pz-0005Cq-2i for qemu-devel@nongnu.org; Thu, 01 Jul 2021 17:50:55 +0000 Received: by mail-oi1-f197.google.com with SMTP id b185-20020acab2c20000b02901f6dd5f89fbso3643177oif.10 for ; Thu, 01 Jul 2021 10:50:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0hb5jXhUNe8PiwGYTA7IRxh9VDG1fm/eOMUKBPvugec=; b=hRd7Hme3N46gcZGXY3ZXLcldWnMKF/gruXs+KwZLN+swAAoM8vTQsWkxD1ybIINLKT cZynSoQLFP5SwmgYM5h2NWPXX+b18ocT3kwOcagNpqhfDfcpRQ3H3W7ZnT7pJRKH+s7w BbI/zi/oAZ3Yz6kf/NzNxBmHOsSKYCeqUN5WbiRUBMWDzrwvLg2FnACPhZwEpz7u6/7c toa78qisl7MUThiaQfFZ9JXA0W/0U6B8PUx5R2lrJ4McTskHFqxpIDuSRU4wjMj8ECr/ C2gkNIROgivcIzCUdHf3HI+4AUAAugYMsOI14CIBBKCmVzRWgX8FQN9ZWwpba2YrjSSJ 979A== X-Gm-Message-State: AOAM5308NLBg61xQYLgN5b8AJWWI9VSY7KBEXdboZq2FwVxzsT2FX96u MdoNMpGHqeRa4MlCuVWOJCHLSYkOOLtD5+uAUoH8yReZHb0iXqjdJENnzW+v/BlLV2V/HK15Vks p9z8Q/5FucUuirAMChYpQzPC5dJzrPLb68c5OCTZY7Rs0VuE= X-Received: by 2002:a9d:1b05:: with SMTP id l5mr942255otl.335.1625161853296; Thu, 01 Jul 2021 10:50:53 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx0OedRYuOmfEiY7u/3GTDlRrzzs4zEuJPhDKNZri8Mxvg3JzXm5Wh5j5hzyBJml2bI6jt46uxh4T1BnVi8qqA= X-Received: by 2002:a9d:1b05:: with SMTP id l5mr942241otl.335.1625161853037; Thu, 01 Jul 2021 10:50:53 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Kamal Mostafa Date: Thu, 1 Jul 2021 10:50:37 -0700 Message-ID: Subject: Re: Possible io_uring regression with QEMU on Ubuntu's kernel To: Juhyung Park Content-Type: multipart/alternative; boundary="000000000000f4066b05c6137a9f" Received-SPF: none client-ip=91.189.89.112; envelope-from=kamal.mostafa@canonical.com; helo=youngberry.canonical.com X-Spam_score_int: -68 X-Spam_score: -6.9 X-Spam_bar: ------ X-Spam_report: (-6.9 / 5.0 requ) BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Thu, 01 Jul 2021 14:57:57 -0400 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jens Axboe , Kamal Mostafa , qemu-devel@nongnu.org, Stefan Bader , Ubuntu Kernel Team , io-uring , Stefano Garzarella Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --000000000000f4066b05c6137a9f Content-Type: text/plain; charset="UTF-8" Hi- Thanks very much for reporting this. We picked up that patch ("io_uring: don't mark S_ISBLK async work as unbounded") for our Ubuntu v5.8 kernel from linux-stable/v5.10.31. Since it's not clear that it's appropriate for v5.8 (or even v5.10-stable?) we'll revert it from Ubuntu v5.8 if you can confirm that actually fixes the problem. Here's a test build of that (5.8.0-59 with that commit reverted). The full set of packages is provided, but you probably only actually need to install the linux-image and linux-modules[-extra] deb's. We'll stand by for your results: https://kernel.ubuntu.com/~kamal/uringrevert0/ Thanks again, -Kamal Mostafa (Canonical Kernel Team) On Wed, Jun 30, 2021 at 1:47 AM Juhyung Park wrote: > Hi everyone. > > With the latest Ubuntu 20.04's HWE kernel 5.8.0-59, I'm noticing some > weirdness when using QEMU/libvirt with the following storage > configuration: > > > discard="unmap" detect_zeroes="unmap"/> > dev="/dev/disk/by-id/md-uuid-df271a1e:9dfb7edb:8dc4fbb8:c43e652f-part1" > index="1"/> > > > >
function="0x0"/> > > > QEMU version is 5.2+dfsg-9ubuntu3 and libvirt version is 7.0.0-2ubuntu2. > > The guest VM is unable to handle I/O properly with io_uring, and > nuking io="io_uring" fixes the issue. > On one machine (EPYC 7742), the partition table cannot be read and on > another (Ryzen 9 3950X), ext4 detects weirdness with journaling and > ultimately remounts the guest disk to R/O: > > [ 2.712321] virtio_blk virtio5: [vda] 3906519775 512-byte logical > blocks (2.00 TB/1.82 TiB) > [ 2.714054] vda: detected capacity change from 0 to 2000138124800 > [ 2.963671] blk_update_request: I/O error, dev vda, sector 0 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.964909] Buffer I/O error on dev vda, logical block 0, async page > read > [ 2.966021] blk_update_request: I/O error, dev vda, sector 1 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.967177] Buffer I/O error on dev vda, logical block 1, async page > read > [ 2.968330] blk_update_request: I/O error, dev vda, sector 2 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.969504] Buffer I/O error on dev vda, logical block 2, async page > read > [ 2.970767] blk_update_request: I/O error, dev vda, sector 3 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.971624] Buffer I/O error on dev vda, logical block 3, async page > read > [ 2.972170] blk_update_request: I/O error, dev vda, sector 4 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.972728] Buffer I/O error on dev vda, logical block 4, async page > read > [ 2.973308] blk_update_request: I/O error, dev vda, sector 5 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.973920] Buffer I/O error on dev vda, logical block 5, async page > read > [ 2.974496] blk_update_request: I/O error, dev vda, sector 6 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.975093] Buffer I/O error on dev vda, logical block 6, async page > read > [ 2.975685] blk_update_request: I/O error, dev vda, sector 7 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.976295] Buffer I/O error on dev vda, logical block 7, async page > read > [ 2.980074] blk_update_request: I/O error, dev vda, sector 0 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.981104] Buffer I/O error on dev vda, logical block 0, async page > read > [ 2.981786] blk_update_request: I/O error, dev vda, sector 1 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 2.982083] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue count > = 63, Tx Queue count = 63 XDP Queue count = 0 > [ 2.982442] Buffer I/O error on dev vda, logical block 1, async page > read > [ 2.983642] ldm_validate_partition_table(): Disk read failed. > > Kernel 5.8.0-55 is fine, and the only io_uring-related change between > 5.8.0-55 and 5.8.0-59 is the commit 4b982bd0f383 ("io_uring: don't > mark S_ISBLK async work as unbounded"). > > The weird thing is that this commit was first introduced with v5.12, > but neither the mainline v5.12.0 or v5.13.0 is affected by this issue. > > I guess one of these commits following the backported commit from > v5.12 fixes the issue, but that's just a guess. It might be another > earlier commit: > c7d95613c7d6 io_uring: fix early sqd_list removal sqpoll hangs > 9728463737db io_uring: fix rw req completion > 6ad7f2332e84 io_uring: clear F_REISSUE right after getting it > e82ad4853948 io_uring: fix !CONFIG_BLOCK compilation failure > 230d50d448ac io_uring: move reissue into regular IO path > 07204f21577a io_uring: fix EIOCBQUEUED iter revert > 696ee88a7c50 io_uring/io-wq: protect against sprintf overflow > > It would be much appreciated if Jens could give pointers to Canonical > developers on how to fix the issue, and hopefully a suggestion to > prevent this from happening again. > > Thanks, > Regards > --000000000000f4066b05c6137a9f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi-

Thanks very much for reportin= g this.=C2=A0 We picked up that patch ("io_uring: don't mark S_ISB= LK async work as unbounded") for our Ubuntu v5.8 kernel from linux-sta= ble/v5.10.31.=C2=A0 Since it's not clear that it's appropriate for = v5.8 (or even v5.10-stable?) we'll revert it from Ubuntu v5.8 if you ca= n confirm=C2=A0that actually fixes the problem.

Here'= ;s a test build of that (5.8.0-59 with that commit reverted).=C2=A0 The ful= l set of packages is provided, but you probably only actually need to insta= ll the linux-image and linux-modules[-extra] deb's. We'll stand by = for your results:
Thanks again,

=C2=A0-Kamal Mostafa (C= anonical Kernel Team)

On Wed, Jun 30, 2021 at 1:47 AM Juhyung Park = <qkrwngud825@gmail.com> = wrote:
Hi everyo= ne.

With the latest Ubuntu 20.04's HWE kernel 5.8.0-59, I'm noticing so= me
weirdness when using QEMU/libvirt with the following storage
configuration:

<disk type=3D"block" device=3D"disk">
=C2=A0 <driver name=3D"qemu" type=3D"raw" cache=3D&q= uot;none" io=3D"io_uring"
discard=3D"unmap" detect_zeroes=3D"unmap"/>
=C2=A0 <source dev=3D"/dev/disk/by-id/md-uuid-df271a1e:9dfb7edb:8dc= 4fbb8:c43e652f-part1"
index=3D"1"/>
=C2=A0 <backingStore/>
=C2=A0 <target dev=3D"vda" bus=3D"virtio"/>
=C2=A0 <alias name=3D"virtio-disk0"/>
=C2=A0 <address type=3D"pci" domain=3D"0x0000" bus= =3D"0x07" slot=3D"0x00" function=3D"0x0"/>=
</disk>

QEMU version is 5.2+dfsg-9ubuntu3 and libvirt version is 7.0.0-2ubuntu2.
The guest VM is unable to handle I/O properly with io_uring, and
nuking io=3D"io_uring" fixes the issue.
On one machine (EPYC 7742), the partition table cannot be read and on
another (Ryzen 9 3950X), ext4 detects weirdness with journaling and
ultimately remounts the guest disk to R/O:

[=C2=A0 =C2=A0 2.712321] virtio_blk virtio5: [vda] 3906519775 512-byte logi= cal
blocks (2.00 TB/1.82 TiB)
[=C2=A0 =C2=A0 2.714054] vda: detected capacity change from 0 to 2000138124= 800
[=C2=A0 =C2=A0 2.963671] blk_update_request: I/O error, dev vda, sector 0 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.964909] Buffer I/O error on dev vda, logical block 0, asyn= c page read
[=C2=A0 =C2=A0 2.966021] blk_update_request: I/O error, dev vda, sector 1 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.967177] Buffer I/O error on dev vda, logical block 1, asyn= c page read
[=C2=A0 =C2=A0 2.968330] blk_update_request: I/O error, dev vda, sector 2 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.969504] Buffer I/O error on dev vda, logical block 2, asyn= c page read
[=C2=A0 =C2=A0 2.970767] blk_update_request: I/O error, dev vda, sector 3 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.971624] Buffer I/O error on dev vda, logical block 3, asyn= c page read
[=C2=A0 =C2=A0 2.972170] blk_update_request: I/O error, dev vda, sector 4 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.972728] Buffer I/O error on dev vda, logical block 4, asyn= c page read
[=C2=A0 =C2=A0 2.973308] blk_update_request: I/O error, dev vda, sector 5 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.973920] Buffer I/O error on dev vda, logical block 5, asyn= c page read
[=C2=A0 =C2=A0 2.974496] blk_update_request: I/O error, dev vda, sector 6 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.975093] Buffer I/O error on dev vda, logical block 6, asyn= c page read
[=C2=A0 =C2=A0 2.975685] blk_update_request: I/O error, dev vda, sector 7 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.976295] Buffer I/O error on dev vda, logical block 7, asyn= c page read
[=C2=A0 =C2=A0 2.980074] blk_update_request: I/O error, dev vda, sector 0 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.981104] Buffer I/O error on dev vda, logical block 0, asyn= c page read
[=C2=A0 =C2=A0 2.981786] blk_update_request: I/O error, dev vda, sector 1 o= p
0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[=C2=A0 =C2=A0 2.982083] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue c= ount
=3D 63, Tx Queue count =3D 63 XDP Queue count =3D 0
[=C2=A0 =C2=A0 2.982442] Buffer I/O error on dev vda, logical block 1, asyn= c page read
[=C2=A0 =C2=A0 2.983642] ldm_validate_partition_table(): Disk read failed.<= br>
Kernel 5.8.0-55 is fine, and the only io_uring-related change between
5.8.0-55 and 5.8.0-59 is the commit 4b982bd0f383 ("io_uring: don't=
mark S_ISBLK async work as unbounded").

The weird thing is that this commit was first introduced with v5.12,
but neither the mainline v5.12.0 or v5.13.0 is affected by this issue.

I guess one of these commits following the backported commit from
v5.12 fixes the issue, but that's just a guess. It might be another
earlier commit:
c7d95613c7d6 io_uring: fix early sqd_list removal sqpoll hangs
9728463737db io_uring: fix rw req completion
6ad7f2332e84 io_uring: clear F_REISSUE right after getting it
e82ad4853948 io_uring: fix !CONFIG_BLOCK compilation failure
230d50d448ac io_uring: move reissue into regular IO path
07204f21577a io_uring: fix EIOCBQUEUED iter revert
696ee88a7c50 io_uring/io-wq: protect against sprintf overflow

It would be much appreciated if Jens could give pointers to Canonical
developers on how to fix the issue, and hopefully a suggestion to
prevent this from happening again.

Thanks,
Regards
--000000000000f4066b05c6137a9f--