From: Greg Kurz <groug@kaod.org> To: qemu-devel@nongnu.org Cc: "Daniel P . Berrangé" <berrange@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>, "Dr. David Alan Gilbert" <dgilbert@redhat.com>, "Greg Kurz" <groug@kaod.org>, virtio-fs@redhat.com, "Stefan Hajnoczi" <stefanha@redhat.com>, "Vivek Goyal" <vgoyal@redhat.com> Subject: [PATCH v2 0/7] virtiofsd: Avoid potential deadlocks Date: Fri, 12 Mar 2021 10:22:05 +0100 [thread overview] Message-ID: <20210312092212.782255-1-groug@kaod.org> (raw) A deadlock condition potentially exists if a vhost-user process needs to request something to QEMU on the slave channel while processing a vhost-user message. This doesn't seem to affect any vhost-user implementation so far, but this is currently biting the upcoming enablement of DAX with virtio-fs. The issue is being observed when the guest does an emergency reboot while a mapping still exits in the DAX window, which is very easy to get with a busy enough workload (e.g. as simulated by blogbench [1]) : - QEMU sends VHOST_USER_GET_VRING_BASE to virtiofsd. - In order to complete the request, virtiofsd then asks QEMU to remove the mapping on the slave channel. All these dialogs are synchronous, hence the deadlock. Another deadlock condition exists in the virtiofsd code itself, related to the use of the vu_dispatch rwlock. This series supercedes Vivek's previous tentative [1] to fix the deadlocks. Basically, instead of introducing new vhost-user protocol message to specifically shutdown the slave channel, this series introduces a nested event loop in vhost_user_read() as suggested by Stefan Hajnoczi. This allows to monitor and service requests on the slave channel while waiting for virtiofsd to answer to a vhost-user request. This was tested with the latest DAX patchset posted to the virtio-fs list [2], rebased on top of the present series [3]. Testing scenario is: 1) start VM with DAX capable vhost-user-fs device 2) mount -o dax in the guest 3) run Blogbench [4] in virtiofs mount 4) wait 10s, which is enough to have active mappings 5) echo b > /proc/sysrq-trigger 6) wait for the guest to reboot and start over from step 2 Without this series, only a couple of reboots ( < 5) are needed to hit a deadlock. With this series applied, an overnight test could reboot the guest 1500+ times without any issues. [1] https://listman.redhat.com/archives/virtio-fs/2021-January/msg00073.html [2] https://listman.redhat.com/archives/virtio-fs/2021-February/msg00058.html [3] Tree with this series + DAX available at: https://gitlab.com/gkurz/qemu/-/commits/virtio-fs-dax-no-deadlock [4] https://github.com/jedisct1/Blogbench Changes since v2: - added some preliminary fixes and cleanups for the slave channel code - re-factored some code and addressed comments (see individual patches) - more testing Greg Kurz (7): vhost-user: Drop misleading EAGAIN checks in slave_read() vhost-user: Fix double-close on slave_read() error path vhost-user: Factor out duplicated slave_fd teardown code vhost-user: Convert slave channel to QIOChannelSocket vhost-user: Introduce nested event loop in vhost_user_read() vhost-user: Monitor slave channel in vhost_user_read() virtiofsd: Release vu_dispatch_lock when stopping queue hw/virtio/vhost-user.c | 217 +++++++++++++++++++++------------- tools/virtiofsd/fuse_virtio.c | 6 + 2 files changed, 144 insertions(+), 79 deletions(-) -- 2.26.2
WARNING: multiple messages have this Message-ID (diff)
From: Greg Kurz <groug@kaod.org> To: qemu-devel@nongnu.org Cc: "Daniel P . Berrangé" <berrange@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>, virtio-fs@redhat.com, "Vivek Goyal" <vgoyal@redhat.com> Subject: [Virtio-fs] [PATCH v2 0/7] virtiofsd: Avoid potential deadlocks Date: Fri, 12 Mar 2021 10:22:05 +0100 [thread overview] Message-ID: <20210312092212.782255-1-groug@kaod.org> (raw) A deadlock condition potentially exists if a vhost-user process needs to request something to QEMU on the slave channel while processing a vhost-user message. This doesn't seem to affect any vhost-user implementation so far, but this is currently biting the upcoming enablement of DAX with virtio-fs. The issue is being observed when the guest does an emergency reboot while a mapping still exits in the DAX window, which is very easy to get with a busy enough workload (e.g. as simulated by blogbench [1]) : - QEMU sends VHOST_USER_GET_VRING_BASE to virtiofsd. - In order to complete the request, virtiofsd then asks QEMU to remove the mapping on the slave channel. All these dialogs are synchronous, hence the deadlock. Another deadlock condition exists in the virtiofsd code itself, related to the use of the vu_dispatch rwlock. This series supercedes Vivek's previous tentative [1] to fix the deadlocks. Basically, instead of introducing new vhost-user protocol message to specifically shutdown the slave channel, this series introduces a nested event loop in vhost_user_read() as suggested by Stefan Hajnoczi. This allows to monitor and service requests on the slave channel while waiting for virtiofsd to answer to a vhost-user request. This was tested with the latest DAX patchset posted to the virtio-fs list [2], rebased on top of the present series [3]. Testing scenario is: 1) start VM with DAX capable vhost-user-fs device 2) mount -o dax in the guest 3) run Blogbench [4] in virtiofs mount 4) wait 10s, which is enough to have active mappings 5) echo b > /proc/sysrq-trigger 6) wait for the guest to reboot and start over from step 2 Without this series, only a couple of reboots ( < 5) are needed to hit a deadlock. With this series applied, an overnight test could reboot the guest 1500+ times without any issues. [1] https://listman.redhat.com/archives/virtio-fs/2021-January/msg00073.html [2] https://listman.redhat.com/archives/virtio-fs/2021-February/msg00058.html [3] Tree with this series + DAX available at: https://gitlab.com/gkurz/qemu/-/commits/virtio-fs-dax-no-deadlock [4] https://github.com/jedisct1/Blogbench Changes since v2: - added some preliminary fixes and cleanups for the slave channel code - re-factored some code and addressed comments (see individual patches) - more testing Greg Kurz (7): vhost-user: Drop misleading EAGAIN checks in slave_read() vhost-user: Fix double-close on slave_read() error path vhost-user: Factor out duplicated slave_fd teardown code vhost-user: Convert slave channel to QIOChannelSocket vhost-user: Introduce nested event loop in vhost_user_read() vhost-user: Monitor slave channel in vhost_user_read() virtiofsd: Release vu_dispatch_lock when stopping queue hw/virtio/vhost-user.c | 217 +++++++++++++++++++++------------- tools/virtiofsd/fuse_virtio.c | 6 + 2 files changed, 144 insertions(+), 79 deletions(-) -- 2.26.2
next reply other threads:[~2021-03-12 9:24 UTC|newest] Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-03-12 9:22 Greg Kurz [this message] 2021-03-12 9:22 ` [Virtio-fs] [PATCH v2 0/7] virtiofsd: Avoid potential deadlocks Greg Kurz 2021-03-12 9:22 ` [PATCH v2 1/7] vhost-user: Drop misleading EAGAIN checks in slave_read() Greg Kurz 2021-03-12 9:22 ` [Virtio-fs] " Greg Kurz 2021-03-15 10:34 ` Stefan Hajnoczi 2021-03-15 10:34 ` [Virtio-fs] " Stefan Hajnoczi 2021-03-12 9:22 ` [PATCH v2 2/7] vhost-user: Fix double-close on slave_read() error path Greg Kurz 2021-03-12 9:22 ` [Virtio-fs] " Greg Kurz 2021-03-15 10:36 ` Stefan Hajnoczi 2021-03-15 10:36 ` [Virtio-fs] " Stefan Hajnoczi 2021-03-12 9:22 ` [PATCH v2 3/7] vhost-user: Factor out duplicated slave_fd teardown code Greg Kurz 2021-03-12 9:22 ` [Virtio-fs] " Greg Kurz 2021-03-15 10:36 ` Stefan Hajnoczi 2021-03-15 10:36 ` [Virtio-fs] " Stefan Hajnoczi 2021-03-12 9:22 ` [PATCH v2 4/7] vhost-user: Convert slave channel to QIOChannelSocket Greg Kurz 2021-03-12 9:22 ` [Virtio-fs] " Greg Kurz 2021-03-15 10:37 ` Stefan Hajnoczi 2021-03-15 10:37 ` [Virtio-fs] " Stefan Hajnoczi 2021-03-12 9:22 ` [PATCH v2 5/7] vhost-user: Introduce nested event loop in vhost_user_read() Greg Kurz 2021-03-12 9:22 ` [Virtio-fs] " Greg Kurz 2021-03-15 10:38 ` Stefan Hajnoczi 2021-03-15 10:38 ` [Virtio-fs] " Stefan Hajnoczi 2021-03-12 9:22 ` [PATCH v2 6/7] vhost-user: Monitor slave channel " Greg Kurz 2021-03-12 9:22 ` [Virtio-fs] " Greg Kurz 2021-03-15 12:20 ` Stefan Hajnoczi 2021-03-15 12:20 ` [Virtio-fs] " Stefan Hajnoczi 2021-03-12 9:22 ` [PATCH v2 7/7] virtiofsd: Release vu_dispatch_lock when stopping queue Greg Kurz 2021-03-12 9:22 ` [Virtio-fs] " Greg Kurz 2021-03-15 14:57 ` Dr. David Alan Gilbert 2021-03-15 14:57 ` [Virtio-fs] " Dr. David Alan Gilbert
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210312092212.782255-1-groug@kaod.org \ --to=groug@kaod.org \ --cc=berrange@redhat.com \ --cc=dgilbert@redhat.com \ --cc=mst@redhat.com \ --cc=qemu-devel@nongnu.org \ --cc=stefanha@redhat.com \ --cc=vgoyal@redhat.com \ --cc=virtio-fs@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.