All of lore.kernel.org
 help / color / mirror / Atom feed
From: Stefan Hajnoczi <stefanha@redhat.com>
To: Greg Kurz <groug@kaod.org>
Cc: "Daniel P . Berrangé" <berrange@redhat.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	qemu-devel@nongnu.org,
	"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
	virtio-fs@redhat.com, "Vivek Goyal" <vgoyal@redhat.com>
Subject: Re: [PATCH v2 5/7] vhost-user: Introduce nested event loop in vhost_user_read()
Date: Mon, 15 Mar 2021 10:38:40 +0000	[thread overview]
Message-ID: <YE85MJl5VNnsN6Y2@stefanha-x1.localdomain> (raw)
In-Reply-To: <20210312092212.782255-6-groug@kaod.org>

[-- Attachment #1: Type: text/plain, Size: 1938 bytes --]

On Fri, Mar 12, 2021 at 10:22:10AM +0100, Greg Kurz wrote:
> A deadlock condition potentially exists if a vhost-user process needs
> to request something to QEMU on the slave channel while processing a
> vhost-user message.
> 
> This doesn't seem to affect any vhost-user implementation so far, but
> this is currently biting the upcoming enablement of DAX with virtio-fs.
> The issue is being observed when the guest does an emergency reboot while
> a mapping still exits in the DAX window, which is very easy to get with
> a busy enough workload (e.g. as simulated by blogbench [1]) :
> 
> - QEMU sends VHOST_USER_GET_VRING_BASE to virtiofsd.
> 
> - In order to complete the request, virtiofsd then asks QEMU to remove
>   the mapping on the slave channel.
> 
> All these dialogs are synchronous, hence the deadlock.
> 
> As pointed out by Stefan Hajnoczi:
> 
> When QEMU's vhost-user master implementation sends a vhost-user protocol
> message, vhost_user_read() does a "blocking" read during which slave_fd
> is not monitored by QEMU.
> 
> The natural solution for this issue is an event loop. The main event
> loop cannot be nested though since we have no guarantees that its
> fd handlers are prepared for re-entrancy.
> 
> Introduce a new event loop that only monitors the chardev I/O for now
> in vhost_user_read() and push the actual reading to a one-shot handler.
> A subsequent patch will teach the loop to monitor and process messages
> from the slave channel as well.
> 
> [1] https://github.com/jedisct1/Blogbench
> 
> Suggested-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Greg Kurz <groug@kaod.org>
> ---
> v2: - Document why a nested loop is needed in vhost_user_read() (Stefan)
> ---
>  hw/virtio/vhost-user.c | 65 ++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 60 insertions(+), 5 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

WARNING: multiple messages have this Message-ID (diff)
From: Stefan Hajnoczi <stefanha@redhat.com>
To: Greg Kurz <groug@kaod.org>
Cc: "Daniel P . Berrangé" <berrange@redhat.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	qemu-devel@nongnu.org, virtio-fs@redhat.com,
	"Vivek Goyal" <vgoyal@redhat.com>
Subject: Re: [Virtio-fs] [PATCH v2 5/7] vhost-user: Introduce nested event loop in vhost_user_read()
Date: Mon, 15 Mar 2021 10:38:40 +0000	[thread overview]
Message-ID: <YE85MJl5VNnsN6Y2@stefanha-x1.localdomain> (raw)
In-Reply-To: <20210312092212.782255-6-groug@kaod.org>

[-- Attachment #1: Type: text/plain, Size: 1938 bytes --]

On Fri, Mar 12, 2021 at 10:22:10AM +0100, Greg Kurz wrote:
> A deadlock condition potentially exists if a vhost-user process needs
> to request something to QEMU on the slave channel while processing a
> vhost-user message.
> 
> This doesn't seem to affect any vhost-user implementation so far, but
> this is currently biting the upcoming enablement of DAX with virtio-fs.
> The issue is being observed when the guest does an emergency reboot while
> a mapping still exits in the DAX window, which is very easy to get with
> a busy enough workload (e.g. as simulated by blogbench [1]) :
> 
> - QEMU sends VHOST_USER_GET_VRING_BASE to virtiofsd.
> 
> - In order to complete the request, virtiofsd then asks QEMU to remove
>   the mapping on the slave channel.
> 
> All these dialogs are synchronous, hence the deadlock.
> 
> As pointed out by Stefan Hajnoczi:
> 
> When QEMU's vhost-user master implementation sends a vhost-user protocol
> message, vhost_user_read() does a "blocking" read during which slave_fd
> is not monitored by QEMU.
> 
> The natural solution for this issue is an event loop. The main event
> loop cannot be nested though since we have no guarantees that its
> fd handlers are prepared for re-entrancy.
> 
> Introduce a new event loop that only monitors the chardev I/O for now
> in vhost_user_read() and push the actual reading to a one-shot handler.
> A subsequent patch will teach the loop to monitor and process messages
> from the slave channel as well.
> 
> [1] https://github.com/jedisct1/Blogbench
> 
> Suggested-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Greg Kurz <groug@kaod.org>
> ---
> v2: - Document why a nested loop is needed in vhost_user_read() (Stefan)
> ---
>  hw/virtio/vhost-user.c | 65 ++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 60 insertions(+), 5 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  reply	other threads:[~2021-03-15 10:39 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-12  9:22 [PATCH v2 0/7] virtiofsd: Avoid potential deadlocks Greg Kurz
2021-03-12  9:22 ` [Virtio-fs] " Greg Kurz
2021-03-12  9:22 ` [PATCH v2 1/7] vhost-user: Drop misleading EAGAIN checks in slave_read() Greg Kurz
2021-03-12  9:22   ` [Virtio-fs] " Greg Kurz
2021-03-15 10:34   ` Stefan Hajnoczi
2021-03-15 10:34     ` [Virtio-fs] " Stefan Hajnoczi
2021-03-12  9:22 ` [PATCH v2 2/7] vhost-user: Fix double-close on slave_read() error path Greg Kurz
2021-03-12  9:22   ` [Virtio-fs] " Greg Kurz
2021-03-15 10:36   ` Stefan Hajnoczi
2021-03-15 10:36     ` [Virtio-fs] " Stefan Hajnoczi
2021-03-12  9:22 ` [PATCH v2 3/7] vhost-user: Factor out duplicated slave_fd teardown code Greg Kurz
2021-03-12  9:22   ` [Virtio-fs] " Greg Kurz
2021-03-15 10:36   ` Stefan Hajnoczi
2021-03-15 10:36     ` [Virtio-fs] " Stefan Hajnoczi
2021-03-12  9:22 ` [PATCH v2 4/7] vhost-user: Convert slave channel to QIOChannelSocket Greg Kurz
2021-03-12  9:22   ` [Virtio-fs] " Greg Kurz
2021-03-15 10:37   ` Stefan Hajnoczi
2021-03-15 10:37     ` [Virtio-fs] " Stefan Hajnoczi
2021-03-12  9:22 ` [PATCH v2 5/7] vhost-user: Introduce nested event loop in vhost_user_read() Greg Kurz
2021-03-12  9:22   ` [Virtio-fs] " Greg Kurz
2021-03-15 10:38   ` Stefan Hajnoczi [this message]
2021-03-15 10:38     ` Stefan Hajnoczi
2021-03-12  9:22 ` [PATCH v2 6/7] vhost-user: Monitor slave channel " Greg Kurz
2021-03-12  9:22   ` [Virtio-fs] " Greg Kurz
2021-03-15 12:20   ` Stefan Hajnoczi
2021-03-15 12:20     ` [Virtio-fs] " Stefan Hajnoczi
2021-03-12  9:22 ` [PATCH v2 7/7] virtiofsd: Release vu_dispatch_lock when stopping queue Greg Kurz
2021-03-12  9:22   ` [Virtio-fs] " Greg Kurz
2021-03-15 14:57   ` Dr. David Alan Gilbert
2021-03-15 14:57     ` [Virtio-fs] " Dr. David Alan Gilbert

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YE85MJl5VNnsN6Y2@stefanha-x1.localdomain \
    --to=stefanha@redhat.com \
    --cc=berrange@redhat.com \
    --cc=dgilbert@redhat.com \
    --cc=groug@kaod.org \
    --cc=mst@redhat.com \
    --cc=qemu-devel@nongnu.org \
    --cc=vgoyal@redhat.com \
    --cc=virtio-fs@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.