From: Lee Jones <lee.jones@linaro.org>
To: Greg KH <gregkh@linuxfoundation.org>
Cc: mst@redhat.com, jasowang@redhat.com,
linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
virtualization@lists.linux-foundation.org,
netdev@vger.kernel.org, stable@vger.kernel.org,
syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com
Subject: Re: [PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use
Date: Tue, 8 Mar 2022 09:15:27 +0000 [thread overview]
Message-ID: <Yicer3yGg5rrdSIs@google.com> (raw)
In-Reply-To: <Yicalf1I6oBytbse@kroah.com>
On Tue, 08 Mar 2022, Greg KH wrote:
> On Tue, Mar 08, 2022 at 08:10:06AM +0000, Lee Jones wrote:
> > On Mon, 07 Mar 2022, Greg KH wrote:
> >
> > > On Mon, Mar 07, 2022 at 07:17:57PM +0000, Lee Jones wrote:
> > > > vhost_vsock_handle_tx_kick() already holds the mutex during its call
> > > > to vhost_get_vq_desc(). All we have to do here is take the same lock
> > > > during virtqueue clean-up and we mitigate the reported issues.
> > > >
> > > > Also WARN() as a precautionary measure. The purpose of this is to
> > > > capture possible future race conditions which may pop up over time.
> > > >
> > > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00
> > > >
> > > > Cc: <stable@vger.kernel.org>
> > > > Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com
> > > > Signed-off-by: Lee Jones <lee.jones@linaro.org>
> > > > ---
> > > > drivers/vhost/vhost.c | 10 ++++++++++
> > > > 1 file changed, 10 insertions(+)
> > > >
> > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > > > index 59edb5a1ffe28..ef7e371e3e649 100644
> > > > --- a/drivers/vhost/vhost.c
> > > > +++ b/drivers/vhost/vhost.c
> > > > @@ -693,6 +693,15 @@ void vhost_dev_cleanup(struct vhost_dev *dev)
> > > > int i;
> > > >
> > > > for (i = 0; i < dev->nvqs; ++i) {
> > > > + /* No workers should run here by design. However, races have
> > > > + * previously occurred where drivers have been unable to flush
> > > > + * all work properly prior to clean-up. Without a successful
> > > > + * flush the guest will malfunction, but avoiding host memory
> > > > + * corruption in those cases does seem preferable.
> > > > + */
> > > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex));
> > >
> > > So you are trading one syzbot triggered issue for another one in the
> > > future? :)
> > >
> > > If this ever can happen, handle it, but don't log it with a WARN_ON() as
> > > that will trigger the panic-on-warn boxes, as well as syzbot. Unless
> > > you want that to happen?
> >
> > No, Syzbot doesn't report warnings, only BUGs and memory corruption.
>
> Has it changed? Last I looked, it did trigger on WARN_* calls, which
> has resulted in a huge number of kernel fixes because of that.
Everything is customisable in syzkaller, so maybe there are specific
builds which panic_on_warn enabled, but none that I'm involved with
do.
Here follows a topical example. The report above in the Link: tag
comes with a crashlog [0]. In there you can see the WARN() at the
bottom of vhost_dev_cleanup() trigger many times due to a populated
(non-flushed) worker list, before finally tripping the BUG() which
triggers the report:
[0] https://syzkaller.appspot.com/text?tag=CrashLog&x=16a61fce700000
> > > And what happens if the mutex is locked _RIGHT_ after you checked it?
> > > You still have a race...
> >
> > No, we miss a warning that one time. Memory is still protected.
>
> Then don't warn on something that doesn't matter. This line can be
> dropped as there's nothing anyone can do about it, right?
You'll have to take that point up with Michael.
--
Lee Jones [李琼斯]
Principal Technical Lead - Developer Services
Linaro.org │ Open source software for Arm SoCs
Follow Linaro: Facebook | Twitter | Blog
next prev parent reply other threads:[~2022-03-08 9:15 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-07 19:17 [PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use Lee Jones
2022-03-07 19:33 ` Greg KH
2022-03-07 22:39 ` Michael S. Tsirkin
2022-03-08 8:10 ` Lee Jones
2022-03-08 8:11 ` Lee Jones
2022-03-08 8:57 ` Greg KH
2022-03-08 9:15 ` Lee Jones [this message]
2022-03-08 9:57 ` Greg KH
2022-03-08 10:08 ` Lee Jones
2022-03-08 10:55 ` Michael S. Tsirkin
2022-03-08 11:45 ` Greg KH
2022-03-08 12:27 ` Michael S. Tsirkin
2022-03-08 13:17 ` Lee Jones
2022-03-08 17:17 ` Michael S. Tsirkin
2022-03-08 11:05 ` Michael S. Tsirkin
2022-03-09 18:52 ` Leon Romanovsky
2022-03-07 22:37 ` Michael S. Tsirkin
2022-03-08 8:01 ` Lee Jones
2022-03-08 11:07 ` Michael S. Tsirkin
2022-03-08 6:15 ` Jason Wang
2022-03-08 8:08 ` Lee Jones
2022-03-08 11:06 ` Michael S. Tsirkin
-- strict thread matches above, loose matches on Subject: below --
2022-03-14 8:43 Lee Jones
2022-03-14 8:56 ` Greg KH
2022-03-14 11:49 ` Michael S. Tsirkin
2022-03-14 12:47 ` Lee Jones
2022-03-02 7:54 Lee Jones
2022-03-02 9:34 ` Stefano Garzarella
2022-03-02 10:07 ` Lee Jones
2022-03-02 13:35 ` Michael S. Tsirkin
2022-03-02 14:11 ` Stefano Garzarella
2022-03-02 14:50 ` Michael S. Tsirkin
2022-03-02 15:36 ` Stefano Garzarella
2022-03-04 16:46 ` Michael S. Tsirkin
2022-03-02 13:30 ` Michael S. Tsirkin
2022-03-02 13:56 ` Lee Jones
2022-03-02 14:51 ` Michael S. Tsirkin
2022-03-02 14:57 ` Lee Jones
2022-03-02 16:28 ` Stefano Garzarella
2022-03-02 16:30 ` Michael S. Tsirkin
2022-03-02 16:49 ` Lee Jones
2022-03-02 17:10 ` Stefano Garzarella
2022-03-03 14:17 ` Lee Jones
2022-03-04 5:00 ` Michael S. Tsirkin
2022-03-04 15:22 ` Lee Jones
2022-03-04 16:48 ` Michael S. Tsirkin
2022-03-04 16:56 ` Lee Jones
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Yicer3yGg5rrdSIs@google.com \
--to=lee.jones@linaro.org \
--cc=gregkh@linuxfoundation.org \
--cc=jasowang@redhat.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mst@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=stable@vger.kernel.org \
--cc=syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).