From: "Michael S. Tsirkin" <mst@redhat.com> To: Greg KH <gregkh@linuxfoundation.org> Cc: Lee Jones <lee.jones@linaro.org>, jasowang@redhat.com, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, stable@vger.kernel.org, syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Subject: Re: [PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use Date: Tue, 8 Mar 2022 07:27:47 -0500 [thread overview] Message-ID: <20220308071718-mutt-send-email-mst@kernel.org> (raw) In-Reply-To: <YidBz7SxED2ii1Lh@kroah.com> On Tue, Mar 08, 2022 at 12:45:19PM +0100, Greg KH wrote: > On Tue, Mar 08, 2022 at 05:55:58AM -0500, Michael S. Tsirkin wrote: > > On Tue, Mar 08, 2022 at 10:57:42AM +0100, Greg KH wrote: > > > On Tue, Mar 08, 2022 at 09:15:27AM +0000, Lee Jones wrote: > > > > On Tue, 08 Mar 2022, Greg KH wrote: > > > > > > > > > On Tue, Mar 08, 2022 at 08:10:06AM +0000, Lee Jones wrote: > > > > > > On Mon, 07 Mar 2022, Greg KH wrote: > > > > > > > > > > > > > On Mon, Mar 07, 2022 at 07:17:57PM +0000, Lee Jones wrote: > > > > > > > > vhost_vsock_handle_tx_kick() already holds the mutex during its call > > > > > > > > to vhost_get_vq_desc(). All we have to do here is take the same lock > > > > > > > > during virtqueue clean-up and we mitigate the reported issues. > > > > > > > > > > > > > > > > Also WARN() as a precautionary measure. The purpose of this is to > > > > > > > > capture possible future race conditions which may pop up over time. > > > > > > > > > > > > > > > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00 > > > > > > > > > > > > > > > > Cc: <stable@vger.kernel.org> > > > > > > > > Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com > > > > > > > > Signed-off-by: Lee Jones <lee.jones@linaro.org> > > > > > > > > --- > > > > > > > > drivers/vhost/vhost.c | 10 ++++++++++ > > > > > > > > 1 file changed, 10 insertions(+) > > > > > > > > > > > > > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > > > > > > > index 59edb5a1ffe28..ef7e371e3e649 100644 > > > > > > > > --- a/drivers/vhost/vhost.c > > > > > > > > +++ b/drivers/vhost/vhost.c > > > > > > > > @@ -693,6 +693,15 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > > > > > > > int i; > > > > > > > > > > > > > > > > for (i = 0; i < dev->nvqs; ++i) { > > > > > > > > + /* No workers should run here by design. However, races have > > > > > > > > + * previously occurred where drivers have been unable to flush > > > > > > > > + * all work properly prior to clean-up. Without a successful > > > > > > > > + * flush the guest will malfunction, but avoiding host memory > > > > > > > > + * corruption in those cases does seem preferable. > > > > > > > > + */ > > > > > > > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > > > > > > > > > > > > > > So you are trading one syzbot triggered issue for another one in the > > > > > > > future? :) > > > > > > > > > > > > > > If this ever can happen, handle it, but don't log it with a WARN_ON() as > > > > > > > that will trigger the panic-on-warn boxes, as well as syzbot. Unless > > > > > > > you want that to happen? > > > > > > > > > > > > No, Syzbot doesn't report warnings, only BUGs and memory corruption. > > > > > > > > > > Has it changed? Last I looked, it did trigger on WARN_* calls, which > > > > > has resulted in a huge number of kernel fixes because of that. > > > > > > > > Everything is customisable in syzkaller, so maybe there are specific > > > > builds which panic_on_warn enabled, but none that I'm involved with > > > > do. > > > > > > Many systems run with panic-on-warn (i.e. the cloud), as they want to > > > drop a box and restart it if anything goes wrong. > > > > > > That's why syzbot reports on WARN_* calls. They should never be > > > reachable by userspace actions. > > > > > > > Here follows a topical example. The report above in the Link: tag > > > > comes with a crashlog [0]. In there you can see the WARN() at the > > > > bottom of vhost_dev_cleanup() trigger many times due to a populated > > > > (non-flushed) worker list, before finally tripping the BUG() which > > > > triggers the report: > > > > > > > > [0] https://syzkaller.appspot.com/text?tag=CrashLog&x=16a61fce700000 > > > > > > Ok, so both happens here. But don't add a warning for something that > > > can't happen. Just handle it and move on. It looks like you are > > > handling it in this code, so please drop the WARN_ON(). > > > > > > thanks, > > > > > > greg k-h > > > > Hmm. Well this will mean if we ever reintroduce the bug then > > syzkaller will not catch it for us :( And the bug is there, > > it just results in a hard to reproduce error for userspace. > > Is this an error you can recover from in the kernel? > What is userspace > supposed to know with this information when it sees it? IIUC we are talking about a use after free here since we somehow managed to have a pointer to the device in a worker while device is being destroyed. That's the point of the warning as use after free is hard to debug. You ask can we recover from a use after free? As regards to the added lock, IIUC it kind of shifts the use after free window to later and since we zero out some of the memory just before we free it, it's a bit more likely to recover. I would still like to see some more analysis on why the situation is always better than it was before though. > > Not sure what to do here. Export panic_on_warn flag to modules > > and check it here? > > Hah, no, never do that :) > > thanks, > > greg k-h
WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com> To: Greg KH <gregkh@linuxfoundation.org> Cc: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com, kvm@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, virtualization@lists.linux-foundation.org, Lee Jones <lee.jones@linaro.org> Subject: Re: [PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use Date: Tue, 8 Mar 2022 07:27:47 -0500 [thread overview] Message-ID: <20220308071718-mutt-send-email-mst@kernel.org> (raw) In-Reply-To: <YidBz7SxED2ii1Lh@kroah.com> On Tue, Mar 08, 2022 at 12:45:19PM +0100, Greg KH wrote: > On Tue, Mar 08, 2022 at 05:55:58AM -0500, Michael S. Tsirkin wrote: > > On Tue, Mar 08, 2022 at 10:57:42AM +0100, Greg KH wrote: > > > On Tue, Mar 08, 2022 at 09:15:27AM +0000, Lee Jones wrote: > > > > On Tue, 08 Mar 2022, Greg KH wrote: > > > > > > > > > On Tue, Mar 08, 2022 at 08:10:06AM +0000, Lee Jones wrote: > > > > > > On Mon, 07 Mar 2022, Greg KH wrote: > > > > > > > > > > > > > On Mon, Mar 07, 2022 at 07:17:57PM +0000, Lee Jones wrote: > > > > > > > > vhost_vsock_handle_tx_kick() already holds the mutex during its call > > > > > > > > to vhost_get_vq_desc(). All we have to do here is take the same lock > > > > > > > > during virtqueue clean-up and we mitigate the reported issues. > > > > > > > > > > > > > > > > Also WARN() as a precautionary measure. The purpose of this is to > > > > > > > > capture possible future race conditions which may pop up over time. > > > > > > > > > > > > > > > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00 > > > > > > > > > > > > > > > > Cc: <stable@vger.kernel.org> > > > > > > > > Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com > > > > > > > > Signed-off-by: Lee Jones <lee.jones@linaro.org> > > > > > > > > --- > > > > > > > > drivers/vhost/vhost.c | 10 ++++++++++ > > > > > > > > 1 file changed, 10 insertions(+) > > > > > > > > > > > > > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > > > > > > > index 59edb5a1ffe28..ef7e371e3e649 100644 > > > > > > > > --- a/drivers/vhost/vhost.c > > > > > > > > +++ b/drivers/vhost/vhost.c > > > > > > > > @@ -693,6 +693,15 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > > > > > > > int i; > > > > > > > > > > > > > > > > for (i = 0; i < dev->nvqs; ++i) { > > > > > > > > + /* No workers should run here by design. However, races have > > > > > > > > + * previously occurred where drivers have been unable to flush > > > > > > > > + * all work properly prior to clean-up. Without a successful > > > > > > > > + * flush the guest will malfunction, but avoiding host memory > > > > > > > > + * corruption in those cases does seem preferable. > > > > > > > > + */ > > > > > > > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > > > > > > > > > > > > > > So you are trading one syzbot triggered issue for another one in the > > > > > > > future? :) > > > > > > > > > > > > > > If this ever can happen, handle it, but don't log it with a WARN_ON() as > > > > > > > that will trigger the panic-on-warn boxes, as well as syzbot. Unless > > > > > > > you want that to happen? > > > > > > > > > > > > No, Syzbot doesn't report warnings, only BUGs and memory corruption. > > > > > > > > > > Has it changed? Last I looked, it did trigger on WARN_* calls, which > > > > > has resulted in a huge number of kernel fixes because of that. > > > > > > > > Everything is customisable in syzkaller, so maybe there are specific > > > > builds which panic_on_warn enabled, but none that I'm involved with > > > > do. > > > > > > Many systems run with panic-on-warn (i.e. the cloud), as they want to > > > drop a box and restart it if anything goes wrong. > > > > > > That's why syzbot reports on WARN_* calls. They should never be > > > reachable by userspace actions. > > > > > > > Here follows a topical example. The report above in the Link: tag > > > > comes with a crashlog [0]. In there you can see the WARN() at the > > > > bottom of vhost_dev_cleanup() trigger many times due to a populated > > > > (non-flushed) worker list, before finally tripping the BUG() which > > > > triggers the report: > > > > > > > > [0] https://syzkaller.appspot.com/text?tag=CrashLog&x=16a61fce700000 > > > > > > Ok, so both happens here. But don't add a warning for something that > > > can't happen. Just handle it and move on. It looks like you are > > > handling it in this code, so please drop the WARN_ON(). > > > > > > thanks, > > > > > > greg k-h > > > > Hmm. Well this will mean if we ever reintroduce the bug then > > syzkaller will not catch it for us :( And the bug is there, > > it just results in a hard to reproduce error for userspace. > > Is this an error you can recover from in the kernel? > What is userspace > supposed to know with this information when it sees it? IIUC we are talking about a use after free here since we somehow managed to have a pointer to the device in a worker while device is being destroyed. That's the point of the warning as use after free is hard to debug. You ask can we recover from a use after free? As regards to the added lock, IIUC it kind of shifts the use after free window to later and since we zero out some of the memory just before we free it, it's a bit more likely to recover. I would still like to see some more analysis on why the situation is always better than it was before though. > > Not sure what to do here. Export panic_on_warn flag to modules > > and check it here? > > Hah, no, never do that :) > > thanks, > > greg k-h _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2022-03-08 12:28 UTC|newest] Thread overview: 94+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-03-07 19:17 [PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use Lee Jones 2022-03-07 19:17 ` Lee Jones 2022-03-07 19:33 ` Greg KH 2022-03-07 19:33 ` Greg KH 2022-03-07 22:39 ` Michael S. Tsirkin 2022-03-07 22:39 ` Michael S. Tsirkin 2022-03-08 8:10 ` Lee Jones 2022-03-08 8:10 ` Lee Jones 2022-03-08 8:11 ` Lee Jones 2022-03-08 8:11 ` Lee Jones 2022-03-08 8:57 ` Greg KH 2022-03-08 8:57 ` Greg KH 2022-03-08 9:15 ` Lee Jones 2022-03-08 9:15 ` Lee Jones 2022-03-08 9:57 ` Greg KH 2022-03-08 9:57 ` Greg KH 2022-03-08 10:08 ` Lee Jones 2022-03-08 10:08 ` Lee Jones 2022-03-08 10:55 ` Michael S. Tsirkin 2022-03-08 10:55 ` Michael S. Tsirkin 2022-03-08 11:45 ` Greg KH 2022-03-08 11:45 ` Greg KH 2022-03-08 12:27 ` Michael S. Tsirkin [this message] 2022-03-08 12:27 ` Michael S. Tsirkin 2022-03-08 13:17 ` Lee Jones 2022-03-08 13:17 ` Lee Jones 2022-03-08 17:17 ` Michael S. Tsirkin 2022-03-08 17:17 ` Michael S. Tsirkin 2022-03-08 11:05 ` Michael S. Tsirkin 2022-03-08 11:05 ` Michael S. Tsirkin 2022-03-09 18:52 ` Leon Romanovsky 2022-03-09 18:52 ` Leon Romanovsky 2022-03-07 22:37 ` Michael S. Tsirkin 2022-03-07 22:37 ` Michael S. Tsirkin 2022-03-08 8:01 ` Lee Jones 2022-03-08 8:01 ` Lee Jones 2022-03-08 11:07 ` Michael S. Tsirkin 2022-03-08 11:07 ` Michael S. Tsirkin 2022-03-08 6:15 ` Jason Wang 2022-03-08 6:15 ` Jason Wang 2022-03-08 8:08 ` Lee Jones 2022-03-08 8:08 ` Lee Jones 2022-03-08 11:06 ` Michael S. Tsirkin 2022-03-08 11:06 ` Michael S. Tsirkin -- strict thread matches above, loose matches on Subject: below -- 2022-03-14 8:43 Lee Jones 2022-03-14 8:43 ` Lee Jones 2022-03-14 8:56 ` Greg KH 2022-03-14 8:56 ` Greg KH 2022-03-14 11:49 ` Michael S. Tsirkin 2022-03-14 11:49 ` Michael S. Tsirkin 2022-03-14 12:47 ` Lee Jones 2022-03-14 12:47 ` Lee Jones 2022-03-02 7:54 Lee Jones 2022-03-02 7:54 ` Lee Jones 2022-03-02 9:34 ` Stefano Garzarella 2022-03-02 9:34 ` Stefano Garzarella 2022-03-02 10:07 ` Lee Jones 2022-03-02 10:07 ` Lee Jones 2022-03-02 13:35 ` Michael S. Tsirkin 2022-03-02 13:35 ` Michael S. Tsirkin 2022-03-02 14:11 ` Stefano Garzarella 2022-03-02 14:11 ` Stefano Garzarella 2022-03-02 14:50 ` Michael S. Tsirkin 2022-03-02 14:50 ` Michael S. Tsirkin 2022-03-02 15:36 ` Stefano Garzarella 2022-03-02 15:36 ` Stefano Garzarella 2022-03-04 16:46 ` Michael S. Tsirkin 2022-03-04 16:46 ` Michael S. Tsirkin 2022-03-02 13:30 ` Michael S. Tsirkin 2022-03-02 13:30 ` Michael S. Tsirkin 2022-03-02 13:56 ` Lee Jones 2022-03-02 13:56 ` Lee Jones 2022-03-02 14:51 ` Michael S. Tsirkin 2022-03-02 14:51 ` Michael S. Tsirkin 2022-03-02 14:57 ` Lee Jones 2022-03-02 14:57 ` Lee Jones 2022-03-02 16:28 ` Stefano Garzarella 2022-03-02 16:28 ` Stefano Garzarella 2022-03-02 16:30 ` Michael S. Tsirkin 2022-03-02 16:30 ` Michael S. Tsirkin 2022-03-02 16:49 ` Lee Jones 2022-03-02 16:49 ` Lee Jones 2022-03-02 17:10 ` Stefano Garzarella 2022-03-02 17:10 ` Stefano Garzarella 2022-03-03 14:17 ` Lee Jones 2022-03-03 14:17 ` Lee Jones 2022-03-04 5:00 ` Michael S. Tsirkin 2022-03-04 5:00 ` Michael S. Tsirkin 2022-03-04 15:22 ` Lee Jones 2022-03-04 15:22 ` Lee Jones 2022-03-04 16:48 ` Michael S. Tsirkin 2022-03-04 16:48 ` Michael S. Tsirkin 2022-03-04 16:56 ` Lee Jones 2022-03-04 16:56 ` Lee Jones
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20220308071718-mutt-send-email-mst@kernel.org \ --to=mst@redhat.com \ --cc=gregkh@linuxfoundation.org \ --cc=jasowang@redhat.com \ --cc=kvm@vger.kernel.org \ --cc=lee.jones@linaro.org \ --cc=linux-kernel@vger.kernel.org \ --cc=netdev@vger.kernel.org \ --cc=stable@vger.kernel.org \ --cc=syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.