From: "Michael S. Tsirkin" <mst@redhat.com>
To: Jason Wang <jasowang@redhat.com>
Cc: virtualization <virtualization@lists.linux-foundation.org>,
linux-kernel <linux-kernel@vger.kernel.org>,
"Hetzelt, Felicitas" <f.hetzelt@tu-berlin.de>,
"kaplan, david" <david.kaplan@amd.com>,
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Subject: Re: [PATCH 0/9] More virtio hardening
Date: Tue, 12 Oct 2021 01:44:10 -0400 [thread overview]
Message-ID: <20211012012617-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <CACGkMEtwAFy=bm62X+rjPMJEwChAhZkZ2bBwDJPULdzhWdzagA@mail.gmail.com>
On Tue, Oct 12, 2021 at 10:43:57AM +0800, Jason Wang wrote:
> On Mon, Oct 11, 2021 at 8:36 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, Oct 11, 2021 at 03:36:51PM +0800, Jason Wang wrote:
> > > On Tue, Oct 5, 2021 at 3:42 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, Sep 13, 2021 at 01:53:44PM +0800, Jason Wang wrote:
> > > > > Hi All:
> > > > >
> > > > > This series treis to do more hardening for virito.
> > > > >
> > > > > patch 1 validates the num_queues for virio-blk device.
> > > > > patch 2-4 validates max_nr_ports for virito-console device.
> > > > > patch 5-7 harden virtio-pci interrupts to make sure no exepcted
> > > > > interrupt handler is tiggered. If this makes sense we can do similar
> > > > > things in other transport drivers.
> > > > > patch 8-9 validate used ring length.
> > > > >
> > > > > Smoking test on blk/net with packed=on/off and iommu_platform=on/off.
> > > > >
> > > > > Please review.
> > > > >
> > > > > Thanks
> > > >
> > > > So I poked at console at least, and I think I see
> > > > an issue: if interrupt handler queues a work/bh,
> > > > then it can still run while reset is in progress.
> > >
> > > Looks like a bug which is unrelated to the hardening?
> >
> > Won't preventing use after free be relevant?
>
> Oh right.
>
> > I frankly don't know what does hardening means then.
> > > E.g the driver
> > > should sync with work/bh before reset.
> >
> > No, there's no way to fix it ATM without extra locks and state which I
> > think we should strive to avoid or make it generic, not per-driver,
> > since sync before reset is useless, new interrupts will just arrive and
> > queue more work. And a sync after reset is too late since driver will
> > try to add buffers.
>
> Can we do something like
>
> 1) disable interrupt
> 2) sync bh
>
> Or I guess this is somehow you meant in the following steps.
So that would mean a new API to disable vq interrupts.
reset will re-enable.
E.g. virtqueue_cancel_cb_before_reset()?
Then drivers can sync, then reset.
This means maintaining more state though, which I don't like.
An alternative is something like this:
static void (*virtio_flush_device)(struct virtio_device *dev);
void virtio_reset_device(struct virtio_device *dev, virtio_flush_device flush)
{
might_sleep();
if (flush) {
dev->config->disable_interrupts(dev);
flush(dev);
dev->config->reset(dev);
dev->config->enable_interrupts(dev);
} else {
dev->config->reset(dev);
}
}
I have patches wrapping all reset calls in virtio_reset_device
(without the flush parameter but that's easy to tweak).
> >
> > Maybe we can break device. Two issues with that
> > - drivers might not be ready to handle add_buf failures
> > - restore needs to unbreak then and we don't have a way to do that yet
> >
> > So .. careful reading of all device drivers and hoping we don't mess
> > things up even more ... here we come.
>
> Yes.
The biggest issue with this trick is drivers not handling add_buf
errors, adding a failure path here risks creating memory leaks.
OTOH with e.g. bounce buffers maybe it's possible for add buf to
fail anyway?
> >
> > > >
> > > > I sent a patch to fix it for console removal specifically,
> > > > but I suspect it's not enough e.g. freeze is still broken.
> > > > And note this has been reported without any TDX things -
> > > > it's not a malicious device issue, can be triggered just
> > > > by module unload.
> > > >
> > > > I am vaguely thinking about new APIs to disable/enable callbacks.
> > > > An alternative:
> > > >
> > > > 1. adding new remove_nocb/freeze_nocb calls
> > > > 2. disabling/enabling interrupts automatically around these
> > > > 3. gradually moving devices to using these
> > > > 4. once/if all device move, removing the old callbacks
> > > >
> > > > the advantage here is that we'll be sure calls are always
> > > > paired correctly.
> > >
> > > I'm not sure I get the idea, but my feeling is that it doesn't
> > > conflict with the interrupt hardening here (or at least the same
> > > method is required e.g NO_AUTO_EN).
> > >
> > > Thanks
> >
> > Right. It's not that it conflicts, it's that I was hoping that
> > since you are working on hardening you can take up fixing that.
> > Let me know whether you have the time. Thanks!
>
> I can do that.
>
> Thanks
>
> >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > Jason Wang (9):
> > > > > virtio-blk: validate num_queues during probe
> > > > > virtio: add doc for validate() method
> > > > > virtio-console: switch to use .validate()
> > > > > virtio_console: validate max_nr_ports before trying to use it
> > > > > virtio_config: introduce a new ready method
> > > > > virtio_pci: harden MSI-X interrupts
> > > > > virtio-pci: harden INTX interrupts
> > > > > virtio_ring: fix typos in vring_desc_extra
> > > > > virtio_ring: validate used buffer length
> > > > >
> > > > > drivers/block/virtio_blk.c | 3 +-
> > > > > drivers/char/virtio_console.c | 51 +++++++++++++++++++++---------
> > > > > drivers/virtio/virtio_pci_common.c | 43 +++++++++++++++++++++----
> > > > > drivers/virtio/virtio_pci_common.h | 7 ++--
> > > > > drivers/virtio/virtio_pci_legacy.c | 5 +--
> > > > > drivers/virtio/virtio_pci_modern.c | 6 ++--
> > > > > drivers/virtio/virtio_ring.c | 27 ++++++++++++++--
> > > > > include/linux/virtio.h | 1 +
> > > > > include/linux/virtio_config.h | 6 ++++
> > > > > 9 files changed, 118 insertions(+), 31 deletions(-)
> > > > >
> > > > > --
> > > > > 2.25.1
> > > >
> >
next prev parent reply other threads:[~2021-10-12 5:44 UTC|newest]
Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-13 5:53 [PATCH 0/9] More virtio hardening Jason Wang
2021-09-13 5:53 ` [PATCH 1/9] virtio-blk: validate num_queues during probe Jason Wang
2021-09-13 7:48 ` Stefano Garzarella
2021-09-14 2:29 ` Jason Wang
2021-09-13 12:05 ` Stefan Hajnoczi
2021-09-13 5:53 ` [PATCH 2/9] virtio: add doc for validate() method Jason Wang
2021-09-13 5:53 ` [PATCH 3/9] virtio-console: switch to use .validate() Jason Wang
2021-09-13 5:53 ` [PATCH 4/9] virtio_console: validate max_nr_ports before trying to use it Jason Wang
2021-09-13 5:53 ` [PATCH 5/9] virtio_config: introduce a new ready method Jason Wang
2021-09-13 5:53 ` [PATCH 6/9] virtio_pci: harden MSI-X interrupts Jason Wang
2021-09-13 6:03 ` Michael S. Tsirkin
2021-09-13 6:08 ` Jason Wang
2021-09-13 6:28 ` Michael S. Tsirkin
2021-09-13 6:34 ` Jason Wang
2021-09-13 6:37 ` Michael S. Tsirkin
2021-09-13 6:43 ` Jason Wang
2021-09-13 7:01 ` Michael S. Tsirkin
2021-09-13 7:15 ` Jason Wang
2021-09-13 6:50 ` Michael S. Tsirkin
2021-09-13 7:07 ` Jason Wang
2021-09-13 19:38 ` Thomas Gleixner
2021-09-13 20:54 ` Michael S. Tsirkin
2021-09-13 22:31 ` Thomas Gleixner
2021-09-14 2:20 ` Jason Wang
2021-09-14 8:29 ` Thomas Gleixner
2021-09-13 5:53 ` [PATCH 7/9] virtio-pci: harden INTX interrupts Jason Wang
2021-09-13 6:33 ` Michael S. Tsirkin
2021-09-13 6:36 ` Jason Wang
2021-09-13 6:41 ` Michael S. Tsirkin
2021-09-13 6:45 ` Jason Wang
2021-09-13 7:02 ` Michael S. Tsirkin
2021-09-13 7:17 ` Jason Wang
2021-09-13 21:36 ` Thomas Gleixner
2021-09-13 22:01 ` Michael S. Tsirkin
2021-09-13 22:20 ` Thomas Gleixner
2021-09-14 2:50 ` Jason Wang
2021-09-14 9:34 ` Boqun Feng
2021-09-14 11:03 ` Peter Zijlstra
2021-09-14 11:09 ` Thomas Gleixner
2021-09-13 5:53 ` [PATCH 8/9] virtio_ring: fix typos in vring_desc_extra Jason Wang
2021-09-13 5:53 ` [PATCH 9/9] virtio_ring: validate used buffer length Jason Wang
2021-09-13 6:36 ` Michael S. Tsirkin
2021-09-13 6:40 ` Jason Wang
2021-09-13 6:57 ` Michael S. Tsirkin
2021-09-13 7:13 ` Jason Wang
2021-10-05 7:42 ` [PATCH 0/9] More virtio hardening Michael S. Tsirkin
2021-10-11 7:36 ` Jason Wang
2021-10-11 12:36 ` Michael S. Tsirkin
2021-10-12 2:43 ` Jason Wang
2021-10-12 5:44 ` Michael S. Tsirkin [this message]
2021-10-12 6:11 ` Jason Wang
2021-10-12 6:35 ` Michael S. Tsirkin
2021-10-12 6:43 ` Jason Wang
2021-10-12 7:03 ` Michael S. Tsirkin
2021-10-12 8:46 ` Jason Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211012012617-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=david.kaplan@amd.com \
--cc=f.hetzelt@tu-berlin.de \
--cc=jasowang@redhat.com \
--cc=konrad.wilk@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).