netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Stefan Hajnoczi <stefanha@redhat.com>
To: Yongji Xie <xieyongji@bytedance.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>,
	"Jason Wang" <jasowang@redhat.com>,
	"Stefano Garzarella" <sgarzare@redhat.com>,
	"Parav Pandit" <parav@nvidia.com>,
	"Christoph Hellwig" <hch@infradead.org>,
	"Christian Brauner" <christian.brauner@canonical.com>,
	"Randy Dunlap" <rdunlap@infradead.org>,
	"Matthew Wilcox" <willy@infradead.org>,
	viro@zeniv.linux.org.uk, "Jens Axboe" <axboe@kernel.dk>,
	bcrl@kvack.org, "Jonathan Corbet" <corbet@lwn.net>,
	"Mika Penttilä" <mika.penttila@nextfour.com>,
	"Dan Carpenter" <dan.carpenter@oracle.com>,
	virtualization@lists.linux-foundation.org,
	netdev@vger.kernel.org, kvm@vger.kernel.org,
	linux-fsdevel@vger.kernel.org
Subject: Re: Re: [PATCH v6 10/10] Documentation: Add documentation for VDUSE
Date: Thu, 15 Apr 2021 08:19:14 +0100	[thread overview]
Message-ID: <YHfo8pc7dIO9lNc3@stefanha-x1.localdomain> (raw)
In-Reply-To: <CACycT3uNR+nZY5gY0UhPkeOyi7Za6XkX4b=hasuDcgqdc7fqfg@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 7947 bytes --]

On Thu, Apr 15, 2021 at 01:38:37PM +0800, Yongji Xie wrote:
> On Wed, Apr 14, 2021 at 10:15 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > On Wed, Mar 31, 2021 at 04:05:19PM +0800, Xie Yongji wrote:
> > > VDUSE (vDPA Device in Userspace) is a framework to support
> > > implementing software-emulated vDPA devices in userspace. This
> > > document is intended to clarify the VDUSE design and usage.
> > >
> > > Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
> > > ---
> > >  Documentation/userspace-api/index.rst |   1 +
> > >  Documentation/userspace-api/vduse.rst | 212 ++++++++++++++++++++++++++++++++++
> > >  2 files changed, 213 insertions(+)
> > >  create mode 100644 Documentation/userspace-api/vduse.rst
> >
> > Just looking over the documentation briefly (I haven't studied the code
> > yet)...
> >
> 
> Thank you!
> 
> > > +How VDUSE works
> > > +------------
> > > +Each userspace vDPA device is created by the VDUSE_CREATE_DEV ioctl on
> > > +the character device (/dev/vduse/control). Then a device file with the
> > > +specified name (/dev/vduse/$NAME) will appear, which can be used to
> > > +implement the userspace vDPA device's control path and data path.
> >
> > These steps are taken after sending the VDPA_CMD_DEV_NEW netlink
> > message? (Please consider reordering the documentation to make it clear
> > what the sequence of steps are.)
> >
> 
> No, VDUSE devices should be created before sending the
> VDPA_CMD_DEV_NEW netlink messages which might produce I/Os to VDUSE.

I see. Please include an overview of the steps before going into detail.
Something like:

  VDUSE devices are started as follows:

  1. Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on
     /dev/vduse/control.

  2. Begin processing VDUSE messages from /dev/vduse/$NAME. The first
     messages will arrive while attaching the VDUSE instance to vDPA.

  3. Send the VDPA_CMD_DEV_NEW netlink message to attach the VDUSE
     instance to vDPA.

  VDUSE devices are stopped as follows:

  ...

> > > +     static int netlink_add_vduse(const char *name, int device_id)
> > > +     {
> > > +             struct nl_sock *nlsock;
> > > +             struct nl_msg *msg;
> > > +             int famid;
> > > +
> > > +             nlsock = nl_socket_alloc();
> > > +             if (!nlsock)
> > > +                     return -ENOMEM;
> > > +
> > > +             if (genl_connect(nlsock))
> > > +                     goto free_sock;
> > > +
> > > +             famid = genl_ctrl_resolve(nlsock, VDPA_GENL_NAME);
> > > +             if (famid < 0)
> > > +                     goto close_sock;
> > > +
> > > +             msg = nlmsg_alloc();
> > > +             if (!msg)
> > > +                     goto close_sock;
> > > +
> > > +             if (!genlmsg_put(msg, NL_AUTO_PORT, NL_AUTO_SEQ, famid, 0, 0,
> > > +                 VDPA_CMD_DEV_NEW, 0))
> > > +                     goto nla_put_failure;
> > > +
> > > +             NLA_PUT_STRING(msg, VDPA_ATTR_DEV_NAME, name);
> > > +             NLA_PUT_STRING(msg, VDPA_ATTR_MGMTDEV_DEV_NAME, "vduse");
> > > +             NLA_PUT_U32(msg, VDPA_ATTR_DEV_ID, device_id);
> >
> > What are the permission/capability requirements for VDUSE?
> >
> 
> Now I think we need privileged permission (root user). Because
> userspace daemon is able to access avail vring, used vring, descriptor
> table in kernel driver directly.

Please state this explicitly at the start of the document. Existing
interfaces like FUSE are designed to avoid trusting userspace. Therefore
people might think the same is the case here. It's critical that people
are aware of this before deploying VDUSE with virtio-vdpa.

We should probably pause here and think about whether it's possible to
avoid trusting userspace. Even if it takes some effort and costs some
performance it would probably be worthwhile.

Is the security situation different with vhost-vdpa? In that case it
seems more likely that the host kernel doesn't need to trust the
userspace VDUSE device.

Regarding privileges in general: userspace VDUSE processes shouldn't
need to run as root. The VDUSE device lifecycle will require privileges
to attach vhost-vdpa and virtio-vdpa devices, but the actual userspace
process that emulates the device should be able to run unprivileged.
Emulated devices are an attack surface and even if you are comfortable
with running them as root in your specific use case, it will be an issue
as soon as other people want to use VDUSE and could give VDUSE a
reputation for poor security.

> > How does VDUSE interact with namespaces?
> >
> 
> Not sure I get your point here. Do you mean how the emulated vDPA
> device interact with namespaces? This should work like hardware vDPA
> devices do. VDUSE daemon can reside outside the namespace of a
> container which uses the vDPA device.

Can VDUSE devices run inside containers? Are /dev/vduse/$NAME and vDPA
device names global?

> > What is the meaning of VDPA_ATTR_DEV_ID? I don't see it in Linux
> > v5.12-rc6 drivers/vdpa/vdpa.c:vdpa_nl_cmd_dev_add_set_doit().
> >
> 
> It means the device id (e.g. VIRTIO_ID_BLOCK) of the vDPA device and
> can be found in include/uapi/linux/vdpa.h.

VDPA_ATTR_DEV_ID is only used by VDPA_CMD_DEV_GET in Linux v5.12-rc6,
not by VDPA_CMD_DEV_NEW.

The example in this document uses VDPA_ATTR_DEV_ID with
VDPA_CMD_DEV_NEW. Is the example outdated?

> 
> > > +MMU-based IOMMU Driver
> > > +----------------------
> > > +VDUSE framework implements an MMU-based on-chip IOMMU driver to support
> > > +mapping the kernel DMA buffer into the userspace iova region dynamically.
> > > +This is mainly designed for virtio-vdpa case (kernel virtio drivers).
> > > +
> > > +The basic idea behind this driver is treating MMU (VA->PA) as IOMMU (IOVA->PA).
> > > +The driver will set up MMU mapping instead of IOMMU mapping for the DMA transfer
> > > +so that the userspace process is able to use its virtual address to access
> > > +the DMA buffer in kernel.
> > > +
> > > +And to avoid security issue, a bounce-buffering mechanism is introduced to
> > > +prevent userspace accessing the original buffer directly which may contain other
> > > +kernel data. During the mapping, unmapping, the driver will copy the data from
> > > +the original buffer to the bounce buffer and back, depending on the direction of
> > > +the transfer. And the bounce-buffer addresses will be mapped into the user address
> > > +space instead of the original one.
> >
> > Is mmap(2) the right interface if memory is not actually shared, why not
> > just use pread(2)/pwrite(2) to make the copy explicit? That way the copy
> > semantics are clear. For example, don't expect to be able to busy wait
> > on the memory because changes will not be visible to the other side.
> >
> > (I guess I'm missing something here and that mmap(2) is the right
> > approach, but maybe this documentation section can be clarified.)
> 
> It's for performance considerations on the one hand. We might need to
> call pread(2)/pwrite(2) multiple times for each request.

Userspace can keep page-sized pread() buffers around to avoid additional
syscalls during a request.

mmap() access does reduce the number of syscalls, but it also introduces
page faults (effectively doing the page-sized pread() I mentioned
above).

It's not obvious to me that there is a fundamental difference between
the two approaches in terms of performance.

> On the other
> hand, we can handle the virtqueue in a unified way for both vhost-vdpa
> case and virtio-vdpa case. Otherwise, userspace daemon needs to know
> which iova ranges need to be accessed with pread(2)/pwrite(2). And in
> the future, we might be able to avoid bouncing in some cases.

Ah, I see. So bounce buffers are not used for vhost-vdpa?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  reply	other threads:[~2021-04-15  7:19 UTC|newest]

Thread overview: 62+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-31  8:05 [PATCH v6 00/10] Introduce VDUSE - vDPA Device in Userspace Xie Yongji
2021-03-31  8:05 ` [PATCH v6 01/10] file: Export receive_fd() to modules Xie Yongji
2021-03-31  9:15   ` Christian Brauner
2021-03-31  9:26     ` Dan Carpenter
2021-03-31  9:28       ` Christian Brauner
2021-03-31 11:32     ` Yongji Xie
2021-03-31 12:23       ` Christian Brauner
2021-03-31 13:59         ` Yongji Xie
2021-03-31 14:07           ` Christian Brauner
2021-03-31 14:37             ` Yongji Xie
2021-03-31  8:05 ` [PATCH v6 02/10] eventfd: Increase the recursion depth of eventfd_signal() Xie Yongji
2021-03-31  8:05 ` [PATCH v6 03/10] vhost-vdpa: protect concurrent access to vhost device iotlb Xie Yongji
2021-04-09 16:15   ` Michael S. Tsirkin
2021-04-11  5:36     ` Yongji Xie
2021-04-11 20:48       ` Michael S. Tsirkin
2021-04-12  2:29         ` Yongji Xie
2021-04-12  9:00           ` Michael S. Tsirkin
2021-03-31  8:05 ` [PATCH v6 04/10] vhost-iotlb: Add an opaque pointer for vhost IOTLB Xie Yongji
2021-03-31  8:05 ` [PATCH v6 05/10] vdpa: Add an opaque pointer for vdpa_config_ops.dma_map() Xie Yongji
2021-03-31  8:05 ` [PATCH v6 06/10] vdpa: factor out vhost_vdpa_pa_map() and vhost_vdpa_pa_unmap() Xie Yongji
2021-03-31  8:05 ` [PATCH v6 07/10] vdpa: Support transferring virtual addressing during DMA mapping Xie Yongji
2021-04-08  2:36   ` Jason Wang
2021-03-31  8:05 ` [PATCH v6 08/10] vduse: Implement an MMU-based IOMMU driver Xie Yongji
2021-04-08  3:25   ` Jason Wang
2021-04-08  5:27     ` Yongji Xie
2021-03-31  8:05 ` [PATCH v6 09/10] vduse: Introduce VDUSE - vDPA Device in Userspace Xie Yongji
2021-04-08  6:57   ` Jason Wang
2021-04-08  9:36     ` Yongji Xie
2021-04-09  5:36       ` Jason Wang
2021-04-09  8:02         ` Yongji Xie
2021-04-12  7:16           ` Jason Wang
2021-04-12  8:02             ` Yongji Xie
2021-04-12  9:37               ` Jason Wang
2021-04-12  9:59                 ` Yongji Xie
2021-04-13  3:35                   ` Jason Wang
2021-04-13  4:28                     ` Yongji Xie
2021-04-14  8:18                       ` Jason Wang
2021-04-16  3:24   ` Jason Wang
2021-04-16  8:43     ` Yongji Xie
2021-03-31  8:05 ` [PATCH v6 10/10] Documentation: Add documentation for VDUSE Xie Yongji
2021-04-08  7:18   ` Jason Wang
2021-04-08  8:09     ` Yongji Xie
2021-04-14 14:14   ` Stefan Hajnoczi
2021-04-15  5:38     ` Yongji Xie
2021-04-15  7:19       ` Stefan Hajnoczi [this message]
2021-04-15  8:33         ` Yongji Xie
2021-04-15 14:17           ` Stefan Hajnoczi
2021-04-15  8:36         ` Jason Wang
2021-04-15  9:04           ` Jason Wang
2021-04-15 11:17             ` Yongji Xie
2021-04-16  2:20               ` Jason Wang
2021-04-16  2:58                 ` Yongji Xie
2021-04-16  3:02                   ` Jason Wang
2021-04-16  3:18                     ` Yongji Xie
2021-04-15 14:38           ` Stefan Hajnoczi
2021-04-16  2:23             ` Jason Wang
2021-04-16  3:19               ` Yongji Xie
2021-04-16  5:39                 ` Jason Wang
2021-04-16  3:13             ` Yongji Xie
2021-04-14  7:34 ` [PATCH v6 00/10] Introduce VDUSE - vDPA Device in Userspace Michael S. Tsirkin
2021-04-14  7:49   ` Jason Wang
2021-04-14  7:54   ` Yongji Xie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YHfo8pc7dIO9lNc3@stefanha-x1.localdomain \
    --to=stefanha@redhat.com \
    --cc=axboe@kernel.dk \
    --cc=bcrl@kvack.org \
    --cc=christian.brauner@canonical.com \
    --cc=corbet@lwn.net \
    --cc=dan.carpenter@oracle.com \
    --cc=hch@infradead.org \
    --cc=jasowang@redhat.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=mika.penttila@nextfour.com \
    --cc=mst@redhat.com \
    --cc=netdev@vger.kernel.org \
    --cc=parav@nvidia.com \
    --cc=rdunlap@infradead.org \
    --cc=sgarzare@redhat.com \
    --cc=viro@zeniv.linux.org.uk \
    --cc=virtualization@lists.linux-foundation.org \
    --cc=willy@infradead.org \
    --cc=xieyongji@bytedance.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).