Re: Re: [RFC v3 08/11] vduse: Introduce VDUSE - vDPA Device in Userspace

[Yongji Xie <xieyongji@bytedance.com>]

On Wed, Jan 20, 2021 at 1:54 AM Randy Dunlap <rdunlap@infradead.org> wrote:
>
> Hi,
>
> Documentation comments only:
>

Will fix it.

Thanks,
Yongji


> On 1/18/21 9:07 PM, Xie Yongji wrote:
> >
> > Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
> > ---
> >  Documentation/driver-api/vduse.rst                 |   85 ++
> >
> > diff --git a/Documentation/driver-api/vduse.rst b/Documentation/driver-api/vduse.rst
> > new file mode 100644
> > index 000000000000..9418a7f6646b
> > --- /dev/null
> > +++ b/Documentation/driver-api/vduse.rst
> > @@ -0,0 +1,85 @@
> > +==================================
> > +VDUSE - "vDPA Device in Userspace"
> > +==================================
> > +
> > +vDPA (virtio data path acceleration) device is a device that uses a
> > +datapath which complies with the virtio specifications with vendor
> > +specific control path. vDPA devices can be both physically located on
> > +the hardware or emulated by software. VDUSE is a framework that makes it
> > +possible to implement software-emulated vDPA devices in userspace.
> > +
> > +How VDUSE works
> > +------------
> > +Each userspace vDPA device is created by the VDUSE_CREATE_DEV ioctl on
> > +the VDUSE character device (/dev/vduse). Then a file descriptor pointing
> > +to the new resources will be returned, which can be used to implement the
> > +userspace vDPA device's control path and data path.
> > +
> > +To implement control path, the read/write operations to the file descriptor
> > +will be used to receive/reply the control messages from/to VDUSE driver.
> > +Those control messages are mostly based on the vdpa_config_ops which defines
> > +a unified interface to control different types of vDPA device.
> > +
> > +The following types of messages are provided by the VDUSE framework now:
> > +
> > +- VDUSE_SET_VQ_ADDR: Set the addresses of the different aspects of virtqueue.
> > +
> > +- VDUSE_SET_VQ_NUM: Set the size of virtqueue
> > +
> > +- VDUSE_SET_VQ_READY: Set ready status of virtqueue
> > +
> > +- VDUSE_GET_VQ_READY: Get ready status of virtqueue
> > +
> > +- VDUSE_SET_VQ_STATE: Set the state (last_avail_idx) for virtqueue
> > +
> > +- VDUSE_GET_VQ_STATE: Get the state (last_avail_idx) for virtqueue
> > +
> > +- VDUSE_SET_FEATURES: Set virtio features supported by the driver
> > +
> > +- VDUSE_GET_FEATURES: Get virtio features supported by the device
> > +
> > +- VDUSE_SET_STATUS: Set the device status
> > +
> > +- VDUSE_GET_STATUS: Get the device status
> > +
> > +- VDUSE_SET_CONFIG: Write to device specific configuration space
> > +
> > +- VDUSE_GET_CONFIG: Read from device specific configuration space
> > +
> > +- VDUSE_UPDATE_IOTLB: Notify userspace to update the memory mapping in device IOTLB
> > +
> > +Please see include/linux/vdpa.h for details.
> > +
> > +In the data path, vDPA device's iova regions will be mapped into userspace with
> > +the help of VDUSE_IOTLB_GET_FD ioctl on the userspace vDPA device fd:
> > +
> > +- VDUSE_IOTLB_GET_FD: get the file descriptor to iova region. Userspace can
> > +  access this iova region by passing the fd to mmap(2).
> > +
> > +Besides, the eventfd mechanism is used to trigger interrupt callbacks and
> > +receive virtqueue kicks in userspace. The following ioctls on the userspace
> > +vDPA device fd are provided to support that:
> > +
> > +- VDUSE_VQ_SETUP_KICKFD: set the kickfd for virtqueue, this eventfd is used
> > +  by VDUSE driver to notify userspace to consume the vring.
> > +
> > +- VDUSE_VQ_SETUP_IRQFD: set the irqfd for virtqueue, this eventfd is used
> > +  by userspace to notify VDUSE driver to trigger interrupt callbacks.
> > +
> > +MMU-based IOMMU Driver
> > +----------------------
> > +In virtio-vdpa case, VDUSE framework implements a MMU-based on-chip IOMMU
>
>                                                    an MMU-based
>
> > +driver to support mapping the kernel dma buffer into the userspace iova
>
>                                         DMA
>
> > +region dynamically.
> > +
> > +The basic idea behind this driver is treating MMU (VA->PA) as IOMMU (IOVA->PA).
> > +The driver will set up MMU mapping instead of IOMMU mapping for the DMA transfer
> > +so that the userspace process is able to use its virtual address to access
> > +the dma buffer in kernel.
>
>        DMA
>
> > +
> > +And to avoid security issue, a bounce-buffering mechanism is introduced to
> > +prevent userspace accessing the original buffer directly which may contain other
> > +kernel data. During the mapping, unmapping, the driver will copy the data from
> > +the original buffer to the bounce buffer and back, depending on the direction of
> > +the transfer. And the bounce-buffer addresses will be mapped into the user address
> > +space instead of the original one.
>
>
> thanks.
> --
> ~Randy
>