From: Yongji Xie <xieyongji@bytedance.com>
To: Jason Wang <jasowang@redhat.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>,
"Stefan Hajnoczi" <stefanha@redhat.com>,
"Stefano Garzarella" <sgarzare@redhat.com>,
"Parav Pandit" <parav@nvidia.com>,
"Christoph Hellwig" <hch@infradead.org>,
"Christian Brauner" <christian.brauner@canonical.com>,
"Randy Dunlap" <rdunlap@infradead.org>,
"Matthew Wilcox" <willy@infradead.org>,
"Al Viro" <viro@zeniv.linux.org.uk>,
"Jens Axboe" <axboe@kernel.dk>,
bcrl@kvack.org, "Jonathan Corbet" <corbet@lwn.net>,
"Mika Penttilä" <mika.penttila@nextfour.com>,
"Dan Carpenter" <dan.carpenter@oracle.com>,
joro@8bytes.org, "Greg KH" <gregkh@linuxfoundation.org>,
"He Zhe" <zhe.he@windriver.com>,
"Liu Xiaodong" <xiaodong.liu@intel.com>,
songmuchun@bytedance.com,
virtualization <virtualization@lists.linux-foundation.org>,
netdev@vger.kernel.org, kvm <kvm@vger.kernel.org>,
linux-fsdevel@vger.kernel.org, iommu@lists.linux-foundation.org,
linux-kernel <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v9 16/17] vduse: Introduce VDUSE - vDPA Device in Userspace
Date: Wed, 14 Jul 2021 14:49:21 +0800 [thread overview]
Message-ID: <CACycT3uh+wUeDM1H7JiCJTMeCVCBngURGKeXD-h+meekNNwiQw@mail.gmail.com> (raw)
In-Reply-To: <26116714-f485-eeab-4939-71c4c10c30de@redhat.com>
On Wed, Jul 14, 2021 at 1:45 PM Jason Wang <jasowang@redhat.com> wrote:
>
>
> 在 2021/7/13 下午4:46, Xie Yongji 写道:
> > This VDUSE driver enables implementing software-emulated vDPA
> > devices in userspace. The vDPA device is created by
> > ioctl(VDUSE_CREATE_DEV) on /dev/vduse/control. Then a char device
> > interface (/dev/vduse/$NAME) is exported to userspace for device
> > emulation.
> >
> > In order to make the device emulation more secure, the device's
> > control path is handled in kernel. A message mechnism is introduced
> > to forward some dataplane related control messages to userspace.
> >
> > And in the data path, the DMA buffer will be mapped into userspace
> > address space through different ways depending on the vDPA bus to
> > which the vDPA device is attached. In virtio-vdpa case, the MMU-based
> > IOMMU driver is used to achieve that. And in vhost-vdpa case, the
> > DMA buffer is reside in a userspace memory region which can be shared
> > to the VDUSE userspace processs via transferring the shmfd.
> >
> > For more details on VDUSE design and usage, please see the follow-on
> > Documentation commit.
> >
> > Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
> > ---
> > Documentation/userspace-api/ioctl/ioctl-number.rst | 1 +
> > drivers/vdpa/Kconfig | 10 +
> > drivers/vdpa/Makefile | 1 +
> > drivers/vdpa/vdpa_user/Makefile | 5 +
> > drivers/vdpa/vdpa_user/vduse_dev.c | 1502 ++++++++++++++++++++
> > include/uapi/linux/vduse.h | 221 +++
> > 6 files changed, 1740 insertions(+)
> > create mode 100644 drivers/vdpa/vdpa_user/Makefile
> > create mode 100644 drivers/vdpa/vdpa_user/vduse_dev.c
> > create mode 100644 include/uapi/linux/vduse.h
> >
> > diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
> > index 1409e40e6345..293ca3aef358 100644
> > --- a/Documentation/userspace-api/ioctl/ioctl-number.rst
> > +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
> > @@ -300,6 +300,7 @@ Code Seq# Include File Comments
> > 'z' 10-4F drivers/s390/crypto/zcrypt_api.h conflict!
> > '|' 00-7F linux/media.h
> > 0x80 00-1F linux/fb.h
> > +0x81 00-1F linux/vduse.h
> > 0x89 00-06 arch/x86/include/asm/sockios.h
> > 0x89 0B-DF linux/sockios.h
> > 0x89 E0-EF linux/sockios.h SIOCPROTOPRIVATE range
> > diff --git a/drivers/vdpa/Kconfig b/drivers/vdpa/Kconfig
> > index a503c1b2bfd9..6e23bce6433a 100644
> > --- a/drivers/vdpa/Kconfig
> > +++ b/drivers/vdpa/Kconfig
> > @@ -33,6 +33,16 @@ config VDPA_SIM_BLOCK
> > vDPA block device simulator which terminates IO request in a
> > memory buffer.
> >
> > +config VDPA_USER
> > + tristate "VDUSE (vDPA Device in Userspace) support"
> > + depends on EVENTFD && MMU && HAS_DMA
> > + select DMA_OPS
> > + select VHOST_IOTLB
> > + select IOMMU_IOVA
> > + help
> > + With VDUSE it is possible to emulate a vDPA Device
> > + in a userspace program.
> > +
> > config IFCVF
> > tristate "Intel IFC VF vDPA driver"
> > depends on PCI_MSI
> > diff --git a/drivers/vdpa/Makefile b/drivers/vdpa/Makefile
> > index 67fe7f3d6943..f02ebed33f19 100644
> > --- a/drivers/vdpa/Makefile
> > +++ b/drivers/vdpa/Makefile
> > @@ -1,6 +1,7 @@
> > # SPDX-License-Identifier: GPL-2.0
> > obj-$(CONFIG_VDPA) += vdpa.o
> > obj-$(CONFIG_VDPA_SIM) += vdpa_sim/
> > +obj-$(CONFIG_VDPA_USER) += vdpa_user/
> > obj-$(CONFIG_IFCVF) += ifcvf/
> > obj-$(CONFIG_MLX5_VDPA) += mlx5/
> > obj-$(CONFIG_VP_VDPA) += virtio_pci/
> > diff --git a/drivers/vdpa/vdpa_user/Makefile b/drivers/vdpa/vdpa_user/Makefile
> > new file mode 100644
> > index 000000000000..260e0b26af99
> > --- /dev/null
> > +++ b/drivers/vdpa/vdpa_user/Makefile
> > @@ -0,0 +1,5 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +
> > +vduse-y := vduse_dev.o iova_domain.o
> > +
> > +obj-$(CONFIG_VDPA_USER) += vduse.o
> > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > new file mode 100644
> > index 000000000000..c994a4a4660c
> > --- /dev/null
> > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > @@ -0,0 +1,1502 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * VDUSE: vDPA Device in Userspace
> > + *
> > + * Copyright (C) 2020-2021 Bytedance Inc. and/or its affiliates. All rights reserved.
> > + *
> > + * Author: Xie Yongji <xieyongji@bytedance.com>
> > + *
> > + */
> > +
> > +#include <linux/init.h>
> > +#include <linux/module.h>
> > +#include <linux/cdev.h>
> > +#include <linux/device.h>
> > +#include <linux/eventfd.h>
> > +#include <linux/slab.h>
> > +#include <linux/wait.h>
> > +#include <linux/dma-map-ops.h>
> > +#include <linux/poll.h>
> > +#include <linux/file.h>
> > +#include <linux/uio.h>
> > +#include <linux/vdpa.h>
> > +#include <linux/nospec.h>
> > +#include <uapi/linux/vduse.h>
> > +#include <uapi/linux/vdpa.h>
> > +#include <uapi/linux/virtio_config.h>
> > +#include <uapi/linux/virtio_ids.h>
> > +#include <uapi/linux/virtio_blk.h>
> > +#include <linux/mod_devicetable.h>
> > +
> > +#include "iova_domain.h"
> > +
> > +#define DRV_AUTHOR "Yongji Xie <xieyongji@bytedance.com>"
> > +#define DRV_DESC "vDPA Device in Userspace"
> > +#define DRV_LICENSE "GPL v2"
> > +
> > +#define VDUSE_DEV_MAX (1U << MINORBITS)
> > +#define VDUSE_MAX_BOUNCE_SIZE (64 * 1024 * 1024)
> > +#define VDUSE_IOVA_SIZE (128 * 1024 * 1024)
> > +#define VDUSE_REQUEST_TIMEOUT 30
> > +
> > +struct vduse_virtqueue {
> > + u16 index;
> > + u16 num_max;
> > + u32 num;
> > + u64 desc_addr;
> > + u64 driver_addr;
> > + u64 device_addr;
> > + struct vdpa_vq_state state;
> > + bool ready;
> > + bool kicked;
> > + spinlock_t kick_lock;
> > + spinlock_t irq_lock;
> > + struct eventfd_ctx *kickfd;
> > + struct vdpa_callback cb;
> > + struct work_struct inject;
> > +};
> > +
> > +struct vduse_dev;
> > +
> > +struct vduse_vdpa {
> > + struct vdpa_device vdpa;
> > + struct vduse_dev *dev;
> > +};
> > +
> > +struct vduse_dev {
> > + struct vduse_vdpa *vdev;
> > + struct device *dev;
> > + struct vduse_virtqueue *vqs;
> > + struct vduse_iova_domain *domain;
> > + char *name;
> > + struct mutex lock;
> > + spinlock_t msg_lock;
> > + u64 msg_unique;
> > + wait_queue_head_t waitq;
> > + struct list_head send_list;
> > + struct list_head recv_list;
> > + struct vdpa_callback config_cb;
> > + struct work_struct inject;
> > + spinlock_t irq_lock;
> > + int minor;
> > + bool connected;
> > + u64 api_version;
> > + u64 device_features;
> > + u64 driver_features;
> > + u32 device_id;
> > + u32 vendor_id;
> > + u32 generation;
> > + u32 config_size;
> > + void *config;
> > + u8 status;
> > + u32 vq_num;
> > + u32 vq_align;
> > +};
> > +
> > +struct vduse_dev_msg {
> > + struct vduse_dev_request req;
> > + struct vduse_dev_response resp;
> > + struct list_head list;
> > + wait_queue_head_t waitq;
> > + bool completed;
> > +};
> > +
> > +struct vduse_control {
> > + u64 api_version;
> > +};
> > +
> > +static DEFINE_MUTEX(vduse_lock);
> > +static DEFINE_IDR(vduse_idr);
> > +
> > +static dev_t vduse_major;
> > +static struct class *vduse_class;
> > +static struct cdev vduse_ctrl_cdev;
> > +static struct cdev vduse_cdev;
> > +static struct workqueue_struct *vduse_irq_wq;
> > +
> > +static u32 allowed_device_id[] = {
> > + VIRTIO_ID_BLOCK,
> > +};
> > +
> > +static inline struct vduse_dev *vdpa_to_vduse(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_vdpa *vdev = container_of(vdpa, struct vduse_vdpa, vdpa);
> > +
> > + return vdev->dev;
> > +}
> > +
> > +static inline struct vduse_dev *dev_to_vduse(struct device *dev)
> > +{
> > + struct vdpa_device *vdpa = dev_to_vdpa(dev);
> > +
> > + return vdpa_to_vduse(vdpa);
> > +}
> > +
> > +static struct vduse_dev_msg *vduse_find_msg(struct list_head *head,
> > + uint32_t request_id)
> > +{
> > + struct vduse_dev_msg *msg;
> > +
> > + list_for_each_entry(msg, head, list) {
> > + if (msg->req.request_id == request_id) {
> > + list_del(&msg->list);
> > + return msg;
> > + }
> > + }
> > +
> > + return NULL;
> > +}
> > +
> > +static struct vduse_dev_msg *vduse_dequeue_msg(struct list_head *head)
> > +{
> > + struct vduse_dev_msg *msg = NULL;
> > +
> > + if (!list_empty(head)) {
> > + msg = list_first_entry(head, struct vduse_dev_msg, list);
> > + list_del(&msg->list);
> > + }
> > +
> > + return msg;
> > +}
> > +
> > +static void vduse_enqueue_msg(struct list_head *head,
> > + struct vduse_dev_msg *msg)
> > +{
> > + list_add_tail(&msg->list, head);
> > +}
> > +
> > +static int vduse_dev_msg_sync(struct vduse_dev *dev,
> > + struct vduse_dev_msg *msg)
> > +{
> > + int ret;
> > +
> > + init_waitqueue_head(&msg->waitq);
> > + spin_lock(&dev->msg_lock);
> > + msg->req.request_id = dev->msg_unique++;
> > + vduse_enqueue_msg(&dev->send_list, msg);
> > + wake_up(&dev->waitq);
> > + spin_unlock(&dev->msg_lock);
> > +
> > + wait_event_killable_timeout(msg->waitq, msg->completed,
> > + VDUSE_REQUEST_TIMEOUT * HZ);
> > + spin_lock(&dev->msg_lock);
> > + if (!msg->completed) {
> > + list_del(&msg->list);
> > + msg->resp.result = VDUSE_REQ_RESULT_FAILED;
> > + }
> > + ret = (msg->resp.result == VDUSE_REQ_RESULT_OK) ? 0 : -EIO;
>
>
> I think we should mark the device as malfunction when there is a timeout
> and forbid any userspace operations except for the destroy aftwards for
> safety.
>
Is it necessary? Actually we can't stop the dataplane processing in
userspace. It doesn’t seem to help much if we only forbid ioctl and
read/write. And there might be some messages that can tolerate
failure. Even userspace can do something like NEEDS_RESET to do
recovery.
>
> > + spin_unlock(&dev->msg_lock);
> > +
> > + return ret;
> > +}
> > +
> > +static int vduse_dev_get_vq_state_packed(struct vduse_dev *dev,
> > + struct vduse_virtqueue *vq,
> > + struct vdpa_vq_state_packed *packed)
> > +{
> > + struct vduse_dev_msg msg = { 0 };
> > + int ret;
> > +
> > + msg.req.type = VDUSE_GET_VQ_STATE;
> > + msg.req.vq_state.index = vq->index;
> > +
> > + ret = vduse_dev_msg_sync(dev, &msg);
> > + if (ret)
> > + return ret;
> > +
> > + packed->last_avail_counter =
> > + msg.resp.vq_state.packed.last_avail_counter;
> > + packed->last_avail_idx = msg.resp.vq_state.packed.last_avail_idx;
> > + packed->last_used_counter = msg.resp.vq_state.packed.last_used_counter;
> > + packed->last_used_idx = msg.resp.vq_state.packed.last_used_idx;
> > +
> > + return 0;
> > +}
> > +
> > +static int vduse_dev_get_vq_state_split(struct vduse_dev *dev,
> > + struct vduse_virtqueue *vq,
> > + struct vdpa_vq_state_split *split)
> > +{
> > + struct vduse_dev_msg msg = { 0 };
> > + int ret;
> > +
> > + msg.req.type = VDUSE_GET_VQ_STATE;
> > + msg.req.vq_state.index = vq->index;
> > +
> > + ret = vduse_dev_msg_sync(dev, &msg);
> > + if (ret)
> > + return ret;
> > +
> > + split->avail_index = msg.resp.vq_state.split.avail_index;
> > +
> > + return 0;
> > +}
> > +
> > +static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
> > +{
> > + struct vduse_dev_msg msg = { 0 };
> > +
> > + msg.req.type = VDUSE_SET_STATUS;
> > + msg.req.s.status = status;
> > +
> > + return vduse_dev_msg_sync(dev, &msg);
> > +}
> > +
> > +static int vduse_dev_update_iotlb(struct vduse_dev *dev,
> > + u64 start, u64 last)
> > +{
> > + struct vduse_dev_msg msg = { 0 };
> > +
> > + if (last < start)
> > + return -EINVAL;
> > +
> > + msg.req.type = VDUSE_UPDATE_IOTLB;
> > + msg.req.iova.start = start;
> > + msg.req.iova.last = last;
> > +
> > + return vduse_dev_msg_sync(dev, &msg);
> > +}
> > +
> > +static ssize_t vduse_dev_read_iter(struct kiocb *iocb, struct iov_iter *to)
> > +{
> > + struct file *file = iocb->ki_filp;
> > + struct vduse_dev *dev = file->private_data;
> > + struct vduse_dev_msg *msg;
> > + int size = sizeof(struct vduse_dev_request);
> > + ssize_t ret;
> > +
> > + if (iov_iter_count(to) < size)
> > + return -EINVAL;
> > +
> > + spin_lock(&dev->msg_lock);
> > + while (1) {
> > + msg = vduse_dequeue_msg(&dev->send_list);
> > + if (msg)
> > + break;
> > +
> > + ret = -EAGAIN;
> > + if (file->f_flags & O_NONBLOCK)
> > + goto unlock;
> > +
> > + spin_unlock(&dev->msg_lock);
> > + ret = wait_event_interruptible_exclusive(dev->waitq,
> > + !list_empty(&dev->send_list));
> > + if (ret)
> > + return ret;
> > +
> > + spin_lock(&dev->msg_lock);
> > + }
> > + spin_unlock(&dev->msg_lock);
> > + ret = copy_to_iter(&msg->req, size, to);
> > + spin_lock(&dev->msg_lock);
> > + if (ret != size) {
> > + ret = -EFAULT;
> > + vduse_enqueue_msg(&dev->send_list, msg);
> > + goto unlock;
> > + }
> > + vduse_enqueue_msg(&dev->recv_list, msg);
> > +unlock:
> > + spin_unlock(&dev->msg_lock);
> > +
> > + return ret;
> > +}
> > +
> > +static ssize_t vduse_dev_write_iter(struct kiocb *iocb, struct iov_iter *from)
> > +{
> > + struct file *file = iocb->ki_filp;
> > + struct vduse_dev *dev = file->private_data;
> > + struct vduse_dev_response resp;
> > + struct vduse_dev_msg *msg;
> > + size_t ret;
> > +
> > + ret = copy_from_iter(&resp, sizeof(resp), from);
> > + if (ret != sizeof(resp))
> > + return -EINVAL;
> > +
> > + spin_lock(&dev->msg_lock);
> > + msg = vduse_find_msg(&dev->recv_list, resp.request_id);
> > + if (!msg) {
> > + ret = -ENOENT;
> > + goto unlock;
> > + }
> > +
> > + memcpy(&msg->resp, &resp, sizeof(resp));
> > + msg->completed = 1;
> > + wake_up(&msg->waitq);
> > +unlock:
> > + spin_unlock(&dev->msg_lock);
> > +
> > + return ret;
> > +}
> > +
> > +static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
> > +{
> > + struct vduse_dev *dev = file->private_data;
> > + __poll_t mask = 0;
> > +
> > + poll_wait(file, &dev->waitq, wait);
> > +
> > + if (!list_empty(&dev->send_list))
> > + mask |= EPOLLIN | EPOLLRDNORM;
> > + if (!list_empty(&dev->recv_list))
> > + mask |= EPOLLOUT | EPOLLWRNORM;
> > +
> > + return mask;
> > +}
> > +
> > +static void vduse_dev_reset(struct vduse_dev *dev)
> > +{
> > + int i;
> > + struct vduse_iova_domain *domain = dev->domain;
> > +
> > + /* The coherent mappings are handled in vduse_dev_free_coherent() */
> > + if (domain->bounce_map)
> > + vduse_domain_reset_bounce_map(domain);
> > +
> > + dev->driver_features = 0;
> > + dev->generation++;
> > + spin_lock(&dev->irq_lock);
> > + dev->config_cb.callback = NULL;
> > + dev->config_cb.private = NULL;
> > + spin_unlock(&dev->irq_lock);
> > + flush_work(&dev->inject);
> > +
> > + for (i = 0; i < dev->vq_num; i++) {
> > + struct vduse_virtqueue *vq = &dev->vqs[i];
> > +
> > + vq->ready = false;
> > + vq->desc_addr = 0;
> > + vq->driver_addr = 0;
> > + vq->device_addr = 0;
> > + vq->num = 0;
> > + memset(&vq->state, 0, sizeof(vq->state));
> > +
> > + spin_lock(&vq->kick_lock);
> > + vq->kicked = false;
> > + if (vq->kickfd)
> > + eventfd_ctx_put(vq->kickfd);
> > + vq->kickfd = NULL;
> > + spin_unlock(&vq->kick_lock);
> > +
> > + spin_lock(&vq->irq_lock);
> > + vq->cb.callback = NULL;
> > + vq->cb.private = NULL;
> > + spin_unlock(&vq->irq_lock);
> > + flush_work(&vq->inject);
> > + }
> > +}
> > +
> > +static int vduse_vdpa_set_vq_address(struct vdpa_device *vdpa, u16 idx,
> > + u64 desc_area, u64 driver_area,
> > + u64 device_area)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + vq->desc_addr = desc_area;
> > + vq->driver_addr = driver_area;
> > + vq->device_addr = device_area;
> > +
> > + return 0;
> > +}
> > +
> > +static void vduse_vdpa_kick_vq(struct vdpa_device *vdpa, u16 idx)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + spin_lock(&vq->kick_lock);
> > + if (!vq->ready)
> > + goto unlock;
> > +
> > + if (vq->kickfd)
> > + eventfd_signal(vq->kickfd, 1);
> > + else
> > + vq->kicked = true;
> > +unlock:
> > + spin_unlock(&vq->kick_lock);
> > +}
> > +
> > +static void vduse_vdpa_set_vq_cb(struct vdpa_device *vdpa, u16 idx,
> > + struct vdpa_callback *cb)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + spin_lock(&vq->irq_lock);
> > + vq->cb.callback = cb->callback;
> > + vq->cb.private = cb->private;
> > + spin_unlock(&vq->irq_lock);
> > +}
> > +
> > +static void vduse_vdpa_set_vq_num(struct vdpa_device *vdpa, u16 idx, u32 num)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + vq->num = num;
> > +}
> > +
> > +static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
> > + u16 idx, bool ready)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + vq->ready = ready;
> > +}
> > +
> > +static bool vduse_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 idx)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + return vq->ready;
> > +}
> > +
> > +static int vduse_vdpa_set_vq_state(struct vdpa_device *vdpa, u16 idx,
> > + const struct vdpa_vq_state *state)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + if (dev->driver_features & BIT_ULL(VIRTIO_F_RING_PACKED)) {
> > + vq->state.packed.last_avail_counter =
> > + state->packed.last_avail_counter;
> > + vq->state.packed.last_avail_idx = state->packed.last_avail_idx;
> > + vq->state.packed.last_used_counter =
> > + state->packed.last_used_counter;
> > + vq->state.packed.last_used_idx = state->packed.last_used_idx;
> > + } else
> > + vq->state.split.avail_index = state->split.avail_index;
> > +
> > + return 0;
> > +}
> > +
> > +static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
> > + struct vdpa_vq_state *state)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + if (dev->driver_features & BIT_ULL(VIRTIO_F_RING_PACKED))
> > + return vduse_dev_get_vq_state_packed(dev, vq, &state->packed);
> > +
> > + return vduse_dev_get_vq_state_split(dev, vq, &state->split);
> > +}
> > +
> > +static u32 vduse_vdpa_get_vq_align(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->vq_align;
> > +}
> > +
> > +static u64 vduse_vdpa_get_features(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->device_features;
> > +}
> > +
> > +static int vduse_vdpa_set_features(struct vdpa_device *vdpa, u64 features)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + dev->driver_features = features;
> > + return 0;
> > +}
> > +
> > +static void vduse_vdpa_set_config_cb(struct vdpa_device *vdpa,
> > + struct vdpa_callback *cb)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + spin_lock(&dev->irq_lock);
> > + dev->config_cb.callback = cb->callback;
> > + dev->config_cb.private = cb->private;
> > + spin_unlock(&dev->irq_lock);
> > +}
> > +
> > +static u16 vduse_vdpa_get_vq_num_max(struct vdpa_device *vdpa, u16 idx)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->vqs[idx].num_max;
> > +}
> > +
> > +static u32 vduse_vdpa_get_device_id(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->device_id;
> > +}
> > +
> > +static u32 vduse_vdpa_get_vendor_id(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->vendor_id;
> > +}
> > +
> > +static u8 vduse_vdpa_get_status(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->status;
> > +}
> > +
> > +static void vduse_vdpa_set_status(struct vdpa_device *vdpa, u8 status)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + if (vduse_dev_set_status(dev, status))
> > + return;
> > +
> > + dev->status = status;
>
>
> It looks to me such design exclude the possibility of letting userspace
> device to set bit like (NEEDS_RESET) in the future.
>
Looks like we can achieve that via the ioctl.
> I wonder if it's better to do something similar to ccw:
>
> 1) requires the userspace to update the status bit in the response
> 2) update the dev->status to the status in the response if no timeout
>
> Then userspace could then set NEEDS_RESET if necessary.
>
But NEEDS_RESET does not only happen in this case.
>
> > + if (status == 0)
> > + vduse_dev_reset(dev);
> > +}
> > +
> > +static size_t vduse_vdpa_get_config_size(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->config_size;
> > +}
> > +
> > +static void vduse_vdpa_get_config(struct vdpa_device *vdpa, unsigned int offset,
> > + void *buf, unsigned int len)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + if (len > dev->config_size - offset)
> > + return;
> > +
> > + memcpy(buf, dev->config + offset, len);
> > +}
> > +
> > +static void vduse_vdpa_set_config(struct vdpa_device *vdpa, unsigned int offset,
> > + const void *buf, unsigned int len)
> > +{
> > + /* Now we only support read-only configuration space */
> > +}
> > +
> > +static u32 vduse_vdpa_get_generation(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->generation;
> > +}
> > +
> > +static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
> > + struct vhost_iotlb *iotlb)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + int ret;
> > +
> > + ret = vduse_domain_set_map(dev->domain, iotlb);
> > + if (ret)
> > + return ret;
> > +
> > + ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
> > + if (ret) {
> > + vduse_domain_clear_map(dev->domain, iotlb);
> > + return ret;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static void vduse_vdpa_free(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + dev->vdev = NULL;
> > +}
> > +
> > +static const struct vdpa_config_ops vduse_vdpa_config_ops = {
> > + .set_vq_address = vduse_vdpa_set_vq_address,
> > + .kick_vq = vduse_vdpa_kick_vq,
> > + .set_vq_cb = vduse_vdpa_set_vq_cb,
> > + .set_vq_num = vduse_vdpa_set_vq_num,
> > + .set_vq_ready = vduse_vdpa_set_vq_ready,
> > + .get_vq_ready = vduse_vdpa_get_vq_ready,
> > + .set_vq_state = vduse_vdpa_set_vq_state,
> > + .get_vq_state = vduse_vdpa_get_vq_state,
> > + .get_vq_align = vduse_vdpa_get_vq_align,
> > + .get_features = vduse_vdpa_get_features,
> > + .set_features = vduse_vdpa_set_features,
> > + .set_config_cb = vduse_vdpa_set_config_cb,
> > + .get_vq_num_max = vduse_vdpa_get_vq_num_max,
> > + .get_device_id = vduse_vdpa_get_device_id,
> > + .get_vendor_id = vduse_vdpa_get_vendor_id,
> > + .get_status = vduse_vdpa_get_status,
> > + .set_status = vduse_vdpa_set_status,
> > + .get_config_size = vduse_vdpa_get_config_size,
> > + .get_config = vduse_vdpa_get_config,
> > + .set_config = vduse_vdpa_set_config,
> > + .get_generation = vduse_vdpa_get_generation,
> > + .set_map = vduse_vdpa_set_map,
> > + .free = vduse_vdpa_free,
> > +};
> > +
> > +static dma_addr_t vduse_dev_map_page(struct device *dev, struct page *page,
> > + unsigned long offset, size_t size,
> > + enum dma_data_direction dir,
> > + unsigned long attrs)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > +
> > + return vduse_domain_map_page(domain, page, offset, size, dir, attrs);
> > +}
> > +
> > +static void vduse_dev_unmap_page(struct device *dev, dma_addr_t dma_addr,
> > + size_t size, enum dma_data_direction dir,
> > + unsigned long attrs)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > +
> > + return vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
> > +}
> > +
> > +static void *vduse_dev_alloc_coherent(struct device *dev, size_t size,
> > + dma_addr_t *dma_addr, gfp_t flag,
> > + unsigned long attrs)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > + unsigned long iova;
> > + void *addr;
> > +
> > + *dma_addr = DMA_MAPPING_ERROR;
> > + addr = vduse_domain_alloc_coherent(domain, size,
> > + (dma_addr_t *)&iova, flag, attrs);
> > + if (!addr)
> > + return NULL;
> > +
> > + *dma_addr = (dma_addr_t)iova;
> > +
> > + return addr;
> > +}
> > +
> > +static void vduse_dev_free_coherent(struct device *dev, size_t size,
> > + void *vaddr, dma_addr_t dma_addr,
> > + unsigned long attrs)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > +
> > + vduse_domain_free_coherent(domain, size, vaddr, dma_addr, attrs);
> > +}
> > +
> > +static size_t vduse_dev_max_mapping_size(struct device *dev)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > +
> > + return domain->bounce_size;
> > +}
> > +
> > +static const struct dma_map_ops vduse_dev_dma_ops = {
> > + .map_page = vduse_dev_map_page,
> > + .unmap_page = vduse_dev_unmap_page,
> > + .alloc = vduse_dev_alloc_coherent,
> > + .free = vduse_dev_free_coherent,
> > + .max_mapping_size = vduse_dev_max_mapping_size,
> > +};
> > +
> > +static unsigned int perm_to_file_flags(u8 perm)
> > +{
> > + unsigned int flags = 0;
> > +
> > + switch (perm) {
> > + case VDUSE_ACCESS_WO:
> > + flags |= O_WRONLY;
> > + break;
> > + case VDUSE_ACCESS_RO:
> > + flags |= O_RDONLY;
> > + break;
> > + case VDUSE_ACCESS_RW:
> > + flags |= O_RDWR;
> > + break;
> > + default:
> > + WARN(1, "invalidate vhost IOTLB permission\n");
> > + break;
> > + }
> > +
> > + return flags;
> > +}
> > +
> > +static int vduse_kickfd_setup(struct vduse_dev *dev,
> > + struct vduse_vq_eventfd *eventfd)
> > +{
> > + struct eventfd_ctx *ctx = NULL;
> > + struct vduse_virtqueue *vq;
> > + u32 index;
> > +
> > + if (eventfd->index >= dev->vq_num)
> > + return -EINVAL;
> > +
> > + index = array_index_nospec(eventfd->index, dev->vq_num);
> > + vq = &dev->vqs[index];
> > + if (eventfd->fd >= 0) {
> > + ctx = eventfd_ctx_fdget(eventfd->fd);
> > + if (IS_ERR(ctx))
> > + return PTR_ERR(ctx);
> > + } else if (eventfd->fd != VDUSE_EVENTFD_DEASSIGN)
> > + return 0;
> > +
> > + spin_lock(&vq->kick_lock);
> > + if (vq->kickfd)
> > + eventfd_ctx_put(vq->kickfd);
> > + vq->kickfd = ctx;
> > + if (vq->ready && vq->kicked && vq->kickfd) {
> > + eventfd_signal(vq->kickfd, 1);
> > + vq->kicked = false;
> > + }
> > + spin_unlock(&vq->kick_lock);
> > +
> > + return 0;
> > +}
> > +
> > +static bool vduse_dev_is_ready(struct vduse_dev *dev)
> > +{
> > + int i;
> > +
> > + for (i = 0; i < dev->vq_num; i++)
> > + if (!dev->vqs[i].num_max)
> > + return false;
> > +
> > + return true;
> > +}
> > +
> > +static void vduse_dev_irq_inject(struct work_struct *work)
> > +{
> > + struct vduse_dev *dev = container_of(work, struct vduse_dev, inject);
> > +
> > + spin_lock_irq(&dev->irq_lock);
> > + if (dev->config_cb.callback)
> > + dev->config_cb.callback(dev->config_cb.private);
> > + spin_unlock_irq(&dev->irq_lock);
> > +}
> > +
> > +static void vduse_vq_irq_inject(struct work_struct *work)
> > +{
> > + struct vduse_virtqueue *vq = container_of(work,
> > + struct vduse_virtqueue, inject);
> > +
> > + spin_lock_irq(&vq->irq_lock);
> > + if (vq->ready && vq->cb.callback)
> > + vq->cb.callback(vq->cb.private);
> > + spin_unlock_irq(&vq->irq_lock);
> > +}
> > +
> > +static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
> > + unsigned long arg)
> > +{
> > + struct vduse_dev *dev = file->private_data;
> > + void __user *argp = (void __user *)arg;
> > + int ret;
> > +
> > + switch (cmd) {
> > + case VDUSE_IOTLB_GET_FD: {
> > + struct vduse_iotlb_entry entry;
> > + struct vhost_iotlb_map *map;
> > + struct vdpa_map_file *map_file;
> > + struct vduse_iova_domain *domain = dev->domain;
> > + struct file *f = NULL;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&entry, argp, sizeof(entry)))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (entry.start > entry.last)
> > + break;
> > +
> > + spin_lock(&domain->iotlb_lock);
> > + map = vhost_iotlb_itree_first(domain->iotlb,
> > + entry.start, entry.last);
> > + if (map) {
> > + map_file = (struct vdpa_map_file *)map->opaque;
> > + f = get_file(map_file->file);
> > + entry.offset = map_file->offset;
> > + entry.start = map->start;
> > + entry.last = map->last;
> > + entry.perm = map->perm;
> > + }
> > + spin_unlock(&domain->iotlb_lock);
> > + ret = -EINVAL;
> > + if (!f)
> > + break;
> > +
> > + ret = -EFAULT;
> > + if (copy_to_user(argp, &entry, sizeof(entry))) {
> > + fput(f);
> > + break;
> > + }
> > + ret = receive_fd(f, perm_to_file_flags(entry.perm));
> > + fput(f);
> > + break;
> > + }
> > + case VDUSE_DEV_GET_FEATURES:
>
>
> Let's add a comment to explain here. E.g we just mirror what driver
> wrote and the drier is expected to check FEATURE_OK.
>
I already document some details in include/uapi/linux/vduse.h and
Documentation/userspace-api/vduse.rst
>
> > + ret = put_user(dev->driver_features, (u64 __user *)argp);
> > + break;
> > + case VDUSE_DEV_SET_CONFIG: {
> > + struct vduse_config_data config;
> > + unsigned long size = offsetof(struct vduse_config_data,
> > + buffer);
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&config, argp, size))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (config.length == 0 ||
> > + config.length > dev->config_size - config.offset)
> > + break;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(dev->config + config.offset, argp + size,
> > + config.length))
> > + break;
> > +
> > + ret = 0;
> > + break;
> > + }
> > + case VDUSE_DEV_INJECT_IRQ:
>
>
> It's better to have "config" in the name.
>
OK.
>
> > + ret = 0;
> > + queue_work(vduse_irq_wq, &dev->inject);
> > + break;
> > + case VDUSE_VQ_SETUP: {
> > + struct vduse_vq_config config;
> > + u32 index;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&config, argp, sizeof(config)))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (config.index >= dev->vq_num)
> > + break;
> > +
> > + index = array_index_nospec(config.index, dev->vq_num);
> > + dev->vqs[index].num_max = config.max_size;
> > + ret = 0;
> > + break;
> > + }
> > + case VDUSE_VQ_GET_INFO: {
> > + struct vduse_vq_info vq_info;
> > + struct vduse_virtqueue *vq;
> > + u32 index;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&vq_info, argp, sizeof(vq_info)))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (vq_info.index >= dev->vq_num)
> > + break;
> > +
> > + index = array_index_nospec(vq_info.index, dev->vq_num);
> > + vq = &dev->vqs[index];
> > + vq_info.desc_addr = vq->desc_addr;
> > + vq_info.driver_addr = vq->driver_addr;
> > + vq_info.device_addr = vq->device_addr;
> > + vq_info.num = vq->num;
> > +
> > + if (dev->driver_features & BIT_ULL(VIRTIO_F_RING_PACKED)) {
> > + vq_info.packed.last_avail_counter =
> > + vq->state.packed.last_avail_counter;
> > + vq_info.packed.last_avail_idx =
> > + vq->state.packed.last_avail_idx;
> > + vq_info.packed.last_used_counter =
> > + vq->state.packed.last_used_counter;
> > + vq_info.packed.last_used_idx =
> > + vq->state.packed.last_used_idx;
> > + } else
> > + vq_info.split.avail_index =
> > + vq->state.split.avail_index;
> > +
> > + vq_info.ready = vq->ready;
> > +
> > + ret = -EFAULT;
> > + if (copy_to_user(argp, &vq_info, sizeof(vq_info)))
> > + break;
> > +
> > + ret = 0;
> > + break;
> > + }
> > + case VDUSE_VQ_SETUP_KICKFD: {
> > + struct vduse_vq_eventfd eventfd;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&eventfd, argp, sizeof(eventfd)))
> > + break;
> > +
> > + ret = vduse_kickfd_setup(dev, &eventfd);
> > + break;
> > + }
> > + case VDUSE_VQ_INJECT_IRQ: {
> > + u32 index;
> > +
> > + ret = -EFAULT;
> > + if (get_user(index, (u32 __user *)argp))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (index >= dev->vq_num)
> > + break;
> > +
> > + ret = 0;
> > + index = array_index_nospec(index, dev->vq_num);
> > + queue_work(vduse_irq_wq, &dev->vqs[index].inject);
> > + break;
> > + }
> > + default:
> > + ret = -ENOIOCTLCMD;
> > + break;
> > + }
> > +
> > + return ret;
> > +}
> > +
> > +static int vduse_dev_release(struct inode *inode, struct file *file)
> > +{
> > + struct vduse_dev *dev = file->private_data;
> > +
> > + spin_lock(&dev->msg_lock);
> > + /* Make sure the inflight messages can processed after reconncection */
> > + list_splice_init(&dev->recv_list, &dev->send_list);
> > + spin_unlock(&dev->msg_lock);
> > + dev->connected = false;
> > +
> > + return 0;
> > +}
> > +
> > +static struct vduse_dev *vduse_dev_get_from_minor(int minor)
> > +{
> > + struct vduse_dev *dev;
> > +
> > + mutex_lock(&vduse_lock);
> > + dev = idr_find(&vduse_idr, minor);
> > + mutex_unlock(&vduse_lock);
> > +
> > + return dev;
> > +}
> > +
> > +static int vduse_dev_open(struct inode *inode, struct file *file)
> > +{
> > + int ret;
> > + struct vduse_dev *dev = vduse_dev_get_from_minor(iminor(inode));
> > +
> > + if (!dev)
> > + return -ENODEV;
> > +
> > + ret = -EBUSY;
> > + mutex_lock(&dev->lock);
> > + if (dev->connected)
> > + goto unlock;
> > +
> > + ret = 0;
> > + dev->connected = true;
> > + file->private_data = dev;
> > +unlock:
> > + mutex_unlock(&dev->lock);
> > +
> > + return ret;
> > +}
> > +
> > +static const struct file_operations vduse_dev_fops = {
> > + .owner = THIS_MODULE,
> > + .open = vduse_dev_open,
> > + .release = vduse_dev_release,
> > + .read_iter = vduse_dev_read_iter,
> > + .write_iter = vduse_dev_write_iter,
> > + .poll = vduse_dev_poll,
> > + .unlocked_ioctl = vduse_dev_ioctl,
> > + .compat_ioctl = compat_ptr_ioctl,
> > + .llseek = noop_llseek,
> > +};
> > +
> > +static struct vduse_dev *vduse_dev_create(void)
> > +{
> > + struct vduse_dev *dev = kzalloc(sizeof(*dev), GFP_KERNEL);
> > +
> > + if (!dev)
> > + return NULL;
> > +
> > + mutex_init(&dev->lock);
> > + spin_lock_init(&dev->msg_lock);
> > + INIT_LIST_HEAD(&dev->send_list);
> > + INIT_LIST_HEAD(&dev->recv_list);
> > + spin_lock_init(&dev->irq_lock);
> > +
> > + INIT_WORK(&dev->inject, vduse_dev_irq_inject);
> > + init_waitqueue_head(&dev->waitq);
> > +
> > + return dev;
> > +}
> > +
> > +static void vduse_dev_destroy(struct vduse_dev *dev)
> > +{
> > + kfree(dev);
> > +}
> > +
> > +static struct vduse_dev *vduse_find_dev(const char *name)
> > +{
> > + struct vduse_dev *dev;
> > + int id;
> > +
> > + idr_for_each_entry(&vduse_idr, dev, id)
> > + if (!strcmp(dev->name, name))
> > + return dev;
> > +
> > + return NULL;
> > +}
> > +
> > +static int vduse_destroy_dev(char *name)
> > +{
> > + struct vduse_dev *dev = vduse_find_dev(name);
> > +
> > + if (!dev)
> > + return -EINVAL;
> > +
> > + mutex_lock(&dev->lock);
> > + if (dev->vdev || dev->connected) {
> > + mutex_unlock(&dev->lock);
> > + return -EBUSY;
> > + }
> > + dev->connected = true;
> > + mutex_unlock(&dev->lock);
> > +
> > + vduse_dev_reset(dev);
> > + device_destroy(vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
> > + idr_remove(&vduse_idr, dev->minor);
> > + kvfree(dev->config);
> > + kfree(dev->vqs);
> > + vduse_domain_destroy(dev->domain);
> > + kfree(dev->name);
> > + vduse_dev_destroy(dev);
> > + module_put(THIS_MODULE);
> > +
> > + return 0;
> > +}
> > +
> > +static bool device_is_allowed(u32 device_id)
> > +{
> > + int i;
> > +
> > + for (i = 0; i < ARRAY_SIZE(allowed_device_id); i++)
> > + if (allowed_device_id[i] == device_id)
> > + return true;
> > +
> > + return false;
> > +}
> > +
> > +static bool features_is_valid(u64 features)
> > +{
> > + if (!(features & (1ULL << VIRTIO_F_ACCESS_PLATFORM)))
> > + return false;
> > +
> > + /* Now we only support read-only configuration space */
> > + if (features & (1ULL << VIRTIO_BLK_F_CONFIG_WCE))
> > + return false;
> > +
> > + return true;
> > +}
> > +
> > +static bool vduse_validate_config(struct vduse_dev_config *config)
> > +{
> > + if (config->bounce_size > VDUSE_MAX_BOUNCE_SIZE)
> > + return false;
> > +
> > + if (config->vq_align > PAGE_SIZE)
> > + return false;
> > +
> > + if (config->config_size > PAGE_SIZE)
> > + return false;
>
>
> Any reason for this check?
>
To limit the kernel buffer size for configuration space. Is the
PAGE_SIZE enough?
Thanks,
Yongji
WARNING: multiple messages have this Message-ID (diff)
From: Yongji Xie <xieyongji@bytedance.com>
To: Jason Wang <jasowang@redhat.com>
Cc: kvm <kvm@vger.kernel.org>, "Michael S. Tsirkin" <mst@redhat.com>,
virtualization <virtualization@lists.linux-foundation.org>,
"Christian Brauner" <christian.brauner@canonical.com>,
"Jonathan Corbet" <corbet@lwn.net>,
"Matthew Wilcox" <willy@infradead.org>,
"Christoph Hellwig" <hch@infradead.org>,
"Dan Carpenter" <dan.carpenter@oracle.com>,
"Stefano Garzarella" <sgarzare@redhat.com>,
"Liu Xiaodong" <xiaodong.liu@intel.com>,
"Al Viro" <viro@zeniv.linux.org.uk>,
"Stefan Hajnoczi" <stefanha@redhat.com>,
songmuchun@bytedance.com, "Jens Axboe" <axboe@kernel.dk>,
"He Zhe" <zhe.he@windriver.com>,
"Greg KH" <gregkh@linuxfoundation.org>,
"Randy Dunlap" <rdunlap@infradead.org>,
linux-kernel <linux-kernel@vger.kernel.org>,
iommu@lists.linux-foundation.org, bcrl@kvack.org,
netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org,
"Mika Penttilä" <mika.penttila@nextfour.com>
Subject: Re: [PATCH v9 16/17] vduse: Introduce VDUSE - vDPA Device in Userspace
Date: Wed, 14 Jul 2021 14:49:21 +0800 [thread overview]
Message-ID: <CACycT3uh+wUeDM1H7JiCJTMeCVCBngURGKeXD-h+meekNNwiQw@mail.gmail.com> (raw)
In-Reply-To: <26116714-f485-eeab-4939-71c4c10c30de@redhat.com>
On Wed, Jul 14, 2021 at 1:45 PM Jason Wang <jasowang@redhat.com> wrote:
>
>
> 在 2021/7/13 下午4:46, Xie Yongji 写道:
> > This VDUSE driver enables implementing software-emulated vDPA
> > devices in userspace. The vDPA device is created by
> > ioctl(VDUSE_CREATE_DEV) on /dev/vduse/control. Then a char device
> > interface (/dev/vduse/$NAME) is exported to userspace for device
> > emulation.
> >
> > In order to make the device emulation more secure, the device's
> > control path is handled in kernel. A message mechnism is introduced
> > to forward some dataplane related control messages to userspace.
> >
> > And in the data path, the DMA buffer will be mapped into userspace
> > address space through different ways depending on the vDPA bus to
> > which the vDPA device is attached. In virtio-vdpa case, the MMU-based
> > IOMMU driver is used to achieve that. And in vhost-vdpa case, the
> > DMA buffer is reside in a userspace memory region which can be shared
> > to the VDUSE userspace processs via transferring the shmfd.
> >
> > For more details on VDUSE design and usage, please see the follow-on
> > Documentation commit.
> >
> > Signed-off-by: Xie Yongji <xieyongji@bytedance.com>
> > ---
> > Documentation/userspace-api/ioctl/ioctl-number.rst | 1 +
> > drivers/vdpa/Kconfig | 10 +
> > drivers/vdpa/Makefile | 1 +
> > drivers/vdpa/vdpa_user/Makefile | 5 +
> > drivers/vdpa/vdpa_user/vduse_dev.c | 1502 ++++++++++++++++++++
> > include/uapi/linux/vduse.h | 221 +++
> > 6 files changed, 1740 insertions(+)
> > create mode 100644 drivers/vdpa/vdpa_user/Makefile
> > create mode 100644 drivers/vdpa/vdpa_user/vduse_dev.c
> > create mode 100644 include/uapi/linux/vduse.h
> >
> > diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
> > index 1409e40e6345..293ca3aef358 100644
> > --- a/Documentation/userspace-api/ioctl/ioctl-number.rst
> > +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
> > @@ -300,6 +300,7 @@ Code Seq# Include File Comments
> > 'z' 10-4F drivers/s390/crypto/zcrypt_api.h conflict!
> > '|' 00-7F linux/media.h
> > 0x80 00-1F linux/fb.h
> > +0x81 00-1F linux/vduse.h
> > 0x89 00-06 arch/x86/include/asm/sockios.h
> > 0x89 0B-DF linux/sockios.h
> > 0x89 E0-EF linux/sockios.h SIOCPROTOPRIVATE range
> > diff --git a/drivers/vdpa/Kconfig b/drivers/vdpa/Kconfig
> > index a503c1b2bfd9..6e23bce6433a 100644
> > --- a/drivers/vdpa/Kconfig
> > +++ b/drivers/vdpa/Kconfig
> > @@ -33,6 +33,16 @@ config VDPA_SIM_BLOCK
> > vDPA block device simulator which terminates IO request in a
> > memory buffer.
> >
> > +config VDPA_USER
> > + tristate "VDUSE (vDPA Device in Userspace) support"
> > + depends on EVENTFD && MMU && HAS_DMA
> > + select DMA_OPS
> > + select VHOST_IOTLB
> > + select IOMMU_IOVA
> > + help
> > + With VDUSE it is possible to emulate a vDPA Device
> > + in a userspace program.
> > +
> > config IFCVF
> > tristate "Intel IFC VF vDPA driver"
> > depends on PCI_MSI
> > diff --git a/drivers/vdpa/Makefile b/drivers/vdpa/Makefile
> > index 67fe7f3d6943..f02ebed33f19 100644
> > --- a/drivers/vdpa/Makefile
> > +++ b/drivers/vdpa/Makefile
> > @@ -1,6 +1,7 @@
> > # SPDX-License-Identifier: GPL-2.0
> > obj-$(CONFIG_VDPA) += vdpa.o
> > obj-$(CONFIG_VDPA_SIM) += vdpa_sim/
> > +obj-$(CONFIG_VDPA_USER) += vdpa_user/
> > obj-$(CONFIG_IFCVF) += ifcvf/
> > obj-$(CONFIG_MLX5_VDPA) += mlx5/
> > obj-$(CONFIG_VP_VDPA) += virtio_pci/
> > diff --git a/drivers/vdpa/vdpa_user/Makefile b/drivers/vdpa/vdpa_user/Makefile
> > new file mode 100644
> > index 000000000000..260e0b26af99
> > --- /dev/null
> > +++ b/drivers/vdpa/vdpa_user/Makefile
> > @@ -0,0 +1,5 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +
> > +vduse-y := vduse_dev.o iova_domain.o
> > +
> > +obj-$(CONFIG_VDPA_USER) += vduse.o
> > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > new file mode 100644
> > index 000000000000..c994a4a4660c
> > --- /dev/null
> > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > @@ -0,0 +1,1502 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * VDUSE: vDPA Device in Userspace
> > + *
> > + * Copyright (C) 2020-2021 Bytedance Inc. and/or its affiliates. All rights reserved.
> > + *
> > + * Author: Xie Yongji <xieyongji@bytedance.com>
> > + *
> > + */
> > +
> > +#include <linux/init.h>
> > +#include <linux/module.h>
> > +#include <linux/cdev.h>
> > +#include <linux/device.h>
> > +#include <linux/eventfd.h>
> > +#include <linux/slab.h>
> > +#include <linux/wait.h>
> > +#include <linux/dma-map-ops.h>
> > +#include <linux/poll.h>
> > +#include <linux/file.h>
> > +#include <linux/uio.h>
> > +#include <linux/vdpa.h>
> > +#include <linux/nospec.h>
> > +#include <uapi/linux/vduse.h>
> > +#include <uapi/linux/vdpa.h>
> > +#include <uapi/linux/virtio_config.h>
> > +#include <uapi/linux/virtio_ids.h>
> > +#include <uapi/linux/virtio_blk.h>
> > +#include <linux/mod_devicetable.h>
> > +
> > +#include "iova_domain.h"
> > +
> > +#define DRV_AUTHOR "Yongji Xie <xieyongji@bytedance.com>"
> > +#define DRV_DESC "vDPA Device in Userspace"
> > +#define DRV_LICENSE "GPL v2"
> > +
> > +#define VDUSE_DEV_MAX (1U << MINORBITS)
> > +#define VDUSE_MAX_BOUNCE_SIZE (64 * 1024 * 1024)
> > +#define VDUSE_IOVA_SIZE (128 * 1024 * 1024)
> > +#define VDUSE_REQUEST_TIMEOUT 30
> > +
> > +struct vduse_virtqueue {
> > + u16 index;
> > + u16 num_max;
> > + u32 num;
> > + u64 desc_addr;
> > + u64 driver_addr;
> > + u64 device_addr;
> > + struct vdpa_vq_state state;
> > + bool ready;
> > + bool kicked;
> > + spinlock_t kick_lock;
> > + spinlock_t irq_lock;
> > + struct eventfd_ctx *kickfd;
> > + struct vdpa_callback cb;
> > + struct work_struct inject;
> > +};
> > +
> > +struct vduse_dev;
> > +
> > +struct vduse_vdpa {
> > + struct vdpa_device vdpa;
> > + struct vduse_dev *dev;
> > +};
> > +
> > +struct vduse_dev {
> > + struct vduse_vdpa *vdev;
> > + struct device *dev;
> > + struct vduse_virtqueue *vqs;
> > + struct vduse_iova_domain *domain;
> > + char *name;
> > + struct mutex lock;
> > + spinlock_t msg_lock;
> > + u64 msg_unique;
> > + wait_queue_head_t waitq;
> > + struct list_head send_list;
> > + struct list_head recv_list;
> > + struct vdpa_callback config_cb;
> > + struct work_struct inject;
> > + spinlock_t irq_lock;
> > + int minor;
> > + bool connected;
> > + u64 api_version;
> > + u64 device_features;
> > + u64 driver_features;
> > + u32 device_id;
> > + u32 vendor_id;
> > + u32 generation;
> > + u32 config_size;
> > + void *config;
> > + u8 status;
> > + u32 vq_num;
> > + u32 vq_align;
> > +};
> > +
> > +struct vduse_dev_msg {
> > + struct vduse_dev_request req;
> > + struct vduse_dev_response resp;
> > + struct list_head list;
> > + wait_queue_head_t waitq;
> > + bool completed;
> > +};
> > +
> > +struct vduse_control {
> > + u64 api_version;
> > +};
> > +
> > +static DEFINE_MUTEX(vduse_lock);
> > +static DEFINE_IDR(vduse_idr);
> > +
> > +static dev_t vduse_major;
> > +static struct class *vduse_class;
> > +static struct cdev vduse_ctrl_cdev;
> > +static struct cdev vduse_cdev;
> > +static struct workqueue_struct *vduse_irq_wq;
> > +
> > +static u32 allowed_device_id[] = {
> > + VIRTIO_ID_BLOCK,
> > +};
> > +
> > +static inline struct vduse_dev *vdpa_to_vduse(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_vdpa *vdev = container_of(vdpa, struct vduse_vdpa, vdpa);
> > +
> > + return vdev->dev;
> > +}
> > +
> > +static inline struct vduse_dev *dev_to_vduse(struct device *dev)
> > +{
> > + struct vdpa_device *vdpa = dev_to_vdpa(dev);
> > +
> > + return vdpa_to_vduse(vdpa);
> > +}
> > +
> > +static struct vduse_dev_msg *vduse_find_msg(struct list_head *head,
> > + uint32_t request_id)
> > +{
> > + struct vduse_dev_msg *msg;
> > +
> > + list_for_each_entry(msg, head, list) {
> > + if (msg->req.request_id == request_id) {
> > + list_del(&msg->list);
> > + return msg;
> > + }
> > + }
> > +
> > + return NULL;
> > +}
> > +
> > +static struct vduse_dev_msg *vduse_dequeue_msg(struct list_head *head)
> > +{
> > + struct vduse_dev_msg *msg = NULL;
> > +
> > + if (!list_empty(head)) {
> > + msg = list_first_entry(head, struct vduse_dev_msg, list);
> > + list_del(&msg->list);
> > + }
> > +
> > + return msg;
> > +}
> > +
> > +static void vduse_enqueue_msg(struct list_head *head,
> > + struct vduse_dev_msg *msg)
> > +{
> > + list_add_tail(&msg->list, head);
> > +}
> > +
> > +static int vduse_dev_msg_sync(struct vduse_dev *dev,
> > + struct vduse_dev_msg *msg)
> > +{
> > + int ret;
> > +
> > + init_waitqueue_head(&msg->waitq);
> > + spin_lock(&dev->msg_lock);
> > + msg->req.request_id = dev->msg_unique++;
> > + vduse_enqueue_msg(&dev->send_list, msg);
> > + wake_up(&dev->waitq);
> > + spin_unlock(&dev->msg_lock);
> > +
> > + wait_event_killable_timeout(msg->waitq, msg->completed,
> > + VDUSE_REQUEST_TIMEOUT * HZ);
> > + spin_lock(&dev->msg_lock);
> > + if (!msg->completed) {
> > + list_del(&msg->list);
> > + msg->resp.result = VDUSE_REQ_RESULT_FAILED;
> > + }
> > + ret = (msg->resp.result == VDUSE_REQ_RESULT_OK) ? 0 : -EIO;
>
>
> I think we should mark the device as malfunction when there is a timeout
> and forbid any userspace operations except for the destroy aftwards for
> safety.
>
Is it necessary? Actually we can't stop the dataplane processing in
userspace. It doesn’t seem to help much if we only forbid ioctl and
read/write. And there might be some messages that can tolerate
failure. Even userspace can do something like NEEDS_RESET to do
recovery.
>
> > + spin_unlock(&dev->msg_lock);
> > +
> > + return ret;
> > +}
> > +
> > +static int vduse_dev_get_vq_state_packed(struct vduse_dev *dev,
> > + struct vduse_virtqueue *vq,
> > + struct vdpa_vq_state_packed *packed)
> > +{
> > + struct vduse_dev_msg msg = { 0 };
> > + int ret;
> > +
> > + msg.req.type = VDUSE_GET_VQ_STATE;
> > + msg.req.vq_state.index = vq->index;
> > +
> > + ret = vduse_dev_msg_sync(dev, &msg);
> > + if (ret)
> > + return ret;
> > +
> > + packed->last_avail_counter =
> > + msg.resp.vq_state.packed.last_avail_counter;
> > + packed->last_avail_idx = msg.resp.vq_state.packed.last_avail_idx;
> > + packed->last_used_counter = msg.resp.vq_state.packed.last_used_counter;
> > + packed->last_used_idx = msg.resp.vq_state.packed.last_used_idx;
> > +
> > + return 0;
> > +}
> > +
> > +static int vduse_dev_get_vq_state_split(struct vduse_dev *dev,
> > + struct vduse_virtqueue *vq,
> > + struct vdpa_vq_state_split *split)
> > +{
> > + struct vduse_dev_msg msg = { 0 };
> > + int ret;
> > +
> > + msg.req.type = VDUSE_GET_VQ_STATE;
> > + msg.req.vq_state.index = vq->index;
> > +
> > + ret = vduse_dev_msg_sync(dev, &msg);
> > + if (ret)
> > + return ret;
> > +
> > + split->avail_index = msg.resp.vq_state.split.avail_index;
> > +
> > + return 0;
> > +}
> > +
> > +static int vduse_dev_set_status(struct vduse_dev *dev, u8 status)
> > +{
> > + struct vduse_dev_msg msg = { 0 };
> > +
> > + msg.req.type = VDUSE_SET_STATUS;
> > + msg.req.s.status = status;
> > +
> > + return vduse_dev_msg_sync(dev, &msg);
> > +}
> > +
> > +static int vduse_dev_update_iotlb(struct vduse_dev *dev,
> > + u64 start, u64 last)
> > +{
> > + struct vduse_dev_msg msg = { 0 };
> > +
> > + if (last < start)
> > + return -EINVAL;
> > +
> > + msg.req.type = VDUSE_UPDATE_IOTLB;
> > + msg.req.iova.start = start;
> > + msg.req.iova.last = last;
> > +
> > + return vduse_dev_msg_sync(dev, &msg);
> > +}
> > +
> > +static ssize_t vduse_dev_read_iter(struct kiocb *iocb, struct iov_iter *to)
> > +{
> > + struct file *file = iocb->ki_filp;
> > + struct vduse_dev *dev = file->private_data;
> > + struct vduse_dev_msg *msg;
> > + int size = sizeof(struct vduse_dev_request);
> > + ssize_t ret;
> > +
> > + if (iov_iter_count(to) < size)
> > + return -EINVAL;
> > +
> > + spin_lock(&dev->msg_lock);
> > + while (1) {
> > + msg = vduse_dequeue_msg(&dev->send_list);
> > + if (msg)
> > + break;
> > +
> > + ret = -EAGAIN;
> > + if (file->f_flags & O_NONBLOCK)
> > + goto unlock;
> > +
> > + spin_unlock(&dev->msg_lock);
> > + ret = wait_event_interruptible_exclusive(dev->waitq,
> > + !list_empty(&dev->send_list));
> > + if (ret)
> > + return ret;
> > +
> > + spin_lock(&dev->msg_lock);
> > + }
> > + spin_unlock(&dev->msg_lock);
> > + ret = copy_to_iter(&msg->req, size, to);
> > + spin_lock(&dev->msg_lock);
> > + if (ret != size) {
> > + ret = -EFAULT;
> > + vduse_enqueue_msg(&dev->send_list, msg);
> > + goto unlock;
> > + }
> > + vduse_enqueue_msg(&dev->recv_list, msg);
> > +unlock:
> > + spin_unlock(&dev->msg_lock);
> > +
> > + return ret;
> > +}
> > +
> > +static ssize_t vduse_dev_write_iter(struct kiocb *iocb, struct iov_iter *from)
> > +{
> > + struct file *file = iocb->ki_filp;
> > + struct vduse_dev *dev = file->private_data;
> > + struct vduse_dev_response resp;
> > + struct vduse_dev_msg *msg;
> > + size_t ret;
> > +
> > + ret = copy_from_iter(&resp, sizeof(resp), from);
> > + if (ret != sizeof(resp))
> > + return -EINVAL;
> > +
> > + spin_lock(&dev->msg_lock);
> > + msg = vduse_find_msg(&dev->recv_list, resp.request_id);
> > + if (!msg) {
> > + ret = -ENOENT;
> > + goto unlock;
> > + }
> > +
> > + memcpy(&msg->resp, &resp, sizeof(resp));
> > + msg->completed = 1;
> > + wake_up(&msg->waitq);
> > +unlock:
> > + spin_unlock(&dev->msg_lock);
> > +
> > + return ret;
> > +}
> > +
> > +static __poll_t vduse_dev_poll(struct file *file, poll_table *wait)
> > +{
> > + struct vduse_dev *dev = file->private_data;
> > + __poll_t mask = 0;
> > +
> > + poll_wait(file, &dev->waitq, wait);
> > +
> > + if (!list_empty(&dev->send_list))
> > + mask |= EPOLLIN | EPOLLRDNORM;
> > + if (!list_empty(&dev->recv_list))
> > + mask |= EPOLLOUT | EPOLLWRNORM;
> > +
> > + return mask;
> > +}
> > +
> > +static void vduse_dev_reset(struct vduse_dev *dev)
> > +{
> > + int i;
> > + struct vduse_iova_domain *domain = dev->domain;
> > +
> > + /* The coherent mappings are handled in vduse_dev_free_coherent() */
> > + if (domain->bounce_map)
> > + vduse_domain_reset_bounce_map(domain);
> > +
> > + dev->driver_features = 0;
> > + dev->generation++;
> > + spin_lock(&dev->irq_lock);
> > + dev->config_cb.callback = NULL;
> > + dev->config_cb.private = NULL;
> > + spin_unlock(&dev->irq_lock);
> > + flush_work(&dev->inject);
> > +
> > + for (i = 0; i < dev->vq_num; i++) {
> > + struct vduse_virtqueue *vq = &dev->vqs[i];
> > +
> > + vq->ready = false;
> > + vq->desc_addr = 0;
> > + vq->driver_addr = 0;
> > + vq->device_addr = 0;
> > + vq->num = 0;
> > + memset(&vq->state, 0, sizeof(vq->state));
> > +
> > + spin_lock(&vq->kick_lock);
> > + vq->kicked = false;
> > + if (vq->kickfd)
> > + eventfd_ctx_put(vq->kickfd);
> > + vq->kickfd = NULL;
> > + spin_unlock(&vq->kick_lock);
> > +
> > + spin_lock(&vq->irq_lock);
> > + vq->cb.callback = NULL;
> > + vq->cb.private = NULL;
> > + spin_unlock(&vq->irq_lock);
> > + flush_work(&vq->inject);
> > + }
> > +}
> > +
> > +static int vduse_vdpa_set_vq_address(struct vdpa_device *vdpa, u16 idx,
> > + u64 desc_area, u64 driver_area,
> > + u64 device_area)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + vq->desc_addr = desc_area;
> > + vq->driver_addr = driver_area;
> > + vq->device_addr = device_area;
> > +
> > + return 0;
> > +}
> > +
> > +static void vduse_vdpa_kick_vq(struct vdpa_device *vdpa, u16 idx)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + spin_lock(&vq->kick_lock);
> > + if (!vq->ready)
> > + goto unlock;
> > +
> > + if (vq->kickfd)
> > + eventfd_signal(vq->kickfd, 1);
> > + else
> > + vq->kicked = true;
> > +unlock:
> > + spin_unlock(&vq->kick_lock);
> > +}
> > +
> > +static void vduse_vdpa_set_vq_cb(struct vdpa_device *vdpa, u16 idx,
> > + struct vdpa_callback *cb)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + spin_lock(&vq->irq_lock);
> > + vq->cb.callback = cb->callback;
> > + vq->cb.private = cb->private;
> > + spin_unlock(&vq->irq_lock);
> > +}
> > +
> > +static void vduse_vdpa_set_vq_num(struct vdpa_device *vdpa, u16 idx, u32 num)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + vq->num = num;
> > +}
> > +
> > +static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
> > + u16 idx, bool ready)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + vq->ready = ready;
> > +}
> > +
> > +static bool vduse_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 idx)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + return vq->ready;
> > +}
> > +
> > +static int vduse_vdpa_set_vq_state(struct vdpa_device *vdpa, u16 idx,
> > + const struct vdpa_vq_state *state)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + if (dev->driver_features & BIT_ULL(VIRTIO_F_RING_PACKED)) {
> > + vq->state.packed.last_avail_counter =
> > + state->packed.last_avail_counter;
> > + vq->state.packed.last_avail_idx = state->packed.last_avail_idx;
> > + vq->state.packed.last_used_counter =
> > + state->packed.last_used_counter;
> > + vq->state.packed.last_used_idx = state->packed.last_used_idx;
> > + } else
> > + vq->state.split.avail_index = state->split.avail_index;
> > +
> > + return 0;
> > +}
> > +
> > +static int vduse_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 idx,
> > + struct vdpa_vq_state *state)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + struct vduse_virtqueue *vq = &dev->vqs[idx];
> > +
> > + if (dev->driver_features & BIT_ULL(VIRTIO_F_RING_PACKED))
> > + return vduse_dev_get_vq_state_packed(dev, vq, &state->packed);
> > +
> > + return vduse_dev_get_vq_state_split(dev, vq, &state->split);
> > +}
> > +
> > +static u32 vduse_vdpa_get_vq_align(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->vq_align;
> > +}
> > +
> > +static u64 vduse_vdpa_get_features(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->device_features;
> > +}
> > +
> > +static int vduse_vdpa_set_features(struct vdpa_device *vdpa, u64 features)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + dev->driver_features = features;
> > + return 0;
> > +}
> > +
> > +static void vduse_vdpa_set_config_cb(struct vdpa_device *vdpa,
> > + struct vdpa_callback *cb)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + spin_lock(&dev->irq_lock);
> > + dev->config_cb.callback = cb->callback;
> > + dev->config_cb.private = cb->private;
> > + spin_unlock(&dev->irq_lock);
> > +}
> > +
> > +static u16 vduse_vdpa_get_vq_num_max(struct vdpa_device *vdpa, u16 idx)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->vqs[idx].num_max;
> > +}
> > +
> > +static u32 vduse_vdpa_get_device_id(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->device_id;
> > +}
> > +
> > +static u32 vduse_vdpa_get_vendor_id(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->vendor_id;
> > +}
> > +
> > +static u8 vduse_vdpa_get_status(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->status;
> > +}
> > +
> > +static void vduse_vdpa_set_status(struct vdpa_device *vdpa, u8 status)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + if (vduse_dev_set_status(dev, status))
> > + return;
> > +
> > + dev->status = status;
>
>
> It looks to me such design exclude the possibility of letting userspace
> device to set bit like (NEEDS_RESET) in the future.
>
Looks like we can achieve that via the ioctl.
> I wonder if it's better to do something similar to ccw:
>
> 1) requires the userspace to update the status bit in the response
> 2) update the dev->status to the status in the response if no timeout
>
> Then userspace could then set NEEDS_RESET if necessary.
>
But NEEDS_RESET does not only happen in this case.
>
> > + if (status == 0)
> > + vduse_dev_reset(dev);
> > +}
> > +
> > +static size_t vduse_vdpa_get_config_size(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->config_size;
> > +}
> > +
> > +static void vduse_vdpa_get_config(struct vdpa_device *vdpa, unsigned int offset,
> > + void *buf, unsigned int len)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + if (len > dev->config_size - offset)
> > + return;
> > +
> > + memcpy(buf, dev->config + offset, len);
> > +}
> > +
> > +static void vduse_vdpa_set_config(struct vdpa_device *vdpa, unsigned int offset,
> > + const void *buf, unsigned int len)
> > +{
> > + /* Now we only support read-only configuration space */
> > +}
> > +
> > +static u32 vduse_vdpa_get_generation(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + return dev->generation;
> > +}
> > +
> > +static int vduse_vdpa_set_map(struct vdpa_device *vdpa,
> > + struct vhost_iotlb *iotlb)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > + int ret;
> > +
> > + ret = vduse_domain_set_map(dev->domain, iotlb);
> > + if (ret)
> > + return ret;
> > +
> > + ret = vduse_dev_update_iotlb(dev, 0ULL, ULLONG_MAX);
> > + if (ret) {
> > + vduse_domain_clear_map(dev->domain, iotlb);
> > + return ret;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static void vduse_vdpa_free(struct vdpa_device *vdpa)
> > +{
> > + struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > +
> > + dev->vdev = NULL;
> > +}
> > +
> > +static const struct vdpa_config_ops vduse_vdpa_config_ops = {
> > + .set_vq_address = vduse_vdpa_set_vq_address,
> > + .kick_vq = vduse_vdpa_kick_vq,
> > + .set_vq_cb = vduse_vdpa_set_vq_cb,
> > + .set_vq_num = vduse_vdpa_set_vq_num,
> > + .set_vq_ready = vduse_vdpa_set_vq_ready,
> > + .get_vq_ready = vduse_vdpa_get_vq_ready,
> > + .set_vq_state = vduse_vdpa_set_vq_state,
> > + .get_vq_state = vduse_vdpa_get_vq_state,
> > + .get_vq_align = vduse_vdpa_get_vq_align,
> > + .get_features = vduse_vdpa_get_features,
> > + .set_features = vduse_vdpa_set_features,
> > + .set_config_cb = vduse_vdpa_set_config_cb,
> > + .get_vq_num_max = vduse_vdpa_get_vq_num_max,
> > + .get_device_id = vduse_vdpa_get_device_id,
> > + .get_vendor_id = vduse_vdpa_get_vendor_id,
> > + .get_status = vduse_vdpa_get_status,
> > + .set_status = vduse_vdpa_set_status,
> > + .get_config_size = vduse_vdpa_get_config_size,
> > + .get_config = vduse_vdpa_get_config,
> > + .set_config = vduse_vdpa_set_config,
> > + .get_generation = vduse_vdpa_get_generation,
> > + .set_map = vduse_vdpa_set_map,
> > + .free = vduse_vdpa_free,
> > +};
> > +
> > +static dma_addr_t vduse_dev_map_page(struct device *dev, struct page *page,
> > + unsigned long offset, size_t size,
> > + enum dma_data_direction dir,
> > + unsigned long attrs)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > +
> > + return vduse_domain_map_page(domain, page, offset, size, dir, attrs);
> > +}
> > +
> > +static void vduse_dev_unmap_page(struct device *dev, dma_addr_t dma_addr,
> > + size_t size, enum dma_data_direction dir,
> > + unsigned long attrs)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > +
> > + return vduse_domain_unmap_page(domain, dma_addr, size, dir, attrs);
> > +}
> > +
> > +static void *vduse_dev_alloc_coherent(struct device *dev, size_t size,
> > + dma_addr_t *dma_addr, gfp_t flag,
> > + unsigned long attrs)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > + unsigned long iova;
> > + void *addr;
> > +
> > + *dma_addr = DMA_MAPPING_ERROR;
> > + addr = vduse_domain_alloc_coherent(domain, size,
> > + (dma_addr_t *)&iova, flag, attrs);
> > + if (!addr)
> > + return NULL;
> > +
> > + *dma_addr = (dma_addr_t)iova;
> > +
> > + return addr;
> > +}
> > +
> > +static void vduse_dev_free_coherent(struct device *dev, size_t size,
> > + void *vaddr, dma_addr_t dma_addr,
> > + unsigned long attrs)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > +
> > + vduse_domain_free_coherent(domain, size, vaddr, dma_addr, attrs);
> > +}
> > +
> > +static size_t vduse_dev_max_mapping_size(struct device *dev)
> > +{
> > + struct vduse_dev *vdev = dev_to_vduse(dev);
> > + struct vduse_iova_domain *domain = vdev->domain;
> > +
> > + return domain->bounce_size;
> > +}
> > +
> > +static const struct dma_map_ops vduse_dev_dma_ops = {
> > + .map_page = vduse_dev_map_page,
> > + .unmap_page = vduse_dev_unmap_page,
> > + .alloc = vduse_dev_alloc_coherent,
> > + .free = vduse_dev_free_coherent,
> > + .max_mapping_size = vduse_dev_max_mapping_size,
> > +};
> > +
> > +static unsigned int perm_to_file_flags(u8 perm)
> > +{
> > + unsigned int flags = 0;
> > +
> > + switch (perm) {
> > + case VDUSE_ACCESS_WO:
> > + flags |= O_WRONLY;
> > + break;
> > + case VDUSE_ACCESS_RO:
> > + flags |= O_RDONLY;
> > + break;
> > + case VDUSE_ACCESS_RW:
> > + flags |= O_RDWR;
> > + break;
> > + default:
> > + WARN(1, "invalidate vhost IOTLB permission\n");
> > + break;
> > + }
> > +
> > + return flags;
> > +}
> > +
> > +static int vduse_kickfd_setup(struct vduse_dev *dev,
> > + struct vduse_vq_eventfd *eventfd)
> > +{
> > + struct eventfd_ctx *ctx = NULL;
> > + struct vduse_virtqueue *vq;
> > + u32 index;
> > +
> > + if (eventfd->index >= dev->vq_num)
> > + return -EINVAL;
> > +
> > + index = array_index_nospec(eventfd->index, dev->vq_num);
> > + vq = &dev->vqs[index];
> > + if (eventfd->fd >= 0) {
> > + ctx = eventfd_ctx_fdget(eventfd->fd);
> > + if (IS_ERR(ctx))
> > + return PTR_ERR(ctx);
> > + } else if (eventfd->fd != VDUSE_EVENTFD_DEASSIGN)
> > + return 0;
> > +
> > + spin_lock(&vq->kick_lock);
> > + if (vq->kickfd)
> > + eventfd_ctx_put(vq->kickfd);
> > + vq->kickfd = ctx;
> > + if (vq->ready && vq->kicked && vq->kickfd) {
> > + eventfd_signal(vq->kickfd, 1);
> > + vq->kicked = false;
> > + }
> > + spin_unlock(&vq->kick_lock);
> > +
> > + return 0;
> > +}
> > +
> > +static bool vduse_dev_is_ready(struct vduse_dev *dev)
> > +{
> > + int i;
> > +
> > + for (i = 0; i < dev->vq_num; i++)
> > + if (!dev->vqs[i].num_max)
> > + return false;
> > +
> > + return true;
> > +}
> > +
> > +static void vduse_dev_irq_inject(struct work_struct *work)
> > +{
> > + struct vduse_dev *dev = container_of(work, struct vduse_dev, inject);
> > +
> > + spin_lock_irq(&dev->irq_lock);
> > + if (dev->config_cb.callback)
> > + dev->config_cb.callback(dev->config_cb.private);
> > + spin_unlock_irq(&dev->irq_lock);
> > +}
> > +
> > +static void vduse_vq_irq_inject(struct work_struct *work)
> > +{
> > + struct vduse_virtqueue *vq = container_of(work,
> > + struct vduse_virtqueue, inject);
> > +
> > + spin_lock_irq(&vq->irq_lock);
> > + if (vq->ready && vq->cb.callback)
> > + vq->cb.callback(vq->cb.private);
> > + spin_unlock_irq(&vq->irq_lock);
> > +}
> > +
> > +static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
> > + unsigned long arg)
> > +{
> > + struct vduse_dev *dev = file->private_data;
> > + void __user *argp = (void __user *)arg;
> > + int ret;
> > +
> > + switch (cmd) {
> > + case VDUSE_IOTLB_GET_FD: {
> > + struct vduse_iotlb_entry entry;
> > + struct vhost_iotlb_map *map;
> > + struct vdpa_map_file *map_file;
> > + struct vduse_iova_domain *domain = dev->domain;
> > + struct file *f = NULL;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&entry, argp, sizeof(entry)))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (entry.start > entry.last)
> > + break;
> > +
> > + spin_lock(&domain->iotlb_lock);
> > + map = vhost_iotlb_itree_first(domain->iotlb,
> > + entry.start, entry.last);
> > + if (map) {
> > + map_file = (struct vdpa_map_file *)map->opaque;
> > + f = get_file(map_file->file);
> > + entry.offset = map_file->offset;
> > + entry.start = map->start;
> > + entry.last = map->last;
> > + entry.perm = map->perm;
> > + }
> > + spin_unlock(&domain->iotlb_lock);
> > + ret = -EINVAL;
> > + if (!f)
> > + break;
> > +
> > + ret = -EFAULT;
> > + if (copy_to_user(argp, &entry, sizeof(entry))) {
> > + fput(f);
> > + break;
> > + }
> > + ret = receive_fd(f, perm_to_file_flags(entry.perm));
> > + fput(f);
> > + break;
> > + }
> > + case VDUSE_DEV_GET_FEATURES:
>
>
> Let's add a comment to explain here. E.g we just mirror what driver
> wrote and the drier is expected to check FEATURE_OK.
>
I already document some details in include/uapi/linux/vduse.h and
Documentation/userspace-api/vduse.rst
>
> > + ret = put_user(dev->driver_features, (u64 __user *)argp);
> > + break;
> > + case VDUSE_DEV_SET_CONFIG: {
> > + struct vduse_config_data config;
> > + unsigned long size = offsetof(struct vduse_config_data,
> > + buffer);
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&config, argp, size))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (config.length == 0 ||
> > + config.length > dev->config_size - config.offset)
> > + break;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(dev->config + config.offset, argp + size,
> > + config.length))
> > + break;
> > +
> > + ret = 0;
> > + break;
> > + }
> > + case VDUSE_DEV_INJECT_IRQ:
>
>
> It's better to have "config" in the name.
>
OK.
>
> > + ret = 0;
> > + queue_work(vduse_irq_wq, &dev->inject);
> > + break;
> > + case VDUSE_VQ_SETUP: {
> > + struct vduse_vq_config config;
> > + u32 index;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&config, argp, sizeof(config)))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (config.index >= dev->vq_num)
> > + break;
> > +
> > + index = array_index_nospec(config.index, dev->vq_num);
> > + dev->vqs[index].num_max = config.max_size;
> > + ret = 0;
> > + break;
> > + }
> > + case VDUSE_VQ_GET_INFO: {
> > + struct vduse_vq_info vq_info;
> > + struct vduse_virtqueue *vq;
> > + u32 index;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&vq_info, argp, sizeof(vq_info)))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (vq_info.index >= dev->vq_num)
> > + break;
> > +
> > + index = array_index_nospec(vq_info.index, dev->vq_num);
> > + vq = &dev->vqs[index];
> > + vq_info.desc_addr = vq->desc_addr;
> > + vq_info.driver_addr = vq->driver_addr;
> > + vq_info.device_addr = vq->device_addr;
> > + vq_info.num = vq->num;
> > +
> > + if (dev->driver_features & BIT_ULL(VIRTIO_F_RING_PACKED)) {
> > + vq_info.packed.last_avail_counter =
> > + vq->state.packed.last_avail_counter;
> > + vq_info.packed.last_avail_idx =
> > + vq->state.packed.last_avail_idx;
> > + vq_info.packed.last_used_counter =
> > + vq->state.packed.last_used_counter;
> > + vq_info.packed.last_used_idx =
> > + vq->state.packed.last_used_idx;
> > + } else
> > + vq_info.split.avail_index =
> > + vq->state.split.avail_index;
> > +
> > + vq_info.ready = vq->ready;
> > +
> > + ret = -EFAULT;
> > + if (copy_to_user(argp, &vq_info, sizeof(vq_info)))
> > + break;
> > +
> > + ret = 0;
> > + break;
> > + }
> > + case VDUSE_VQ_SETUP_KICKFD: {
> > + struct vduse_vq_eventfd eventfd;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(&eventfd, argp, sizeof(eventfd)))
> > + break;
> > +
> > + ret = vduse_kickfd_setup(dev, &eventfd);
> > + break;
> > + }
> > + case VDUSE_VQ_INJECT_IRQ: {
> > + u32 index;
> > +
> > + ret = -EFAULT;
> > + if (get_user(index, (u32 __user *)argp))
> > + break;
> > +
> > + ret = -EINVAL;
> > + if (index >= dev->vq_num)
> > + break;
> > +
> > + ret = 0;
> > + index = array_index_nospec(index, dev->vq_num);
> > + queue_work(vduse_irq_wq, &dev->vqs[index].inject);
> > + break;
> > + }
> > + default:
> > + ret = -ENOIOCTLCMD;
> > + break;
> > + }
> > +
> > + return ret;
> > +}
> > +
> > +static int vduse_dev_release(struct inode *inode, struct file *file)
> > +{
> > + struct vduse_dev *dev = file->private_data;
> > +
> > + spin_lock(&dev->msg_lock);
> > + /* Make sure the inflight messages can processed after reconncection */
> > + list_splice_init(&dev->recv_list, &dev->send_list);
> > + spin_unlock(&dev->msg_lock);
> > + dev->connected = false;
> > +
> > + return 0;
> > +}
> > +
> > +static struct vduse_dev *vduse_dev_get_from_minor(int minor)
> > +{
> > + struct vduse_dev *dev;
> > +
> > + mutex_lock(&vduse_lock);
> > + dev = idr_find(&vduse_idr, minor);
> > + mutex_unlock(&vduse_lock);
> > +
> > + return dev;
> > +}
> > +
> > +static int vduse_dev_open(struct inode *inode, struct file *file)
> > +{
> > + int ret;
> > + struct vduse_dev *dev = vduse_dev_get_from_minor(iminor(inode));
> > +
> > + if (!dev)
> > + return -ENODEV;
> > +
> > + ret = -EBUSY;
> > + mutex_lock(&dev->lock);
> > + if (dev->connected)
> > + goto unlock;
> > +
> > + ret = 0;
> > + dev->connected = true;
> > + file->private_data = dev;
> > +unlock:
> > + mutex_unlock(&dev->lock);
> > +
> > + return ret;
> > +}
> > +
> > +static const struct file_operations vduse_dev_fops = {
> > + .owner = THIS_MODULE,
> > + .open = vduse_dev_open,
> > + .release = vduse_dev_release,
> > + .read_iter = vduse_dev_read_iter,
> > + .write_iter = vduse_dev_write_iter,
> > + .poll = vduse_dev_poll,
> > + .unlocked_ioctl = vduse_dev_ioctl,
> > + .compat_ioctl = compat_ptr_ioctl,
> > + .llseek = noop_llseek,
> > +};
> > +
> > +static struct vduse_dev *vduse_dev_create(void)
> > +{
> > + struct vduse_dev *dev = kzalloc(sizeof(*dev), GFP_KERNEL);
> > +
> > + if (!dev)
> > + return NULL;
> > +
> > + mutex_init(&dev->lock);
> > + spin_lock_init(&dev->msg_lock);
> > + INIT_LIST_HEAD(&dev->send_list);
> > + INIT_LIST_HEAD(&dev->recv_list);
> > + spin_lock_init(&dev->irq_lock);
> > +
> > + INIT_WORK(&dev->inject, vduse_dev_irq_inject);
> > + init_waitqueue_head(&dev->waitq);
> > +
> > + return dev;
> > +}
> > +
> > +static void vduse_dev_destroy(struct vduse_dev *dev)
> > +{
> > + kfree(dev);
> > +}
> > +
> > +static struct vduse_dev *vduse_find_dev(const char *name)
> > +{
> > + struct vduse_dev *dev;
> > + int id;
> > +
> > + idr_for_each_entry(&vduse_idr, dev, id)
> > + if (!strcmp(dev->name, name))
> > + return dev;
> > +
> > + return NULL;
> > +}
> > +
> > +static int vduse_destroy_dev(char *name)
> > +{
> > + struct vduse_dev *dev = vduse_find_dev(name);
> > +
> > + if (!dev)
> > + return -EINVAL;
> > +
> > + mutex_lock(&dev->lock);
> > + if (dev->vdev || dev->connected) {
> > + mutex_unlock(&dev->lock);
> > + return -EBUSY;
> > + }
> > + dev->connected = true;
> > + mutex_unlock(&dev->lock);
> > +
> > + vduse_dev_reset(dev);
> > + device_destroy(vduse_class, MKDEV(MAJOR(vduse_major), dev->minor));
> > + idr_remove(&vduse_idr, dev->minor);
> > + kvfree(dev->config);
> > + kfree(dev->vqs);
> > + vduse_domain_destroy(dev->domain);
> > + kfree(dev->name);
> > + vduse_dev_destroy(dev);
> > + module_put(THIS_MODULE);
> > +
> > + return 0;
> > +}
> > +
> > +static bool device_is_allowed(u32 device_id)
> > +{
> > + int i;
> > +
> > + for (i = 0; i < ARRAY_SIZE(allowed_device_id); i++)
> > + if (allowed_device_id[i] == device_id)
> > + return true;
> > +
> > + return false;
> > +}
> > +
> > +static bool features_is_valid(u64 features)
> > +{
> > + if (!(features & (1ULL << VIRTIO_F_ACCESS_PLATFORM)))
> > + return false;
> > +
> > + /* Now we only support read-only configuration space */
> > + if (features & (1ULL << VIRTIO_BLK_F_CONFIG_WCE))
> > + return false;
> > +
> > + return true;
> > +}
> > +
> > +static bool vduse_validate_config(struct vduse_dev_config *config)
> > +{
> > + if (config->bounce_size > VDUSE_MAX_BOUNCE_SIZE)
> > + return false;
> > +
> > + if (config->vq_align > PAGE_SIZE)
> > + return false;
> > +
> > + if (config->config_size > PAGE_SIZE)
> > + return false;
>
>
> Any reason for this check?
>
To limit the kernel buffer size for configuration space. Is the
PAGE_SIZE enough?
Thanks,
Yongji
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
next prev parent reply other threads:[~2021-07-14 6:49 UTC|newest]
Thread overview: 108+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-13 8:46 [PATCH v9 00/17] Introduce VDUSE - vDPA Device in Userspace Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 01/17] iova: Export alloc_iova_fast() and free_iova_fast() Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 02/17] file: Export receive_fd() to modules Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 03/17] vdpa: Fix code indentation Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-14 4:20 ` Joe Perches
2021-07-14 4:20 ` Joe Perches
2021-07-14 4:20 ` Joe Perches
2021-07-14 5:48 ` Yongji Xie
2021-07-14 5:48 ` Yongji Xie
2021-07-13 8:46 ` [PATCH v9 04/17] vdpa: Fail the vdpa_reset() if fail to set device status to zero Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 05/17] vhost-vdpa: Fail the vhost_vdpa_set_status() on reset failure Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 06/17] vhost-vdpa: Handle the failure of vdpa_reset() Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 07/17] virtio: Don't set FAILED status bit on device index allocation failure Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 11:02 ` Dan Carpenter
2021-07-13 11:02 ` Dan Carpenter
2021-07-13 11:02 ` Dan Carpenter
2021-07-13 11:25 ` Yongji Xie
2021-07-13 11:25 ` Yongji Xie
2021-07-13 8:46 ` [PATCH v9 08/17] virtio_config: Add a return value to reset function Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-14 10:21 ` kernel test robot
2021-07-15 20:37 ` kernel test robot
2021-07-13 8:46 ` [PATCH v9 09/17] virtio-vdpa: Handle the failure of vdpa_reset() Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 10/17] virtio: Handle device reset failure in register_virtio_device() Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 11/17] vhost-iotlb: Add an opaque pointer for vhost IOTLB Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 12/17] vdpa: Add an opaque pointer for vdpa_config_ops.dma_map() Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 13/17] vdpa: factor out vhost_vdpa_pa_map() and vhost_vdpa_pa_unmap() Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 11:31 ` Dan Carpenter
2021-07-13 11:31 ` Dan Carpenter
2021-07-13 11:31 ` Dan Carpenter
2021-07-14 2:14 ` Jason Wang
2021-07-14 2:14 ` Jason Wang
2021-07-14 2:14 ` Jason Wang
2021-07-14 8:05 ` Dan Carpenter
2021-07-14 8:05 ` Dan Carpenter
2021-07-14 8:05 ` Dan Carpenter
2021-07-14 9:41 ` Jason Wang
2021-07-14 9:41 ` Jason Wang
2021-07-14 9:41 ` Jason Wang
2021-07-14 9:57 ` Dan Carpenter
2021-07-14 9:57 ` Dan Carpenter
2021-07-14 9:57 ` Dan Carpenter
2021-07-15 2:20 ` Jason Wang
2021-07-15 2:20 ` Jason Wang
2021-07-15 2:20 ` Jason Wang
2021-07-14 5:24 ` Yongji Xie
2021-07-14 5:24 ` Yongji Xie
2021-07-13 8:46 ` [PATCH v9 14/17] vdpa: Support transferring virtual addressing during DMA mapping Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 15/17] vduse: Implement an MMU-based IOMMU driver Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 8:46 ` [PATCH v9 16/17] vduse: Introduce VDUSE - vDPA Device in Userspace Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-13 13:27 ` Dan Carpenter
2021-07-13 13:27 ` Dan Carpenter
2021-07-13 13:27 ` Dan Carpenter
2021-07-14 2:54 ` Jason Wang
2021-07-14 2:54 ` Jason Wang
2021-07-14 2:54 ` Jason Wang
2021-07-14 5:45 ` Yongji Xie
2021-07-14 5:45 ` Yongji Xie
2021-07-14 5:45 ` Jason Wang
2021-07-14 5:45 ` Jason Wang
2021-07-14 5:45 ` Jason Wang
2021-07-14 5:54 ` Michael S. Tsirkin
2021-07-14 5:54 ` Michael S. Tsirkin
2021-07-14 5:54 ` Michael S. Tsirkin
2021-07-14 6:02 ` Jason Wang
2021-07-14 6:02 ` Jason Wang
2021-07-14 6:02 ` Jason Wang
2021-07-14 6:47 ` Greg KH
2021-07-14 6:47 ` Greg KH
2021-07-14 6:47 ` Greg KH
2021-07-14 8:56 ` Jason Wang
2021-07-14 8:56 ` Jason Wang
2021-07-14 8:56 ` Jason Wang
2021-07-14 6:49 ` Yongji Xie [this message]
2021-07-14 6:49 ` Yongji Xie
2021-07-14 9:12 ` Jason Wang
2021-07-14 9:12 ` Jason Wang
2021-07-14 9:12 ` Jason Wang
2021-07-15 4:03 ` Yongji Xie
2021-07-15 4:03 ` Yongji Xie
2021-07-15 5:00 ` Jason Wang
2021-07-15 5:00 ` Jason Wang
2021-07-15 5:00 ` Jason Wang
2021-07-13 8:46 ` [PATCH v9 17/17] Documentation: Add documentation for VDUSE Xie Yongji
2021-07-13 8:46 ` Xie Yongji
2021-07-15 5:18 ` Jason Wang
2021-07-15 5:18 ` Jason Wang
2021-07-15 5:18 ` Jason Wang
2021-07-15 7:27 ` Yongji Xie
2021-07-15 7:27 ` Yongji Xie
2021-12-15 10:10 ` [PATCH v9 00/17] Introduce VDUSE - vDPA Device in Userspace Liuxiangdong
2021-12-16 3:14 ` Yongji Xie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CACycT3uh+wUeDM1H7JiCJTMeCVCBngURGKeXD-h+meekNNwiQw@mail.gmail.com \
--to=xieyongji@bytedance.com \
--cc=axboe@kernel.dk \
--cc=bcrl@kvack.org \
--cc=christian.brauner@canonical.com \
--cc=corbet@lwn.net \
--cc=dan.carpenter@oracle.com \
--cc=gregkh@linuxfoundation.org \
--cc=hch@infradead.org \
--cc=iommu@lists.linux-foundation.org \
--cc=jasowang@redhat.com \
--cc=joro@8bytes.org \
--cc=kvm@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mika.penttila@nextfour.com \
--cc=mst@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=parav@nvidia.com \
--cc=rdunlap@infradead.org \
--cc=sgarzare@redhat.com \
--cc=songmuchun@bytedance.com \
--cc=stefanha@redhat.com \
--cc=viro@zeniv.linux.org.uk \
--cc=virtualization@lists.linux-foundation.org \
--cc=willy@infradead.org \
--cc=xiaodong.liu@intel.com \
--cc=zhe.he@windriver.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.