From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1E67C433E0 for ; Wed, 23 Dec 2020 12:14:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 44FCB22472 for ; Wed, 23 Dec 2020 12:14:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 44FCB22472 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F0678D0001; Wed, 23 Dec 2020 07:14:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 69FF38D0026; Wed, 23 Dec 2020 07:14:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B7888D0001; Wed, 23 Dec 2020 07:14:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id 429DD8D0028 for ; Wed, 23 Dec 2020 07:14:18 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 006F6180AD820 for ; Wed, 23 Dec 2020 12:14:17 +0000 (UTC) X-FDA: 77624439396.11.duck72_171632627468 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id D020C180F8B81 for ; Wed, 23 Dec 2020 12:14:17 +0000 (UTC) X-HE-Tag: duck72_171632627468 X-Filterd-Recvd-Size: 11761 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Wed, 23 Dec 2020 12:14:17 +0000 (UTC) Received: by mail-ej1-f49.google.com with SMTP id g20so22584487ejb.1 for ; Wed, 23 Dec 2020 04:14:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=35ZYuDz/NHbYVvs0C4L+62CLzf5DN21GLodcbsEQFC8=; b=aJinXqPssChyo572Y/V2zRy6J5OkTR1TQN+MEuQ4oNarcAhB2somhmUhWV0HmgGikG e+TIqXxNraeIrU+WeupO3HTQPFteA8XEkMdqkUmBSeAWWQs239UH4vagR6/13LiAJHu8 yfWpPCNf4skU5igECuQuVO1SHWwpluiHTqNlURKhYiKtdGrksxeXzX0NDUalwkVuAbQA JYQ12n2V0vxQ1VS2QymReXwjFDLc0DV3Lf73j6tylIA9XUeIy1YBOKqikAtUUQm99Hoa /NV96IPBMICQtMBOSsG2cV1Ym4ZQLD07TZef94tSuVP7IBfcKCUbc+9mwgP2H9hgqYLx Ly5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=35ZYuDz/NHbYVvs0C4L+62CLzf5DN21GLodcbsEQFC8=; b=Zj8xWRjbMrvQUOS4E/i2uP/jbmXbexgX8pTRE0VWrcNLYkcVmScpZVhlSoO38ORCyL ZKsJMZ02CXmYuaKFoBSgnQDNubRuOFOZqywbCJyShBRVyb7k8pNJoayIN3HH3VKPmKif k0vWYafNB2Tn4h3tjdx5IIZEVd07MyRZYkA4Ylqzgyyf7LVMCDaNWPj8Jn6hmozlmT9D V9AAmrs7N+v3mgSLJvl4B8lckBVhTNvy9XF8Qe5QgvdGUxP7e60YB4kp9TyEEGuQXBkX FEZE/rnHI2mYHkkvoyvgQRC7jL+FOBws95H9kHnwiNJtx6pl6rt1zwChK9ALIpDSsCYa rI0w== X-Gm-Message-State: AOAM530DyUTFQ55ZfC22q8GHWladm9gjDeGbFoCL2235xhDQKuV95mEI mAQTKjfZ+Hh1iYcVFxhJKJKg3Wjix4xj6Ds8IgZj X-Google-Smtp-Source: ABdhPJxvFGv55SXnvmstsRrPts85FfsZeW8DhW7IZ56OFMBLubAQ02qrxBnCF2WTw9XipJ1ju0Clb5hATd78E6jruXA= X-Received: by 2002:a17:906:94c5:: with SMTP id d5mr23137335ejy.427.1608725655618; Wed, 23 Dec 2020 04:14:15 -0800 (PST) MIME-Version: 1.0 References: <20201222145221.711-1-xieyongji@bytedance.com> <20201222145221.711-10-xieyongji@bytedance.com> <6818a214-d587-4f0b-7de6-13c4e7e94ab6@redhat.com> In-Reply-To: <6818a214-d587-4f0b-7de6-13c4e7e94ab6@redhat.com> From: Yongji Xie Date: Wed, 23 Dec 2020 20:14:04 +0800 Message-ID: Subject: Re: [External] Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message To: Jason Wang Cc: "Michael S. Tsirkin" , Stefan Hajnoczi , sgarzare@redhat.com, Parav Pandit , akpm@linux-foundation.org, Randy Dunlap , Matthew Wilcox , viro@zeniv.linux.org.uk, axboe@kernel.dk, bcrl@kvack.org, corbet@lwn.net, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, kvm@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Dec 23, 2020 at 5:05 PM Jason Wang wrote: > > > On 2020/12/22 =E4=B8=8B=E5=8D=8810:52, Xie Yongji wrote: > > To support vhost-vdpa bus driver, we need a way to share the > > vhost-vdpa backend process's memory with the userspace VDUSE process. > > > > This patch tries to make use of the vhost iotlb message to achieve > > that. We will get the shm file from the iotlb message and pass it > > to the userspace VDUSE process. > > > > Signed-off-by: Xie Yongji > > --- > > Documentation/driver-api/vduse.rst | 15 +++- > > drivers/vdpa/vdpa_user/vduse_dev.c | 147 ++++++++++++++++++++++++++++= ++++++++- > > include/uapi/linux/vduse.h | 11 +++ > > 3 files changed, 171 insertions(+), 2 deletions(-) > > > > diff --git a/Documentation/driver-api/vduse.rst b/Documentation/driver-= api/vduse.rst > > index 623f7b040ccf..48e4b1ba353f 100644 > > --- a/Documentation/driver-api/vduse.rst > > +++ b/Documentation/driver-api/vduse.rst > > @@ -46,13 +46,26 @@ The following types of messages are provided by the= VDUSE framework now: > > > > - VDUSE_GET_CONFIG: Read from device specific configuration space > > > > +- VDUSE_UPDATE_IOTLB: Update the memory mapping in device IOTLB > > + > > +- VDUSE_INVALIDATE_IOTLB: Invalidate the memory mapping in device IOTL= B > > + > > Please see include/linux/vdpa.h for details. > > > > -In the data path, VDUSE framework implements a MMU-based on-chip IOMMU > > +The data path of userspace vDPA device is implemented in different way= s > > +depending on the vdpa bus to which it is attached. > > + > > +In virtio-vdpa case, VDUSE framework implements a MMU-based on-chip IO= MMU > > driver which supports mapping the kernel dma buffer to a userspace io= va > > region dynamically. The userspace iova region can be created by passi= ng > > the userspace vDPA device fd to mmap(2). > > > > +In vhost-vdpa case, the dma buffer is reside in a userspace memory reg= ion > > +which will be shared to the VDUSE userspace processs via the file > > +descriptor in VDUSE_UPDATE_IOTLB message. And the corresponding addres= s > > +mapping (IOVA of dma buffer <-> VA of the memory region) is also inclu= ded > > +in this message. > > + > > Besides, the eventfd mechanism is used to trigger interrupt callbacks= and > > receive virtqueue kicks in userspace. The following ioctls on the use= rspace > > vDPA device fd are provided to support that: > > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_use= r/vduse_dev.c > > index b974333ed4e9..d24aaacb6008 100644 > > --- a/drivers/vdpa/vdpa_user/vduse_dev.c > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c > > @@ -34,6 +34,7 @@ > > > > struct vduse_dev_msg { > > struct vduse_dev_request req; > > + struct file *iotlb_file; > > struct vduse_dev_response resp; > > struct list_head list; > > wait_queue_head_t waitq; > > @@ -325,12 +326,80 @@ static int vduse_dev_set_vq_state(struct vduse_de= v *dev, > > return ret; > > } > > > > +static int vduse_dev_update_iotlb(struct vduse_dev *dev, struct file *= file, > > + u64 offset, u64 iova, u64 size, u8 perm) > > +{ > > + struct vduse_dev_msg *msg; > > + int ret; > > + > > + if (!size) > > + return -EINVAL; > > + > > + msg =3D vduse_dev_new_msg(dev, VDUSE_UPDATE_IOTLB); > > + msg->req.size =3D sizeof(struct vduse_iotlb); > > + msg->req.iotlb.offset =3D offset; > > + msg->req.iotlb.iova =3D iova; > > + msg->req.iotlb.size =3D size; > > + msg->req.iotlb.perm =3D perm; > > + msg->req.iotlb.fd =3D -1; > > + msg->iotlb_file =3D get_file(file); > > + > > + ret =3D vduse_dev_msg_sync(dev, msg); > > > My feeling is that we should provide consistent API for the userspace > device to use. > > E.g we'd better carry the IOTLB message for both virtio/vhost drivers. > > It looks to me for virtio drivers we can still use UPDAT_IOTLB message > by using VDUSE file as msg->iotlb_file here. > It's OK for me. One problem is when to transfer the UPDATE_IOTLB message in virtio cases. > > > + vduse_dev_msg_put(msg); > > + fput(file); > > + > > + return ret; > > +} > > + > > +static int vduse_dev_invalidate_iotlb(struct vduse_dev *dev, > > + u64 iova, u64 size) > > +{ > > + struct vduse_dev_msg *msg; > > + int ret; > > + > > + if (!size) > > + return -EINVAL; > > + > > + msg =3D vduse_dev_new_msg(dev, VDUSE_INVALIDATE_IOTLB); > > + msg->req.size =3D sizeof(struct vduse_iotlb); > > + msg->req.iotlb.iova =3D iova; > > + msg->req.iotlb.size =3D size; > > + > > + ret =3D vduse_dev_msg_sync(dev, msg); > > + vduse_dev_msg_put(msg); > > + > > + return ret; > > +} > > + > > +static unsigned int perm_to_file_flags(u8 perm) > > +{ > > + unsigned int flags =3D 0; > > + > > + switch (perm) { > > + case VHOST_ACCESS_WO: > > + flags |=3D O_WRONLY; > > + break; > > + case VHOST_ACCESS_RO: > > + flags |=3D O_RDONLY; > > + break; > > + case VHOST_ACCESS_RW: > > + flags |=3D O_RDWR; > > + break; > > + default: > > + WARN(1, "invalidate vhost IOTLB permission\n"); > > + break; > > + } > > + > > + return flags; > > +} > > + > > static ssize_t vduse_dev_read_iter(struct kiocb *iocb, struct iov_ite= r *to) > > { > > struct file *file =3D iocb->ki_filp; > > struct vduse_dev *dev =3D file->private_data; > > struct vduse_dev_msg *msg; > > - int size =3D sizeof(struct vduse_dev_request); > > + unsigned int flags; > > + int fd, size =3D sizeof(struct vduse_dev_request); > > ssize_t ret =3D 0; > > > > if (iov_iter_count(to) < size) > > @@ -349,6 +418,18 @@ static ssize_t vduse_dev_read_iter(struct kiocb *i= ocb, struct iov_iter *to) > > if (ret) > > return ret; > > } > > + > > + if (msg->req.type =3D=3D VDUSE_UPDATE_IOTLB && msg->req.iotlb.fd = =3D=3D -1) { > > + flags =3D perm_to_file_flags(msg->req.iotlb.perm); > > + fd =3D get_unused_fd_flags(flags); > > + if (fd < 0) { > > + vduse_dev_enqueue_msg(dev, msg, &dev->send_list); > > + return fd; > > + } > > + fd_install(fd, get_file(msg->iotlb_file)); > > + msg->req.iotlb.fd =3D fd; > > + } > > + > > ret =3D copy_to_iter(&msg->req, size, to); > > if (ret !=3D size) { > > vduse_dev_enqueue_msg(dev, msg, &dev->send_list); > > @@ -565,6 +646,69 @@ static void vduse_vdpa_set_config(struct vdpa_devi= ce *vdpa, unsigned int offset, > > vduse_dev_set_config(dev, offset, buf, len); > > } > > > > +static void vduse_vdpa_invalidate_iotlb(struct vduse_dev *dev, > > + struct vhost_iotlb_msg *msg) > > +{ > > + vduse_dev_invalidate_iotlb(dev, msg->iova, msg->size); > > +} > > + > > +static int vduse_vdpa_update_iotlb(struct vduse_dev *dev, > > + struct vhost_iotlb_msg *msg) > > +{ > > + u64 uaddr =3D msg->uaddr; > > + u64 iova =3D msg->iova; > > + u64 size =3D msg->size; > > + u64 offset; > > + struct vm_area_struct *vma; > > + int ret; > > + > > + while (uaddr < msg->uaddr + msg->size) { > > + vma =3D find_vma(current->mm, uaddr); > > + ret =3D -EINVAL; > > + if (!vma) > > + goto err; > > + > > + size =3D min(msg->size, vma->vm_end - uaddr); > > + offset =3D (vma->vm_pgoff << PAGE_SHIFT) + uaddr - vma->v= m_start; > > + if (vma->vm_file && (vma->vm_flags & VM_SHARED)) { > > + ret =3D vduse_dev_update_iotlb(dev, vma->vm_file,= offset, > > + iova, size, msg->= perm); > > + if (ret) > > + goto err; > > > My understanding is that vma is something that should not be known by a > device. So I suggest to move the above processing to vhost-vdpa.c. > Will do it. Thanks, Yongji