From: Yongji Xie <xieyongji@bytedance.com>
To: "Liu, Xiaodong" <xiaodong.liu@intel.com>
Cc: Jason Wang <jasowang@redhat.com>,
"mst@redhat.com" <mst@redhat.com>,
"stefanha@redhat.com" <stefanha@redhat.com>,
"sgarzare@redhat.com" <sgarzare@redhat.com>,
"parav@nvidia.com" <parav@nvidia.com>,
"hch@infradead.org" <hch@infradead.org>,
"christian.brauner@canonical.com"
<christian.brauner@canonical.com>,
"rdunlap@infradead.org" <rdunlap@infradead.org>,
"willy@infradead.org" <willy@infradead.org>,
"viro@zeniv.linux.org.uk" <viro@zeniv.linux.org.uk>,
"axboe@kernel.dk" <axboe@kernel.dk>,
"bcrl@kvack.org" <bcrl@kvack.org>,
"corbet@lwn.net" <corbet@lwn.net>,
"mika.penttila@nextfour.com" <mika.penttila@nextfour.com>,
"dan.carpenter@oracle.com" <dan.carpenter@oracle.com>,
"joro@8bytes.org" <joro@8bytes.org>,
"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
"songmuchun@bytedance.com" <songmuchun@bytedance.com>,
"virtualization@lists.linux-foundation.org"
<virtualization@lists.linux-foundation.org>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: RE: [PATCH v8 00/10] Introduce VDUSE - vDPA Device in Userspace
Date: Tue, 29 Jun 2021 16:14:35 +0800 [thread overview]
Message-ID: <CACycT3sfZCpWqB+GKQYMi3WjOkU0jAkBPU-u+MHHDCbLUvvu4w@mail.gmail.com> (raw)
In-Reply-To: <BYAPR11MB266276002F42D91FCE6E83CE8C029@BYAPR11MB2662.namprd11.prod.outlook.com>
On Tue, Jun 29, 2021 at 3:56 PM Liu, Xiaodong <xiaodong.liu@intel.com> wrote:
>
>
>
> >-----Original Message-----
> >From: Jason Wang <jasowang@redhat.com>
> >Sent: Tuesday, June 29, 2021 12:11 PM
> >To: Liu, Xiaodong <xiaodong.liu@intel.com>; Xie Yongji
> ><xieyongji@bytedance.com>; mst@redhat.com; stefanha@redhat.com;
> >sgarzare@redhat.com; parav@nvidia.com; hch@infradead.org;
> >christian.brauner@canonical.com; rdunlap@infradead.org; willy@infradead.org;
> >viro@zeniv.linux.org.uk; axboe@kernel.dk; bcrl@kvack.org; corbet@lwn.net;
> >mika.penttila@nextfour.com; dan.carpenter@oracle.com; joro@8bytes.org;
> >gregkh@linuxfoundation.org
> >Cc: songmuchun@bytedance.com; virtualization@lists.linux-foundation.org;
> >netdev@vger.kernel.org; kvm@vger.kernel.org; linux-fsdevel@vger.kernel.org;
> >iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
> >Subject: Re: [PATCH v8 00/10] Introduce VDUSE - vDPA Device in Userspace
> >
> >
> >在 2021/6/28 下午1:54, Liu, Xiaodong 写道:
> >>> Several issues:
> >>>
> >>> - VDUSE needs to limit the total size of the bounce buffers (64M if I was not
> >>> wrong). Does it work for SPDK?
> >> Yes, Jason. It is enough and works for SPDK.
> >> Since it's a kind of bounce buffer mainly for in-flight IO, so limited size like
> >> 64MB is enough.
> >
> >
> >Ok.
> >
> >
> >>
> >>> - VDUSE can use hugepages but I'm not sure we can mandate hugepages (or
> >we
> >>> need introduce new flags for supporting this)
> >> Same with your worry, I'm afraid too that it is a hard for a kernel module
> >> to directly preallocate hugepage internal.
> >> What I tried is that:
> >> 1. A simple agent daemon (represents for one device) `preallocates` and maps
> >> dozens of 2MB hugepages (like 64MB) for one device.
> >> 2. The daemon passes its mapping addr&len and hugepage fd to kernel
> >> module through created IOCTL.
> >> 3. Kernel module remaps the hugepages inside kernel.
> >
> >
> >Such model should work, but the main "issue" is that it introduce
> >overheads in the case of vhost-vDPA.
> >
> >Note that in the case of vhost-vDPA, we don't use bounce buffer, the
> >userspace pages were shared directly.
> >
> >And since DMA is not done per page, it prevents us from using tricks
> >like vm_insert_page() in those cases.
> >
>
> Yes, really, it's a problem to handle vhost-vDPA case.
> But there are already several solutions to get VM served, like vhost-user,
> vfio-user, so at least for SPDK, it won't serve VM through VDUSE. If a user
> still want to do that, then the user should tolerate Introduced overhead.
>
> In other words, software backend like SPDK, will appreciate the virtio
> datapath of VDUSE to serve local host instead of VM. That's why I also drafted
> a "virtio-local" to bridge vhost-user target and local host kernel virtio-blk.
>
> >
> >> 4. Vhost user target gets and maps hugepage fd from kernel module
> >> in vhost-user msg through Unix Domain Socket cmsg.
> >> Then kernel module and target map on the same hugepage based
> >> bounce buffer for in-flight IO.
> >>
> >> If there is one option in VDUSE to map userspace preallocated memory, then
> >> VDUSE should be able to mandate it even it is hugepage based.
> >>
> >
> >As above, this requires some kind of re-design since VDUSE depends on
> >the model of mmap(MAP_SHARED) instead of umem registering.
>
> Got it, Jason, this may be hard for current version of VDUSE.
> Maybe we can consider these options after VDUSE merged later.
>
> Since if VDUSE datapath could be directly leveraged by vhost-user target,
> its value will be propagated immediately.
>
Agreed!
Thanks,
Yongji
next prev parent reply other threads:[~2021-06-29 8:14 UTC|newest]
Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-15 14:13 [PATCH v8 00/10] Introduce VDUSE - vDPA Device in Userspace Xie Yongji
2021-06-15 14:13 ` [PATCH v8 01/10] iova: Export alloc_iova_fast() and free_iova_fast(); Xie Yongji
2021-06-15 14:13 ` [PATCH v8 02/10] file: Export receive_fd() to modules Xie Yongji
2021-06-15 14:13 ` [PATCH v8 03/10] eventfd: Increase the recursion depth of eventfd_signal() Xie Yongji
2021-06-17 8:33 ` He Zhe
2021-06-18 3:29 ` Yongji Xie
2021-06-18 8:41 ` He Zhe
2021-06-18 8:44 ` [PATCH] eventfd: Enlarge recursion limit to allow vhost to work He Zhe
2021-07-03 8:31 ` Michael S. Tsirkin
2021-08-25 7:57 ` Yongji Xie
2021-06-15 14:13 ` [PATCH v8 04/10] vhost-iotlb: Add an opaque pointer for vhost IOTLB Xie Yongji
2021-06-15 14:13 ` [PATCH v8 05/10] vdpa: Add an opaque pointer for vdpa_config_ops.dma_map() Xie Yongji
2021-06-15 14:13 ` [PATCH v8 06/10] vdpa: factor out vhost_vdpa_pa_map() and vhost_vdpa_pa_unmap() Xie Yongji
2021-06-15 14:13 ` [PATCH v8 07/10] vdpa: Support transferring virtual addressing during DMA mapping Xie Yongji
2021-06-15 14:13 ` [PATCH v8 08/10] vduse: Implement an MMU-based IOMMU driver Xie Yongji
2021-06-15 14:13 ` [PATCH v8 09/10] vduse: Introduce VDUSE - vDPA Device in Userspace Xie Yongji
2021-06-21 9:13 ` Jason Wang
2021-06-21 10:41 ` Yongji Xie
2021-06-22 5:06 ` Jason Wang
2021-06-22 7:22 ` Yongji Xie
2021-06-22 7:49 ` Jason Wang
2021-06-22 8:14 ` Yongji Xie
2021-06-23 3:30 ` Jason Wang
2021-06-23 5:50 ` Yongji Xie
2021-06-24 3:34 ` Jason Wang
2021-06-24 4:46 ` Yongji Xie
2021-06-24 8:13 ` Jason Wang
2021-06-24 9:16 ` Yongji Xie
2021-06-25 3:08 ` Jason Wang
2021-06-25 4:19 ` Yongji Xie
2021-06-28 4:40 ` Jason Wang
2021-06-29 2:26 ` Yongji Xie
2021-06-29 3:29 ` Jason Wang
2021-06-29 3:56 ` Yongji Xie
2021-06-29 4:03 ` Jason Wang
2021-06-24 14:46 ` Stefan Hajnoczi
2021-06-29 2:59 ` Yongji Xie
2021-06-30 9:51 ` Stefan Hajnoczi
2021-07-01 6:50 ` Yongji Xie
2021-07-01 7:55 ` Jason Wang
2021-07-01 10:26 ` Yongji Xie
2021-07-02 3:25 ` Jason Wang
2021-07-07 8:52 ` Stefan Hajnoczi
2021-07-07 9:19 ` Yongji Xie
2021-06-15 14:13 ` [PATCH v8 10/10] Documentation: Add documentation for VDUSE Xie Yongji
2021-06-24 13:01 ` Stefan Hajnoczi
2021-06-29 5:43 ` Yongji Xie
2021-06-30 10:06 ` Stefan Hajnoczi
2021-07-01 10:00 ` Yongji Xie
2021-07-01 13:15 ` Stefan Hajnoczi
2021-07-04 9:49 ` Yongji Xie
2021-07-05 3:36 ` Jason Wang
2021-07-05 12:49 ` Stefan Hajnoczi
2021-07-06 2:34 ` Jason Wang
2021-07-06 10:14 ` Stefan Hajnoczi
[not found] ` <CACGkMEs2HHbUfarum8uQ6wuXoDwLQUSXTsa-huJFiqr__4cwRg@mail.gmail.com>
[not found] ` <YOSOsrQWySr0andk@stefanha-x1.localdomain>
[not found] ` <100e6788-7fdf-1505-d69c-bc28a8bc7a78@redhat.com>
[not found] ` <YOVr801d01YOPzLL@stefanha-x1.localdomain>
2021-07-07 9:24 ` Jason Wang
2021-07-07 15:54 ` Stefan Hajnoczi
2021-07-08 4:17 ` Jason Wang
2021-07-08 9:06 ` Stefan Hajnoczi
2021-07-08 12:35 ` Yongji Xie
2021-07-06 3:04 ` Yongji Xie
2021-07-06 10:22 ` Stefan Hajnoczi
2021-07-07 9:09 ` Yongji Xie
2021-07-08 9:07 ` Stefan Hajnoczi
2021-06-24 15:12 ` [PATCH v8 00/10] Introduce VDUSE - vDPA Device in Userspace Stefan Hajnoczi
2021-06-29 3:15 ` Yongji Xie
2021-06-28 10:33 ` Liu Xiaodong
2021-06-28 4:35 ` Jason Wang
2021-06-28 5:54 ` Liu, Xiaodong
2021-06-29 4:10 ` Jason Wang
2021-06-29 7:56 ` Liu, Xiaodong
2021-06-29 8:14 ` Yongji Xie [this message]
2021-06-28 10:32 ` Yongji Xie
2021-06-29 4:12 ` Jason Wang
2021-06-29 6:40 ` Yongji Xie
2021-06-29 7:33 ` Jason Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CACycT3sfZCpWqB+GKQYMi3WjOkU0jAkBPU-u+MHHDCbLUvvu4w@mail.gmail.com \
--to=xieyongji@bytedance.com \
--cc=axboe@kernel.dk \
--cc=bcrl@kvack.org \
--cc=christian.brauner@canonical.com \
--cc=corbet@lwn.net \
--cc=dan.carpenter@oracle.com \
--cc=gregkh@linuxfoundation.org \
--cc=hch@infradead.org \
--cc=iommu@lists.linux-foundation.org \
--cc=jasowang@redhat.com \
--cc=joro@8bytes.org \
--cc=kvm@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mika.penttila@nextfour.com \
--cc=mst@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=parav@nvidia.com \
--cc=rdunlap@infradead.org \
--cc=sgarzare@redhat.com \
--cc=songmuchun@bytedance.com \
--cc=stefanha@redhat.com \
--cc=viro@zeniv.linux.org.uk \
--cc=virtualization@lists.linux-foundation.org \
--cc=willy@infradead.org \
--cc=xiaodong.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).