From: Liu Xiaodong <xiaodong.liu@intel.com> To: Xie Yongji <xieyongji@bytedance.com> Cc: mst@redhat.com, jasowang@redhat.com, xiaodong.liu@intel.com, maxime.coquelin@redhat.com, stefanha@redhat.com, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 0/6] VDUSE: Support registering userspace memory as bounce buffer Date: Mon, 4 Jul 2022 05:26:52 -0400 [thread overview] Message-ID: <20220704092652.GB105370@storage2.sh.intel.com> (raw) In-Reply-To: <20220629082541.118-1-xieyongji@bytedance.com> On Wed, Jun 29, 2022 at 04:25:35PM +0800, Xie Yongji wrote: > Hi all, > > This series introduces some new ioctls: VDUSE_IOTLB_GET_INFO, > VDUSE_IOTLB_REG_UMEM and VDUSE_IOTLB_DEREG_UMEM to support > registering and de-registering userspace memory for IOTLB > as bounce buffer in virtio-vdpa case. > > The VDUSE_IOTLB_GET_INFO ioctl can help user to query IOLTB > information such as bounce buffer size. Then user can use > those information on VDUSE_IOTLB_REG_UMEM and > VDUSE_IOTLB_DEREG_UMEM ioctls to register and de-register > userspace memory for IOTLB. > > During registering and de-registering, the DMA data in use > would be copied from kernel bounce pages to userspace bounce > pages and back. > > With this feature, some existing application such as SPDK > and DPDK can leverage the datapath of VDUSE directly and > efficiently as discussed before [1]. They can register some > preallocated hugepages to VDUSE to avoid an extra memcpy > from bounce-buffer to hugepages. Hi, Yongji Very glad to see this enhancement in VDUSE. Thank you. It is really helpful and essential to SPDK. With this new feature, we can get VDUSE transferred data accessed directly by userspace physical backends, like RDMA and PCIe devices. In SPDK roadmap, it's one important work to export block services to local host, especially for container scenario. This patch could help SPDK do that with its userspace backend stacks while keeping high efficiency and performance. So the whole SPDK ecosystem can get benefited. Based on this enhancement, as discussed, I drafted a VDUSE prototype module in SPDK for initial evaluation: [TEST]vduse: prototype for initial draft https://review.spdk.io/gerrit/c/spdk/spdk/+/13534 Running SPDK on single CPU core, configured with 2 P3700 NVMe, and exported block devices to local host kernel via different protocols. The randwrite IOPS through each protocol are: NBD 121K NVMf-tcp loopback 274K VDUSE 463K SPDK with RDMA backends should have a similar ratio. VDUSE has a great performance advantage for SPDK. We have kept investigating on this usage for years. Originally, some SPDK users used NBD. Then NVMf-tcp loopback is SPDK community accommended way. In future, VDUSE could be the preferred way. > The kernel and userspace codes could be found in github: > > https://github.com/bytedance/linux/tree/vduse-umem > https://github.com/bytedance/qemu/tree/vduse-umem > > To test it with qemu-storage-daemon: > > $ qemu-storage-daemon \ > --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server=on,wait=off \ > --monitor chardev=charmonitor \ > --blockdev driver=host_device,cache.direct=on,aio=native,filename=/dev/nullb0,node-name=disk0 > \ > --export type=vduse-blk,id=vduse-test,name=vduse-test,node-name=disk0,writable=on > > [1] https://lkml.org/lkml/2021/6/27/318 > > Please review, thanks! Waiting for its review process. Thanks Xiaodong
WARNING: multiple messages have this Message-ID (diff)
From: Liu Xiaodong <xiaodong.liu@intel.com> To: Xie Yongji <xieyongji@bytedance.com> Cc: mst@redhat.com, xiaodong.liu@intel.com, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, maxime.coquelin@redhat.com, stefanha@redhat.com Subject: Re: [PATCH 0/6] VDUSE: Support registering userspace memory as bounce buffer Date: Mon, 4 Jul 2022 05:26:52 -0400 [thread overview] Message-ID: <20220704092652.GB105370@storage2.sh.intel.com> (raw) In-Reply-To: <20220629082541.118-1-xieyongji@bytedance.com> On Wed, Jun 29, 2022 at 04:25:35PM +0800, Xie Yongji wrote: > Hi all, > > This series introduces some new ioctls: VDUSE_IOTLB_GET_INFO, > VDUSE_IOTLB_REG_UMEM and VDUSE_IOTLB_DEREG_UMEM to support > registering and de-registering userspace memory for IOTLB > as bounce buffer in virtio-vdpa case. > > The VDUSE_IOTLB_GET_INFO ioctl can help user to query IOLTB > information such as bounce buffer size. Then user can use > those information on VDUSE_IOTLB_REG_UMEM and > VDUSE_IOTLB_DEREG_UMEM ioctls to register and de-register > userspace memory for IOTLB. > > During registering and de-registering, the DMA data in use > would be copied from kernel bounce pages to userspace bounce > pages and back. > > With this feature, some existing application such as SPDK > and DPDK can leverage the datapath of VDUSE directly and > efficiently as discussed before [1]. They can register some > preallocated hugepages to VDUSE to avoid an extra memcpy > from bounce-buffer to hugepages. Hi, Yongji Very glad to see this enhancement in VDUSE. Thank you. It is really helpful and essential to SPDK. With this new feature, we can get VDUSE transferred data accessed directly by userspace physical backends, like RDMA and PCIe devices. In SPDK roadmap, it's one important work to export block services to local host, especially for container scenario. This patch could help SPDK do that with its userspace backend stacks while keeping high efficiency and performance. So the whole SPDK ecosystem can get benefited. Based on this enhancement, as discussed, I drafted a VDUSE prototype module in SPDK for initial evaluation: [TEST]vduse: prototype for initial draft https://review.spdk.io/gerrit/c/spdk/spdk/+/13534 Running SPDK on single CPU core, configured with 2 P3700 NVMe, and exported block devices to local host kernel via different protocols. The randwrite IOPS through each protocol are: NBD 121K NVMf-tcp loopback 274K VDUSE 463K SPDK with RDMA backends should have a similar ratio. VDUSE has a great performance advantage for SPDK. We have kept investigating on this usage for years. Originally, some SPDK users used NBD. Then NVMf-tcp loopback is SPDK community accommended way. In future, VDUSE could be the preferred way. > The kernel and userspace codes could be found in github: > > https://github.com/bytedance/linux/tree/vduse-umem > https://github.com/bytedance/qemu/tree/vduse-umem > > To test it with qemu-storage-daemon: > > $ qemu-storage-daemon \ > --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server=on,wait=off \ > --monitor chardev=charmonitor \ > --blockdev driver=host_device,cache.direct=on,aio=native,filename=/dev/nullb0,node-name=disk0 > \ > --export type=vduse-blk,id=vduse-test,name=vduse-test,node-name=disk0,writable=on > > [1] https://lkml.org/lkml/2021/6/27/318 > > Please review, thanks! Waiting for its review process. Thanks Xiaodong _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2022-07-04 9:27 UTC|newest] Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-06-29 8:25 [PATCH 0/6] VDUSE: Support registering userspace memory as bounce buffer Xie Yongji 2022-06-29 8:25 ` [PATCH 1/6] vduse: Remove unnecessary spin lock protection Xie Yongji 2022-06-29 8:25 ` [PATCH 2/6] vduse: Use memcpy_{to,from}_page() in do_bounce() Xie Yongji 2022-06-29 8:25 ` [PATCH 3/6] vduse: Support using userspace pages as bounce buffer Xie Yongji 2022-06-29 8:25 ` [PATCH 4/6] vduse: Support querying IOLTB information Xie Yongji 2022-06-29 8:25 ` [PATCH 5/6] vduse: Support registering userspace memory for IOTLB Xie Yongji 2022-06-29 8:42 ` Michael S. Tsirkin 2022-06-29 8:42 ` Michael S. Tsirkin 2022-06-29 9:26 ` Yongji Xie 2022-06-29 9:54 ` Michael S. Tsirkin 2022-06-29 9:54 ` Michael S. Tsirkin 2022-06-29 10:19 ` Yongji Xie 2022-06-29 11:28 ` Michael S. Tsirkin 2022-06-29 11:28 ` Michael S. Tsirkin 2022-06-29 8:25 ` [PATCH 6/6] vduse: Update api version to 1 Xie Yongji 2022-06-29 8:33 ` Michael S. Tsirkin 2022-06-29 8:33 ` Michael S. Tsirkin 2022-06-29 9:02 ` Yongji Xie 2022-06-29 9:22 ` Michael S. Tsirkin 2022-06-29 9:22 ` Michael S. Tsirkin 2022-06-29 9:28 ` Yongji Xie 2022-07-04 9:26 ` Liu Xiaodong [this message] 2022-07-04 9:26 ` [PATCH 0/6] VDUSE: Support registering userspace memory as bounce buffer Liu Xiaodong 2022-07-04 10:02 ` Yongji Xie
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20220704092652.GB105370@storage2.sh.intel.com \ --to=xiaodong.liu@intel.com \ --cc=jasowang@redhat.com \ --cc=linux-kernel@vger.kernel.org \ --cc=maxime.coquelin@redhat.com \ --cc=mst@redhat.com \ --cc=stefanha@redhat.com \ --cc=virtualization@lists.linux-foundation.org \ --cc=xieyongji@bytedance.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.