From: "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> To: qemu-devel@nongnu.org, vgoyal@redhat.com, stefanha@redhat.com, groug@kaod.org Cc: virtio-fs@redhat.com Subject: [PATCH v3 00/26] virtiofs dax patches Date: Wed, 28 Apr 2021 12:00:34 +0100 [thread overview] Message-ID: <20210428110100.27757-1-dgilbert@redhat.com> (raw) From: "Dr. David Alan Gilbert" <dgilbert@redhat.com> This series adds support for acceleration of virtiofs via DAX mapping, using features added in the 5.11 Linux kernel. DAX originally existed in the kernel for mapping real storage devices directly into memory, so that reads/writes turn into reads/writes directly mapped into the storage device. virtiofs's DAX support is similar; a PCI BAR is exposed on the virtiofs device corresponding to a DAX 'cache' of a user defined size. The guest daemon then requests files to be mapped into that cache; when that happens the virtiofsd sends filedescriptors and commands back to the QEMU that mmap's those files directly into the memory slot exposed to kvm. The guest can then directly read/write to the files exposed by virtiofs by reading/writing into the BAR. A typical invocation would be: -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=4G and then the guest must mount with -o dax Note that the cache doesn't really take VM up on the host, because everything placed there is just an mmap of a file, so you can afford to use quite a large cache size. Unlike a real DAX device, the cache is a finite size that's potentially smaller than the underlying filesystem (especially when mapping granuality is taken into account). Mapping, unmapping and remapping must take place to juggle files into the cache if it's too small. Some workloads benefit more than others. Gotchas: a) If something else on the host truncates an mmap'd file, kvm gets rather upset; for this reason it's advised that DAX is currently only suitable for use on non-shared filesystems. (This series, with a couple of other patches, is at: https://gitlab.com/virtio-fs/qemu/-/tree/dgilbert-dax-2021-04-28 ) Dave v3 Review cleanups Fix some printf formating issues Dr. David Alan Gilbert (21): virtiofs: Fixup printf args virtiofsd: Don't assume header layout DAX: vhost-user: Rework slave return values DAX: libvhost-user: Route slave message payload DAX: libvhost-user: Allow popping a queue element with bad pointers DAX subprojects/libvhost-user: Add virtio-fs slave types DAX: virtio: Add shared memory capability DAX: virtio-fs: Add cache BAR DAX: virtio-fs: Add vhost-user slave commands for mapping DAX: virtio-fs: Fill in slave commands for mapping DAX: virtiofsd Add cache accessor functions DAX: virtiofsd: Add setup/remove mappings fuse commands DAX: virtiofsd: Add setup/remove mapping handlers to passthrough_ll DAX: virtiofsd: Wire up passthrough_ll's lo_setupmapping DAX: virtiofsd: route se down to destroy method DAX: virtiofsd: Perform an unmap on destroy DAX/unmap: virtiofsd: Add VHOST_USER_SLAVE_FS_IO DAX/unmap virtiofsd: Add wrappers for VHOST_USER_SLAVE_FS_IO DAX/unmap virtiofsd: Parse unmappable elements DAX/unmap virtiofsd: Route unmappable reads DAX/unmap virtiofsd: route unmappable write to slave command Stefan Hajnoczi (1): DAX:virtiofsd: implement FUSE_INIT map_alignment field Vivek Goyal (4): DAX: virtiofsd: Make lo_removemapping() work vhost-user-fs: Extend VhostUserFSSlaveMsg to pass additional info vhost-user-fs: Implement drop CAP_FSETID functionality virtiofsd: Ask qemu to drop CAP_FSETID if client asked for it block/export/vhost-user-blk-server.c | 2 +- contrib/vhost-user-blk/vhost-user-blk.c | 3 +- contrib/vhost-user-gpu/vhost-user-gpu.c | 5 +- contrib/vhost-user-input/main.c | 4 +- contrib/vhost-user-scsi/vhost-user-scsi.c | 2 +- docs/interop/vhost-user.rst | 37 ++ hw/virtio/meson.build | 1 + hw/virtio/trace-events | 6 + hw/virtio/vhost-backend.c | 6 +- hw/virtio/vhost-user-fs-pci.c | 32 ++ hw/virtio/vhost-user-fs.c | 395 ++++++++++++++++++++++ hw/virtio/vhost-user.c | 62 +++- hw/virtio/virtio-pci.c | 20 ++ hw/virtio/virtio-pci.h | 4 + include/hw/virtio/vhost-backend.h | 2 +- include/hw/virtio/vhost-user-fs.h | 43 +++ meson.build | 6 + subprojects/libvhost-user/libvhost-user.c | 113 ++++++- subprojects/libvhost-user/libvhost-user.h | 57 +++- tests/vhost-user-bridge.c | 4 +- tools/virtiofsd/buffer.c | 22 +- tools/virtiofsd/fuse_common.h | 17 +- tools/virtiofsd/fuse_lowlevel.c | 92 ++++- tools/virtiofsd/fuse_lowlevel.h | 78 ++++- tools/virtiofsd/fuse_virtio.c | 372 ++++++++++++++++---- tools/virtiofsd/passthrough_ll.c | 138 +++++++- 26 files changed, 1393 insertions(+), 130 deletions(-) -- 2.31.1
WARNING: multiple messages have this Message-ID (diff)
From: "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> To: qemu-devel@nongnu.org, vgoyal@redhat.com, stefanha@redhat.com, groug@kaod.org Cc: virtio-fs@redhat.com Subject: [Virtio-fs] [PATCH v3 00/26] virtiofs dax patches Date: Wed, 28 Apr 2021 12:00:34 +0100 [thread overview] Message-ID: <20210428110100.27757-1-dgilbert@redhat.com> (raw) From: "Dr. David Alan Gilbert" <dgilbert@redhat.com> This series adds support for acceleration of virtiofs via DAX mapping, using features added in the 5.11 Linux kernel. DAX originally existed in the kernel for mapping real storage devices directly into memory, so that reads/writes turn into reads/writes directly mapped into the storage device. virtiofs's DAX support is similar; a PCI BAR is exposed on the virtiofs device corresponding to a DAX 'cache' of a user defined size. The guest daemon then requests files to be mapped into that cache; when that happens the virtiofsd sends filedescriptors and commands back to the QEMU that mmap's those files directly into the memory slot exposed to kvm. The guest can then directly read/write to the files exposed by virtiofs by reading/writing into the BAR. A typical invocation would be: -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=4G and then the guest must mount with -o dax Note that the cache doesn't really take VM up on the host, because everything placed there is just an mmap of a file, so you can afford to use quite a large cache size. Unlike a real DAX device, the cache is a finite size that's potentially smaller than the underlying filesystem (especially when mapping granuality is taken into account). Mapping, unmapping and remapping must take place to juggle files into the cache if it's too small. Some workloads benefit more than others. Gotchas: a) If something else on the host truncates an mmap'd file, kvm gets rather upset; for this reason it's advised that DAX is currently only suitable for use on non-shared filesystems. (This series, with a couple of other patches, is at: https://gitlab.com/virtio-fs/qemu/-/tree/dgilbert-dax-2021-04-28 ) Dave v3 Review cleanups Fix some printf formating issues Dr. David Alan Gilbert (21): virtiofs: Fixup printf args virtiofsd: Don't assume header layout DAX: vhost-user: Rework slave return values DAX: libvhost-user: Route slave message payload DAX: libvhost-user: Allow popping a queue element with bad pointers DAX subprojects/libvhost-user: Add virtio-fs slave types DAX: virtio: Add shared memory capability DAX: virtio-fs: Add cache BAR DAX: virtio-fs: Add vhost-user slave commands for mapping DAX: virtio-fs: Fill in slave commands for mapping DAX: virtiofsd Add cache accessor functions DAX: virtiofsd: Add setup/remove mappings fuse commands DAX: virtiofsd: Add setup/remove mapping handlers to passthrough_ll DAX: virtiofsd: Wire up passthrough_ll's lo_setupmapping DAX: virtiofsd: route se down to destroy method DAX: virtiofsd: Perform an unmap on destroy DAX/unmap: virtiofsd: Add VHOST_USER_SLAVE_FS_IO DAX/unmap virtiofsd: Add wrappers for VHOST_USER_SLAVE_FS_IO DAX/unmap virtiofsd: Parse unmappable elements DAX/unmap virtiofsd: Route unmappable reads DAX/unmap virtiofsd: route unmappable write to slave command Stefan Hajnoczi (1): DAX:virtiofsd: implement FUSE_INIT map_alignment field Vivek Goyal (4): DAX: virtiofsd: Make lo_removemapping() work vhost-user-fs: Extend VhostUserFSSlaveMsg to pass additional info vhost-user-fs: Implement drop CAP_FSETID functionality virtiofsd: Ask qemu to drop CAP_FSETID if client asked for it block/export/vhost-user-blk-server.c | 2 +- contrib/vhost-user-blk/vhost-user-blk.c | 3 +- contrib/vhost-user-gpu/vhost-user-gpu.c | 5 +- contrib/vhost-user-input/main.c | 4 +- contrib/vhost-user-scsi/vhost-user-scsi.c | 2 +- docs/interop/vhost-user.rst | 37 ++ hw/virtio/meson.build | 1 + hw/virtio/trace-events | 6 + hw/virtio/vhost-backend.c | 6 +- hw/virtio/vhost-user-fs-pci.c | 32 ++ hw/virtio/vhost-user-fs.c | 395 ++++++++++++++++++++++ hw/virtio/vhost-user.c | 62 +++- hw/virtio/virtio-pci.c | 20 ++ hw/virtio/virtio-pci.h | 4 + include/hw/virtio/vhost-backend.h | 2 +- include/hw/virtio/vhost-user-fs.h | 43 +++ meson.build | 6 + subprojects/libvhost-user/libvhost-user.c | 113 ++++++- subprojects/libvhost-user/libvhost-user.h | 57 +++- tests/vhost-user-bridge.c | 4 +- tools/virtiofsd/buffer.c | 22 +- tools/virtiofsd/fuse_common.h | 17 +- tools/virtiofsd/fuse_lowlevel.c | 92 ++++- tools/virtiofsd/fuse_lowlevel.h | 78 ++++- tools/virtiofsd/fuse_virtio.c | 372 ++++++++++++++++---- tools/virtiofsd/passthrough_ll.c | 138 +++++++- 26 files changed, 1393 insertions(+), 130 deletions(-) -- 2.31.1
next reply other threads:[~2021-04-28 11:04 UTC|newest] Thread overview: 132+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-04-28 11:00 Dr. David Alan Gilbert (git) [this message] 2021-04-28 11:00 ` [Virtio-fs] [PATCH v3 00/26] virtiofs dax patches Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 01/26] virtiofs: Fixup printf args Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-04 14:54 ` Stefan Hajnoczi 2021-05-04 14:54 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-05 11:06 ` Dr. David Alan Gilbert 2021-05-05 11:06 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-05-06 15:56 ` Dr. David Alan Gilbert 2021-05-06 15:56 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:00 ` [PATCH v3 02/26] virtiofsd: Don't assume header layout Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-04 15:12 ` Stefan Hajnoczi 2021-05-04 15:12 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-06 15:56 ` Dr. David Alan Gilbert 2021-05-06 15:56 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:00 ` [PATCH v3 03/26] DAX: vhost-user: Rework slave return values Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-04 15:23 ` Stefan Hajnoczi 2021-05-04 15:23 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-27 15:59 ` Dr. David Alan Gilbert 2021-05-27 15:59 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:00 ` [PATCH v3 04/26] DAX: libvhost-user: Route slave message payload Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-04 15:26 ` Stefan Hajnoczi 2021-05-04 15:26 ` [Virtio-fs] " Stefan Hajnoczi 2021-04-28 11:00 ` [PATCH v3 05/26] DAX: libvhost-user: Allow popping a queue element with bad pointers Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 06/26] DAX subprojects/libvhost-user: Add virtio-fs slave types Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-29 15:48 ` Dr. David Alan Gilbert 2021-04-29 15:48 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:00 ` [PATCH v3 07/26] DAX: virtio: Add shared memory capability Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 08/26] DAX: virtio-fs: Add cache BAR Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-05 12:12 ` Stefan Hajnoczi 2021-05-05 12:12 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-05 18:59 ` Dr. David Alan Gilbert 2021-05-05 18:59 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:00 ` [PATCH v3 09/26] DAX: virtio-fs: Add vhost-user slave commands for mapping Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-05 14:15 ` Stefan Hajnoczi 2021-05-05 14:15 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-27 16:57 ` Dr. David Alan Gilbert 2021-05-27 16:57 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:00 ` [PATCH v3 10/26] DAX: virtio-fs: Fill in " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-05 16:43 ` Stefan Hajnoczi 2021-05-05 16:43 ` [Virtio-fs] " Stefan Hajnoczi 2021-04-28 11:00 ` [PATCH v3 11/26] DAX: virtiofsd Add cache accessor functions Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 12/26] DAX: virtiofsd: Add setup/remove mappings fuse commands Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-06 15:02 ` Stefan Hajnoczi 2021-05-06 15:02 ` [Virtio-fs] " Stefan Hajnoczi 2021-04-28 11:00 ` [PATCH v3 13/26] DAX: virtiofsd: Add setup/remove mapping handlers to passthrough_ll Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 14/26] DAX: virtiofsd: Wire up passthrough_ll's lo_setupmapping Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 15/26] DAX: virtiofsd: Make lo_removemapping() work Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 16/26] DAX: virtiofsd: route se down to destroy method Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 17/26] DAX: virtiofsd: Perform an unmap on destroy Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 18/26] DAX/unmap: virtiofsd: Add VHOST_USER_SLAVE_FS_IO Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-06 15:12 ` Stefan Hajnoczi 2021-05-06 15:12 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-27 17:44 ` Dr. David Alan Gilbert 2021-05-27 17:44 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-05-06 15:16 ` Stefan Hajnoczi 2021-05-06 15:16 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-27 17:31 ` Dr. David Alan Gilbert 2021-05-27 17:31 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:00 ` [PATCH v3 19/26] DAX/unmap virtiofsd: Add wrappers for VHOST_USER_SLAVE_FS_IO Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 12:53 ` Dr. David Alan Gilbert 2021-04-28 12:53 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:00 ` [PATCH v3 20/26] DAX/unmap virtiofsd: Parse unmappable elements Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-06 15:23 ` Stefan Hajnoczi 2021-05-06 15:23 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-27 17:56 ` Dr. David Alan Gilbert 2021-05-27 17:56 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:00 ` [PATCH v3 21/26] DAX/unmap virtiofsd: Route unmappable reads Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-06 15:27 ` Stefan Hajnoczi 2021-05-06 15:27 ` [Virtio-fs] " Stefan Hajnoczi 2021-04-28 11:00 ` [PATCH v3 22/26] DAX/unmap virtiofsd: route unmappable write to slave command Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-06 15:28 ` Stefan Hajnoczi 2021-05-06 15:28 ` [Virtio-fs] " Stefan Hajnoczi 2021-04-28 11:00 ` [PATCH v3 23/26] DAX:virtiofsd: implement FUSE_INIT map_alignment field Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [PATCH v3 24/26] vhost-user-fs: Extend VhostUserFSSlaveMsg to pass additional info Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-06 15:31 ` Stefan Hajnoczi 2021-05-06 15:31 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-06 15:32 ` Stefan Hajnoczi 2021-05-06 15:32 ` [Virtio-fs] " Stefan Hajnoczi 2021-04-28 11:00 ` [PATCH v3 25/26] vhost-user-fs: Implement drop CAP_FSETID functionality Dr. David Alan Gilbert (git) 2021-04-28 11:00 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-04-28 11:01 ` [PATCH v3 26/26] virtiofsd: Ask qemu to drop CAP_FSETID if client asked for it Dr. David Alan Gilbert (git) 2021-04-28 11:01 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-05-06 15:37 ` Stefan Hajnoczi 2021-05-06 15:37 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-06 16:02 ` Vivek Goyal 2021-05-06 16:02 ` [Virtio-fs] " Vivek Goyal 2021-05-10 9:05 ` Stefan Hajnoczi 2021-05-10 9:05 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-10 15:23 ` Vivek Goyal 2021-05-10 15:23 ` [Virtio-fs] " Vivek Goyal 2021-05-10 15:32 ` Stefan Hajnoczi 2021-05-10 15:32 ` [Virtio-fs] " Stefan Hajnoczi 2021-05-27 19:09 ` Dr. David Alan Gilbert 2021-05-27 19:09 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-06-10 15:29 ` Dr. David Alan Gilbert 2021-06-10 15:29 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-06-10 16:23 ` Stefan Hajnoczi 2021-06-10 16:23 ` [Virtio-fs] " Stefan Hajnoczi 2021-06-16 12:36 ` Dr. David Alan Gilbert 2021-06-16 12:36 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-06-16 15:29 ` Stefan Hajnoczi 2021-06-16 15:29 ` [Virtio-fs] " Stefan Hajnoczi 2021-06-16 18:35 ` Dr. David Alan Gilbert 2021-06-16 18:35 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-04-28 11:27 ` [PATCH v3 00/26] virtiofs dax patches no-reply 2021-04-28 11:27 ` [Virtio-fs] " no-reply 2021-05-06 15:37 ` Stefan Hajnoczi 2021-05-06 15:37 ` [Virtio-fs] " Stefan Hajnoczi
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210428110100.27757-1-dgilbert@redhat.com \ --to=dgilbert@redhat.com \ --cc=groug@kaod.org \ --cc=qemu-devel@nongnu.org \ --cc=stefanha@redhat.com \ --cc=vgoyal@redhat.com \ --cc=virtio-fs@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.