From: "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> To: qemu-devel@nongnu.org, vgoyal@redhat.com, stefanha@redhat.com, virtio-fs@redhat.com, marcandre.lureau@redhat.com, mst@redhat.com Subject: [PATCH 00/24] virtiofs dax patches Date: Tue, 9 Feb 2021 19:02:00 +0000 [thread overview] Message-ID: <20210209190224.62827-1-dgilbert@redhat.com> (raw) From: "Dr. David Alan Gilbert" <dgilbert@redhat.com> Hi, This series adds support for acceleration of virtiofs via DAX mapping, using features added in the 5.11 Linux kernel. DAX originally existed in the kernel for mapping real storage devices directly into memory, so that reads/writes turn into reads/writes directly mapped into the storage device. virtiofs's DAX support is similar; a PCI BAR is exposed on the virtiofs device corresponding to a DAX 'cache' of a user defined size. The guest daemon then requests files to be mapped into that cache; when that happens the virtiofsd sends filedescriptors and commands back to the QEMU that mmap's those files directly into the memory slot exposed to kvm. The guest can then directly read/write to the files exposed by virtiofs by reading/writing into the BAR. A typical invocation would be: -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=4G and then the guest must mount with -o mount. Note that the cache doesn't really take VM up on the host, because everything placed there is just an mmap of a file, so you can afford to use quite a large cache size. Unlike a real DAX device, the cache is a finite size that's potentially smaller than the underlying filesystem (especially when mapping granuality is taken into account). Mapping, unmapping and remapping must take place to juggle files into the cache if it's too small. Some workloads benefit more than others. Gotchas: a) The vhost-user slave channel has some bad reset behaviours; these are fixed by Vivek's '[RFC PATCH 0/6] vhost-user: Shutdown/Flush slave channel properly' that are on the list. b) If something else on the host truncates an mmap'd file, kvm gets rather upset; for this reason it's advised that DAX is currently only suitable for use on non-shared filesystems. Thanks a lot to Vivek who has spent a lot of time on the kernel side and cleaning this series up. Dave Dr. David Alan Gilbert (19): DAX: vhost-user: Rework slave return values DAX: libvhost-user: Route slave message payload DAX: libvhost-user: Allow popping a queue element with bad pointers DAX subprojects/libvhost-user: Add virtio-fs slave types DAX: virtio: Add shared memory capability DAX: virtio-fs: Add cache BAR DAX: virtio-fs: Add vhost-user slave commands for mapping DAX: virtio-fs: Fill in slave commands for mapping DAX: virtiofsd Add cache accessor functions DAX: virtiofsd: Add setup/remove mappings fuse commands DAX: virtiofsd: Add setup/remove mapping handlers to passthrough_ll DAX: virtiofsd: Wire up passthrough_ll's lo_setupmapping DAX: virtiofsd: route se down to destroy method DAX: virtiofsd: Perform an unmap on destroy DAX/unmap: virtiofsd: Add VHOST_USER_SLAVE_FS_IO DAX/unmap virtiofsd: Add wrappers for VHOST_USER_SLAVE_FS_IO DAX/unmap virtiofsd: Parse unmappable elements DAX/unmap virtiofsd: Route unmappable reads DAX/unmap virtiofsd: route unmappable write to slave command Stefan Hajnoczi (1): DAX:virtiofsd: implement FUSE_INIT map_alignment field Vivek Goyal (4): DAX: virtiofsd: Make lo_removemapping() work vhost-user-fs: Extend VhostUserFSSlaveMsg to pass additional info vhost-user-fs: Implement drop CAP_FSETID functionality virtiofsd: Ask qemu to drop CAP_FSETID if client asked for it block/export/vhost-user-blk-server.c | 2 +- contrib/vhost-user-blk/vhost-user-blk.c | 3 +- contrib/vhost-user-gpu/vhost-user-gpu.c | 5 +- contrib/vhost-user-input/main.c | 4 +- contrib/vhost-user-scsi/vhost-user-scsi.c | 2 +- docs/interop/vhost-user.rst | 31 ++ hw/virtio/meson.build | 1 + hw/virtio/trace-events | 6 + hw/virtio/vhost-backend.c | 4 +- hw/virtio/vhost-user-fs-pci.c | 25 ++ hw/virtio/vhost-user-fs.c | 330 ++++++++++++++++++++++ hw/virtio/vhost-user.c | 50 +++- hw/virtio/virtio-pci.c | 20 ++ hw/virtio/virtio-pci.h | 4 + include/hw/virtio/vhost-backend.h | 2 +- include/hw/virtio/vhost-user-fs.h | 34 +++ meson.build | 6 + subprojects/libvhost-user/libvhost-user.c | 106 ++++++- subprojects/libvhost-user/libvhost-user.h | 48 +++- tests/vhost-user-bridge.c | 4 +- tools/virtiofsd/buffer.c | 22 +- tools/virtiofsd/fuse_common.h | 17 +- tools/virtiofsd/fuse_lowlevel.c | 91 +++++- tools/virtiofsd/fuse_lowlevel.h | 78 ++++- tools/virtiofsd/fuse_virtio.c | 282 ++++++++++++++---- tools/virtiofsd/passthrough_ll.c | 103 ++++++- 26 files changed, 1166 insertions(+), 114 deletions(-) -- 2.29.2
WARNING: multiple messages have this Message-ID (diff)
From: "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> To: qemu-devel@nongnu.org, vgoyal@redhat.com, stefanha@redhat.com, virtio-fs@redhat.com, marcandre.lureau@redhat.com, mst@redhat.com Subject: [Virtio-fs] [PATCH 00/24] virtiofs dax patches Date: Tue, 9 Feb 2021 19:02:00 +0000 [thread overview] Message-ID: <20210209190224.62827-1-dgilbert@redhat.com> (raw) From: "Dr. David Alan Gilbert" <dgilbert@redhat.com> Hi, This series adds support for acceleration of virtiofs via DAX mapping, using features added in the 5.11 Linux kernel. DAX originally existed in the kernel for mapping real storage devices directly into memory, so that reads/writes turn into reads/writes directly mapped into the storage device. virtiofs's DAX support is similar; a PCI BAR is exposed on the virtiofs device corresponding to a DAX 'cache' of a user defined size. The guest daemon then requests files to be mapped into that cache; when that happens the virtiofsd sends filedescriptors and commands back to the QEMU that mmap's those files directly into the memory slot exposed to kvm. The guest can then directly read/write to the files exposed by virtiofs by reading/writing into the BAR. A typical invocation would be: -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=4G and then the guest must mount with -o mount. Note that the cache doesn't really take VM up on the host, because everything placed there is just an mmap of a file, so you can afford to use quite a large cache size. Unlike a real DAX device, the cache is a finite size that's potentially smaller than the underlying filesystem (especially when mapping granuality is taken into account). Mapping, unmapping and remapping must take place to juggle files into the cache if it's too small. Some workloads benefit more than others. Gotchas: a) The vhost-user slave channel has some bad reset behaviours; these are fixed by Vivek's '[RFC PATCH 0/6] vhost-user: Shutdown/Flush slave channel properly' that are on the list. b) If something else on the host truncates an mmap'd file, kvm gets rather upset; for this reason it's advised that DAX is currently only suitable for use on non-shared filesystems. Thanks a lot to Vivek who has spent a lot of time on the kernel side and cleaning this series up. Dave Dr. David Alan Gilbert (19): DAX: vhost-user: Rework slave return values DAX: libvhost-user: Route slave message payload DAX: libvhost-user: Allow popping a queue element with bad pointers DAX subprojects/libvhost-user: Add virtio-fs slave types DAX: virtio: Add shared memory capability DAX: virtio-fs: Add cache BAR DAX: virtio-fs: Add vhost-user slave commands for mapping DAX: virtio-fs: Fill in slave commands for mapping DAX: virtiofsd Add cache accessor functions DAX: virtiofsd: Add setup/remove mappings fuse commands DAX: virtiofsd: Add setup/remove mapping handlers to passthrough_ll DAX: virtiofsd: Wire up passthrough_ll's lo_setupmapping DAX: virtiofsd: route se down to destroy method DAX: virtiofsd: Perform an unmap on destroy DAX/unmap: virtiofsd: Add VHOST_USER_SLAVE_FS_IO DAX/unmap virtiofsd: Add wrappers for VHOST_USER_SLAVE_FS_IO DAX/unmap virtiofsd: Parse unmappable elements DAX/unmap virtiofsd: Route unmappable reads DAX/unmap virtiofsd: route unmappable write to slave command Stefan Hajnoczi (1): DAX:virtiofsd: implement FUSE_INIT map_alignment field Vivek Goyal (4): DAX: virtiofsd: Make lo_removemapping() work vhost-user-fs: Extend VhostUserFSSlaveMsg to pass additional info vhost-user-fs: Implement drop CAP_FSETID functionality virtiofsd: Ask qemu to drop CAP_FSETID if client asked for it block/export/vhost-user-blk-server.c | 2 +- contrib/vhost-user-blk/vhost-user-blk.c | 3 +- contrib/vhost-user-gpu/vhost-user-gpu.c | 5 +- contrib/vhost-user-input/main.c | 4 +- contrib/vhost-user-scsi/vhost-user-scsi.c | 2 +- docs/interop/vhost-user.rst | 31 ++ hw/virtio/meson.build | 1 + hw/virtio/trace-events | 6 + hw/virtio/vhost-backend.c | 4 +- hw/virtio/vhost-user-fs-pci.c | 25 ++ hw/virtio/vhost-user-fs.c | 330 ++++++++++++++++++++++ hw/virtio/vhost-user.c | 50 +++- hw/virtio/virtio-pci.c | 20 ++ hw/virtio/virtio-pci.h | 4 + include/hw/virtio/vhost-backend.h | 2 +- include/hw/virtio/vhost-user-fs.h | 34 +++ meson.build | 6 + subprojects/libvhost-user/libvhost-user.c | 106 ++++++- subprojects/libvhost-user/libvhost-user.h | 48 +++- tests/vhost-user-bridge.c | 4 +- tools/virtiofsd/buffer.c | 22 +- tools/virtiofsd/fuse_common.h | 17 +- tools/virtiofsd/fuse_lowlevel.c | 91 +++++- tools/virtiofsd/fuse_lowlevel.h | 78 ++++- tools/virtiofsd/fuse_virtio.c | 282 ++++++++++++++---- tools/virtiofsd/passthrough_ll.c | 103 ++++++- 26 files changed, 1166 insertions(+), 114 deletions(-) -- 2.29.2
next reply other threads:[~2021-02-09 19:09 UTC|newest] Thread overview: 138+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-02-09 19:02 Dr. David Alan Gilbert (git) [this message] 2021-02-09 19:02 ` [Virtio-fs] [PATCH 00/24] virtiofs dax patches Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [PATCH 01/24] DAX: vhost-user: Rework slave return values Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 9:59 ` Stefan Hajnoczi 2021-02-11 9:59 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-11 15:27 ` Vivek Goyal 2021-02-11 15:27 ` [Virtio-fs] " Vivek Goyal 2021-02-18 12:18 ` Dr. David Alan Gilbert 2021-02-18 12:18 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-02-09 19:02 ` [PATCH 02/24] DAX: libvhost-user: Route slave message payload Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 10:05 ` Stefan Hajnoczi 2021-02-11 10:05 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 03/24] DAX: libvhost-user: Allow popping a queue element with bad pointers Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 10:12 ` Stefan Hajnoczi 2021-02-11 10:12 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 04/24] DAX subprojects/libvhost-user: Add virtio-fs slave types Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 10:16 ` Stefan Hajnoczi 2021-02-11 10:16 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 05/24] DAX: virtio: Add shared memory capability Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 10:17 ` Stefan Hajnoczi 2021-02-11 10:17 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 06/24] DAX: virtio-fs: Add cache BAR Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 10:25 ` Stefan Hajnoczi 2021-02-11 10:25 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-18 17:33 ` Dr. David Alan Gilbert 2021-02-18 17:33 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-02-09 19:02 ` [PATCH 07/24] DAX: virtio-fs: Add vhost-user slave commands for mapping Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 10:32 ` Stefan Hajnoczi 2021-02-11 10:32 ` [Virtio-fs] " Stefan Hajnoczi 2021-03-08 17:04 ` Dr. David Alan Gilbert 2021-03-08 17:04 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-02-15 10:35 ` Chirantan Ekbote 2021-02-15 10:35 ` Chirantan Ekbote 2021-02-15 13:25 ` Dr. David Alan Gilbert 2021-02-15 13:25 ` Dr. David Alan Gilbert 2021-02-15 14:24 ` Vivek Goyal 2021-02-15 14:24 ` Vivek Goyal 2021-03-11 12:15 ` Dr. David Alan Gilbert 2021-03-11 12:15 ` Dr. David Alan Gilbert 2021-03-11 13:50 ` Vivek Goyal 2021-03-11 13:50 ` Vivek Goyal 2021-03-11 18:52 ` Dr. David Alan Gilbert 2021-03-11 18:52 ` Dr. David Alan Gilbert 2021-02-09 19:02 ` [PATCH 08/24] DAX: virtio-fs: Fill in " Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 10:57 ` Stefan Hajnoczi 2021-02-11 10:57 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-18 10:59 ` Dr. David Alan Gilbert 2021-02-18 10:59 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-02-09 19:02 ` [PATCH 09/24] DAX: virtiofsd Add cache accessor functions Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 12:31 ` Stefan Hajnoczi 2021-02-11 12:31 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 10/24] DAX: virtiofsd: Add setup/remove mappings fuse commands Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 12:37 ` Stefan Hajnoczi 2021-02-11 12:37 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-11 16:39 ` Dr. David Alan Gilbert 2021-02-11 16:39 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-02-11 18:30 ` Vivek Goyal 2021-02-11 18:30 ` [Virtio-fs] " Vivek Goyal 2021-02-11 19:50 ` Dr. David Alan Gilbert 2021-02-11 19:50 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-02-11 20:15 ` Vivek Goyal 2021-02-11 20:15 ` [Virtio-fs] " Vivek Goyal 2021-02-09 19:02 ` [PATCH 11/24] DAX: virtiofsd: Add setup/remove mapping handlers to passthrough_ll Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 12:37 ` Stefan Hajnoczi 2021-02-11 12:37 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 12/24] DAX: virtiofsd: Wire up passthrough_ll's lo_setupmapping Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 12:41 ` Stefan Hajnoczi 2021-02-11 12:41 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-11 16:05 ` Vivek Goyal 2021-02-11 16:05 ` [Virtio-fs] " Vivek Goyal 2021-02-09 19:02 ` [PATCH 13/24] DAX: virtiofsd: Make lo_removemapping() work Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 12:41 ` Stefan Hajnoczi 2021-02-11 12:41 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 14/24] DAX: virtiofsd: route se down to destroy method Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 12:42 ` Stefan Hajnoczi 2021-02-11 12:42 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 15/24] DAX: virtiofsd: Perform an unmap on destroy Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 12:42 ` Stefan Hajnoczi 2021-02-11 12:42 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 16/24] DAX/unmap: virtiofsd: Add VHOST_USER_SLAVE_FS_IO Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 14:17 ` Stefan Hajnoczi 2021-02-11 14:17 ` [Virtio-fs] " Stefan Hajnoczi 2021-03-16 19:59 ` Dr. David Alan Gilbert 2021-03-16 19:59 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-03-31 10:12 ` Stefan Hajnoczi 2021-03-31 10:12 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 17/24] DAX/unmap virtiofsd: Add wrappers for VHOST_USER_SLAVE_FS_IO Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 14:18 ` Stefan Hajnoczi 2021-02-11 14:18 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 18/24] DAX/unmap virtiofsd: Parse unmappable elements Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 14:29 ` Stefan Hajnoczi 2021-02-11 14:29 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-25 10:19 ` Dr. David Alan Gilbert 2021-02-25 10:19 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-03-31 10:14 ` Stefan Hajnoczi 2021-03-31 10:14 ` [Virtio-fs] " Stefan Hajnoczi 2021-03-17 10:33 ` Dr. David Alan Gilbert 2021-03-17 10:33 ` [Virtio-fs] " Dr. David Alan Gilbert 2021-02-09 19:02 ` [PATCH 19/24] DAX/unmap virtiofsd: Route unmappable reads Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [PATCH 20/24] DAX/unmap virtiofsd: route unmappable write to slave command Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [PATCH 21/24] DAX:virtiofsd: implement FUSE_INIT map_alignment field Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [PATCH 22/24] vhost-user-fs: Extend VhostUserFSSlaveMsg to pass additional info Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [PATCH 23/24] vhost-user-fs: Implement drop CAP_FSETID functionality Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git) 2021-02-11 14:35 ` Stefan Hajnoczi 2021-02-11 14:35 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-11 14:40 ` Vivek Goyal 2021-02-11 14:40 ` [Virtio-fs] " Vivek Goyal 2021-02-15 15:57 ` Stefan Hajnoczi 2021-02-15 15:57 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-16 15:57 ` Vivek Goyal 2021-02-16 15:57 ` [Virtio-fs] " Vivek Goyal 2021-02-22 16:53 ` Stefan Hajnoczi 2021-02-22 16:53 ` [Virtio-fs] " Stefan Hajnoczi 2021-02-09 19:02 ` [PATCH 24/24] virtiofsd: Ask qemu to drop CAP_FSETID if client asked for it Dr. David Alan Gilbert (git) 2021-02-09 19:02 ` [Virtio-fs] " Dr. David Alan Gilbert (git)
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210209190224.62827-1-dgilbert@redhat.com \ --to=dgilbert@redhat.com \ --cc=marcandre.lureau@redhat.com \ --cc=mst@redhat.com \ --cc=qemu-devel@nongnu.org \ --cc=stefanha@redhat.com \ --cc=vgoyal@redhat.com \ --cc=virtio-fs@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.