From: David Howells <dhowells@redhat.com> To: Jeffle Xu <jefflexu@linux.alibaba.com> Cc: linux-kernel@vger.kernel.org, dhowells@redhat.com, joseph.qi@linux.alibaba.com, linux-cachefs@redhat.com, linux-fsdevel@vger.kernel.org, gerry@linux.alibaba.com, linux-erofs@lists.ozlabs.org Subject: Re: [PATCH v2 00/20] fscache, erofs: fscache-based demand-read semantics Date: Tue, 25 Jan 2022 16:15:30 +0000 [thread overview] Message-ID: <2815558.1643127330@warthog.procyon.org.uk> (raw) In-Reply-To: <20220118131216.85338-1-jefflexu@linux.alibaba.com> Jeffle Xu <jefflexu@linux.alibaba.com> wrote: > The following issues still need further discussion. Thanks for your time > and patience. > > 1. I noticed that there's refactoring of netfs library[1], > ... > [1] https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=netfs-lib Yes. I'm working towards getting netfslib to do handling writes and dio as well as reads, along with content crypto/compression, and the idea I'm aiming towards is that you just point your address_space_ops at netfs directly if possible - but it's going to require its own context now to manage pending writes. See my netfs-experimental branch for more of that - it's still a work in progress, though. Btw, you could set rreq->netfs_priv in ->init_rreq() rather than passing it in to netfs_readpage(). > 2. The current implementation will severely conflict with the > refactoring of netfs library[1][2]. The assumption of 'struct > netfs_i_context' [2] is that, every file in the upper netfs will > correspond to only one backing file. While in our scenario, one file in > erofs can correspond to multiple backing files. That is, the content of > one file can be divided into multiple chunks, and are distrubuted over > multiple blob files, i.e. multiple backing files. Currently I have no > good idea solving this conflic. I can think of a couple of options to explore: (1) Duplicate the cachefiles backend. You can discard a lot of it, since a much of it is concerned with managing local modifications - which you're not going to do since you have a R/O filesystem and you're looking at importing files into the cache externally to the kernel. I would suggest looking to see if you can do the blob mapping in the backend rather than passing the offset down. Maybe make the cookie index key hold the index too, e.g. "/path/to/file+offset". Btw, do you still need cachefilesd for its culling duties? (2) Do you actually need to go through netfslib? Might it be easier to call fscache_read() directly? Have a look at fs/nfs/fscache.c > Besides there are still two quetions: > - What's the plan of [1]? When is it planned to be merged? Hopefully next merge window, but that's going to depend on a number of things. > - It seems that all upper fs using fscache is going to use netfs API, > while the APIs like fscache_read_or_alloc_page() are deprecated. Is > that true? fscache_read_or_alloc_page() is gone completely. You don't have to use the netfs API. You can talk to fscache directly, doing DIO from the cache to an xarray-class iov_iter constructed from your inode's pagecache. netfslib provides/will provide a number of services, such as multipage folios, transparent caching, crypto, compression and hiding the existence of pages/folios from the filesystem as entirely as possible. However, you already have some of these implemented on top of iomap for the blockdev interface, it would appear. David
WARNING: multiple messages have this Message-ID (diff)
From: David Howells <dhowells@redhat.com> To: Jeffle Xu <jefflexu@linux.alibaba.com> Cc: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org, linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 00/20] fscache,erofs: fscache-based demand-read semantics Date: Tue, 25 Jan 2022 16:15:30 +0000 [thread overview] Message-ID: <2815558.1643127330@warthog.procyon.org.uk> (raw) In-Reply-To: <20220118131216.85338-1-jefflexu@linux.alibaba.com> Jeffle Xu <jefflexu@linux.alibaba.com> wrote: > The following issues still need further discussion. Thanks for your time > and patience. > > 1. I noticed that there's refactoring of netfs library[1], > ... > [1] https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=netfs-lib Yes. I'm working towards getting netfslib to do handling writes and dio as well as reads, along with content crypto/compression, and the idea I'm aiming towards is that you just point your address_space_ops at netfs directly if possible - but it's going to require its own context now to manage pending writes. See my netfs-experimental branch for more of that - it's still a work in progress, though. Btw, you could set rreq->netfs_priv in ->init_rreq() rather than passing it in to netfs_readpage(). > 2. The current implementation will severely conflict with the > refactoring of netfs library[1][2]. The assumption of 'struct > netfs_i_context' [2] is that, every file in the upper netfs will > correspond to only one backing file. While in our scenario, one file in > erofs can correspond to multiple backing files. That is, the content of > one file can be divided into multiple chunks, and are distrubuted over > multiple blob files, i.e. multiple backing files. Currently I have no > good idea solving this conflic. I can think of a couple of options to explore: (1) Duplicate the cachefiles backend. You can discard a lot of it, since a much of it is concerned with managing local modifications - which you're not going to do since you have a R/O filesystem and you're looking at importing files into the cache externally to the kernel. I would suggest looking to see if you can do the blob mapping in the backend rather than passing the offset down. Maybe make the cookie index key hold the index too, e.g. "/path/to/file+offset". Btw, do you still need cachefilesd for its culling duties? (2) Do you actually need to go through netfslib? Might it be easier to call fscache_read() directly? Have a look at fs/nfs/fscache.c > Besides there are still two quetions: > - What's the plan of [1]? When is it planned to be merged? Hopefully next merge window, but that's going to depend on a number of things. > - It seems that all upper fs using fscache is going to use netfs API, > while the APIs like fscache_read_or_alloc_page() are deprecated. Is > that true? fscache_read_or_alloc_page() is gone completely. You don't have to use the netfs API. You can talk to fscache directly, doing DIO from the cache to an xarray-class iov_iter constructed from your inode's pagecache. netfslib provides/will provide a number of services, such as multipage folios, transparent caching, crypto, compression and hiding the existence of pages/folios from the filesystem as entirely as possible. However, you already have some of these implemented on top of iomap for the blockdev interface, it would appear. David
next prev parent reply other threads:[~2022-01-25 16:16 UTC|newest] Thread overview: 68+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-01-18 13:11 [PATCH v2 00/20] fscache,erofs: fscache-based demand-read semantics Jeffle Xu 2022-01-18 13:11 ` Jeffle Xu 2022-01-18 13:11 ` [PATCH v2 01/20] netfs: make @file optional in netfs_alloc_read_request() Jeffle Xu 2022-01-18 13:11 ` Jeffle Xu 2022-01-18 13:11 ` [PATCH v2 02/20] netfs,cachefiles: manage logical/physical offset separately Jeffle Xu 2022-01-18 13:11 ` [PATCH v2 02/20] netfs, cachefiles: " Jeffle Xu 2022-01-18 13:11 ` [PATCH v2 03/20] netfs,fscache: support on-demand reading Jeffle Xu 2022-01-18 13:11 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 04/20] cachefiles: extract generic daemon write function Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 05/20] cachefiles: detect backing file size in on-demand read mode Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 06/20] cachefiles: introduce new devnode for " Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 07/20] erofs: use meta buffers for erofs_read_superblock() Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 08/20] erofs: export erofs_map_blocks() Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 09/20] erofs: add mode checking helper Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 10/20] erofs: register global fscache volume Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 11/20] erofs: add cookie context helper functions Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 12/20] erofs: add anonymous inode managing page cache of blob file Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 13/20] erofs: register cookie context for bootstrap blob Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 14/20] erofs: implement fscache-based metadata read Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 15/20] erofs: implement fscache-based data read for non-inline layout Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 16/20] erofs: implement fscache-based data read for inline layout Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 17/20] erofs: register cookie context for data blobs Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 18/20] erofs: implement fscache-based data read " Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 19/20] erofs: add 'uuid' mount option Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-18 13:12 ` [PATCH v2 20/20] erofs: support on-demand reading Jeffle Xu 2022-01-18 13:12 ` Jeffle Xu 2022-01-19 6:40 ` [Linux-cachefs] [PATCH v2 00/20] fscache, erofs: fscache-based demand-read semantics Gao Xiang 2022-01-19 6:40 ` Gao Xiang 2022-01-21 10:57 ` JeffleXu 2022-01-21 10:57 ` JeffleXu 2022-01-24 17:23 ` [PATCH v2 00/20] fscache,erofs: " David Howells 2022-01-24 17:23 ` [PATCH v2 00/20] fscache, erofs: " David Howells 2022-01-25 1:53 ` [PATCH v2 00/20] fscache,erofs: " JeffleXu 2022-01-25 1:53 ` JeffleXu 2022-01-25 2:55 ` JeffleXu 2022-01-25 2:55 ` JeffleXu 2022-01-25 15:34 ` [PATCH v2 11/20] erofs: add cookie context helper functions David Howells 2022-01-25 15:34 ` David Howells 2022-01-26 6:45 ` JeffleXu 2022-01-26 6:45 ` JeffleXu 2022-01-25 16:15 ` David Howells [this message] 2022-01-25 16:15 ` [PATCH v2 00/20] fscache,erofs: fscache-based demand-read semantics David Howells 2022-01-26 6:10 ` JeffleXu 2022-01-26 6:10 ` JeffleXu 2022-01-26 8:51 ` David Howells 2022-01-26 8:51 ` [PATCH v2 00/20] fscache, erofs: " David Howells 2022-01-27 7:07 ` [PATCH v2 00/20] fscache,erofs: " JeffleXu 2022-01-27 7:07 ` JeffleXu 2022-01-25 20:27 ` David Howells 2022-01-25 20:27 ` [PATCH v2 00/20] fscache, erofs: " David Howells 2022-01-26 5:26 ` [PATCH v2 00/20] fscache,erofs: " JeffleXu 2022-01-26 5:26 ` JeffleXu
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=2815558.1643127330@warthog.procyon.org.uk \ --to=dhowells@redhat.com \ --cc=gerry@linux.alibaba.com \ --cc=jefflexu@linux.alibaba.com \ --cc=joseph.qi@linux.alibaba.com \ --cc=linux-cachefs@redhat.com \ --cc=linux-erofs@lists.ozlabs.org \ --cc=linux-fsdevel@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.