* Re: Readahead for compressed data
[not found] <YXHK5HrQpJu9oy8w@casper.infradead.org>
@ 2021-10-29 6:15 ` Coly Li
0 siblings, 0 replies; only message in thread
From: Coly Li @ 2021-10-29 6:15 UTC (permalink / raw)
To: Matthew Wilcox, linux-fsdevel, Jan Kara, Phillip Lougher,
linux-erofs, linux-btrfs, linux-ntfs-dev, ntfs3, linux-bcache,
Hsin-Yi Wang, David Howells
Cc: linux-bcachefs, Kent Overstreet
On 10/22/21 4:17 AM, Matthew Wilcox wrote:
> As far as I can tell, the following filesystems support compressed data:
>
> bcachefs, btrfs, erofs, ntfs, squashfs, zisofs
Hi Matthew,
There is a new bcachefs mailing list linux-bcachefs@vger.kernel.org for
bcachefs. I add it in Cc in this reply email.
Just FYI for you and other receivers.
Thanks.
Coly Li
>
> I'd like to make it easier and more efficient for filesystems to
> implement compressed data. There are a lot of approaches in use today,
> but none of them seem quite right to me. I'm going to lay out a few
> design considerations next and then propose a solution. Feel free to
> tell me I've got the constraints wrong, or suggest alternative solutions.
>
> When we call ->readahead from the VFS, the VFS has decided which pages
> are going to be the most useful to bring in, but it doesn't know how
> pages are bundled together into blocks. As I've learned from talking to
> Gao Xiang, sometimes the filesystem doesn't know either, so this isn't
> something we can teach the VFS.
>
> We (David) added readahead_expand() recently to let the filesystem
> opportunistically add pages to the page cache "around" the area requested
> by the VFS. That reduces the number of times the filesystem has to
> decompress the same block. But it can fail (due to memory allocation
> failures or pages already being present in the cache). So filesystems
> still have to implement some kind of fallback.
>
> For many (all?) compression algorithms (all?) the data must be mapped at
> all times. Calling kmap() and kunmap() would be an intolerable overhead.
> At the same time, we cannot write to a page in the page cache which is
> marked Uptodate. It might be mapped into userspace, or a read() be in
> progress against it. For writable filesystems, it might even be dirty!
> As far as I know, no compression algorithm supports "holes", implying
> that we must allocate memory which is then discarded.
>
> To me, this calls for a vmap() based approach. So I'm thinking
> something like ...
>
> void *readahead_get_block(struct readahead_control *ractl, loff_t start,
> size_t len);
> void readahead_put_block(struct readahead_control *ractl, void *addr,
> bool success);
>
> Once you've figured out which bytes this encrypted block expands to, you
> call readahead_get_block(), specifying the offset in the file and length
> and get back a pointer. When you're done decompressing that block of
> the file, you get rid of it again.
>
> It's the job of readahead_get_block() to allocate additional pages
> into the page cache or temporary pages. readahead_put_block() will
> mark page cache pages as Uptodate if 'success' is true, and unlock
> them. It'll free any temporary pages.
>
> Thoughts? Anyone want to be the guinea pig? ;-)
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2021-10-29 6:15 UTC | newest]
Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <YXHK5HrQpJu9oy8w@casper.infradead.org>
2021-10-29 6:15 ` Readahead for compressed data Coly Li
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).