Re: fs compression

From: Tom Marshall <tom@cyngn.com>
To: Theodore Ts'o <tytso@mit.edu>
Cc: Jaegeuk Kim <jaegeuk@kernel.org>, linux-fsdevel@vger.kernel.org
Subject: Re: fs compression
Date: Wed, 20 May 2015 21:28:19 -0700	[thread overview]
Message-ID: <20150521042819.GA14709@eden.sea.cyngn.com> (raw)
In-Reply-To: <20150520224630.GA10927@eden.sea.cyngn.com>

On Wed, May 20, 2015 at 03:46:30PM -0700, Tom Marshall wrote:
> So I have this all working as described.  I haven't implemented readahead
> yet (readpages) so it's slow.  I'll be doing that next.
> 
> The other thing to note is that since the uncompressed size is stored inside
> the file data, stat() requires reading both the inode and the first page of
> the file.  That's not optimal, but I don't know if other generic out-of-band
> solutions (such as xattrs) would be any better.  I suppose it depends on
> whether the xattr info is read in with the inode or not.
> 
> Also, on the subject of file size, I'm currently swapping out the i_size for
> the compressed i_size before calling down into the filesystem's readpage. 
> Yes that's a nasty hack that I'll need to address.
> 
> On Wed, May 20, 2015 at 05:36:41PM -0400, Theodore Ts'o wrote:
> > On Wed, May 20, 2015 at 10:46:35AM -0700, Tom Marshall wrote:
> > > So I've been playing around a bit and I have a basic strategy laid out.
> > > Please let me know if I'm on the right track.
> > > 
> > > Compressed file attributes
> > > ==========================
> > > 
> > > The filesystem is responsible for detecting whether a file is compressed and
> > > hooking into the compression lib.  This may be done with an inode flag,
> > > xattr, or any other applicable method.  No other special attributes are
> > > necessary.
> > 
> > So I assume what you are implementing is read-only compression; that
> > is, once the file is written, and the attribute set indicating that
> > this is a compressed file, it is now immutable.
> 
> That is TBD.  Our use case right now only requires read-only, but I think
> read-write would be a nice thing if it's not too convoluted.
> 
> fallocate() is supported on the major filesystems, and I imagine the same
> mechanisms could be used to provide rewriting of the "compression clusters".

It just occurred to me that this could be the answer to both a consistent
i_size and providing read-write access.

If compression clusters were implemented by holes in the underlying file and
the header+blockmap were not stored in the file proper, then the compressed
and uncompressed file sizes would match.

Further, rewriting a compression cluster would be a simple matter of
adjusting the allocated compression cluster blocks as fallocate() provides.

I honestly don't have a clue how to storing the header+blockmap outside the
file yet, but if we could, that would seem to tie everything together quite
nicely.

Thoughts?