Re: fs compression

[Tom Marshall <tom@cyngn.com>]

So I've been playing around a bit and I have a basic strategy laid out.
Please let me know if I'm on the right track.

Compressed file attributes
==========================

The filesystem is responsible for detecting whether a file is compressed and
hooking into the compression lib.  This may be done with an inode flag,
xattr, or any other applicable method.  No other special attributes are
necessary.

Compressed file format
======================

Compressed files shall have header, block map, and data sections.

Header:

byte[4]		magic		'zzzz' (not strictly needed)
byte		param1		method and flags
	bits 0..3 = compression method (1=zlib, 2=lz4, etc.)
	bits 4..7 = flags (none defined yet)
byte		blocksize	log2 of blocksize (max 31)
le48		orig_size	original uncompressed file size


Block map:

Vector of le16 (if blocksize <= 16) or le32 (if blocksize > 16).  Each entry
is the compressed size of the block.  Zero indicates that the block is
stored uncompressed, in case compression expanded the block.

Data:

Compressed data.

Compression library
===================

I'm just fleshing this out and trying to make it work.  The basic strategy
is to wrap the underlying readpage/readpages.  Right now I have the
following:

/*
 * For convenience.
 */
typedef int (*readpage_t)(struct page *);

/*
 * Indicate whether compression is enabled.  It may be desirable to disable
 * compression for test, backup, or maintenance activities.  Controlled by
 * a sysfs file.
 */
extern int zfile_enabled(void);

/*
 * Initialize internal data structures for a given inode.  This will
 * result in reading the file header and block map, so the inode must
 * be fully populated and ready to accept readpage requests.
 */
extern int zinode_init(struct inode *inode, readpage_t lower_readpage);

/*
 * Wrapper for filesystem's readpage.  Consults the block map and reads
 * the appropriate compressed pages for the requested block, decompresses
 * them, and populates the page with uncompressed data.
 */
extern int zinode_readpage(readpage_t lower_readpage,
		struct page *page);

/*
 * As above, but for multiple pages.
 */
extern int zinode_readpages(readpage_t lower_readpage,
		struct address_space *mapping,
		struct list_head *pages, unsigned nr_pages);

Questions and issues
====================

Should there be any padding for the data blocks?  For example, if writing is
to be supported, padding the compressed data to the filesystem block size
would allow for easy rewriting of individual blocks without disturbing the
surrounding blocks.  Perhaps padding could be indicated by a flag.

Note readpage_t does not take a file ptr, as it's not available at inode
creation time.  ext4 only uses it to lookup the inode ptr.  Is this true
across all filesystems, or do some actually require the file ptr in their
readpage?

The compression code must be able to read pages from the underlying
filesystem.  This involves using the pagecache.  But the uncompressed data
is what ultimately should end up in the pagecache.  This is where I'm
currently stuck.  How do I implement the code such that the underlying
compressed data may be read (using the pagecache or not) while not
disturbing the pagecache for the uncompressed data?  I'm wondering if I need
to create an internal address_space to pass down into the underlying
readpage?  Or is there another way to do this?