Re: partially encrypted filesystem

[Phillip Lougher <phillip@lougher.demon.co.uk>]

Matthew Wilcox wrote:
> On Fri, Dec 05, 2003 at 02:47:56PM -0500, Erez Zadok wrote:
> 
>>Thanks for the info, Matthew.  Yes, clearly a scheme that keeps some "holes"
>>in compressed files can help; one of our ideas was to leave sparse holes
>>every N blocks, exactly for this kind of expansion, and to update the index
>>file's format to record where the spaces are (so we can efficiently
>>calculate how many holes we need to consume upon a new write).

FYI, Acorn's scheme was described in "Compressed Executables: An 
Exercise in Thinking Small" by Mark Taunton, in the Usenix Spring '91 
conference, it doesn't seem to be online, but a search on google groups 
for "group:comp.unix.internals taunton compressed executables" brings up 
a description.  I used to work with Mark Taunton at Acorn.

> 
> But the genius is that you don't need to calculate anything.  If the
> data block turns out to be incompressible (those damn .tar.bz2s!), you
> just write the block in-place.  If it is compressible, you write as much
> into that block's entry as you need and leave a gap.  The underlying
> file system doesn't write any data there.  There's no need for an index
> file -- you know exactly where to start reading each block.
> 

Of course this is all being done at the file level, which relies on 
proper support of holes in the underlying filesystem (which Acorn's BSD 
FFS filesystem did).  FiST's scheme is much more how it would be 
implemented without hole support, where you *have* to pack the data, 
otherwise the "unused" space would physically consume disk blocks. In 
this case an index to find the start of each compressed block is essential.

I'm guessing that FiST lacks support for holes or data insertion in the 
  filesystem model, which explains why on writing to the middle of a 
file, the entire file from that point has to be re-written.

Of course, all this is at the logical file level, and ignores the 
physical blocks on disk.  All filesystems assume physical data blocks 
can be updated in place.  With compression it is possible a new physical 
block has to be found, especially if blocks are highly packed and not 
aligned to block boundaries.  I expect this is at least partially why 
JFFS2 is a log structured filesystem.

Phillip