Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

["Charles P. Wright" <cpwright@cpwright.com>]

On Thu, 2004-09-02 at 15:42, Linus Torvalds wrote:
> On Thu, 2 Sep 2004, Charles P. Wright wrote:
> >
> > Stackable file systems are a good fit for doing things like this.  They
> > layer on top of an existing directory and intercept all of the vfs_*
> > calls.  You can do lots of neat things like encryption versioning,
> > compression, and more.
> 
> Stackable filesystems are really hard, and ambiguous on where things go, 
> and what happens when you remove an entry that hides another one. Sure, 
> you can make up any amount of semantics, but the fact is, all the 
> semantics _are_ going to be pretty arbitrary, and fairly complex.
I think by stackable you are thinking what we call a Union.

By stackable, I mean that you just layer some functionality on an
existing file system, without having to modify it.

I agree that Unioning semantics are difficult.  We've actually
implemented a stackable unioning file system, and dealing with an
arbitrary number of branches, especially in the face of some operations
failing in the middle, is quite complex.  Rename is especially hard.

I think for the type of problem, where you have the file system jump
down into archives or do other like-minded operations, you will also end
up with having to make some arbitrary decisions with complex semantics.

> I'd frankly _much_ prefer not having stackable filesystems supported by 
> the VFS layer.

> That said, I think you can make stackable filesystems as a _separate_
> filesystem (aka "view" or whatever), which is slightly different from
> supporting them in the VFS layer itself: make it a filesystem of its own,
> rather than supporting it on a VFS layer as a merge of two different
> filesystems. That way you still have totally arbitrary semantics for how
> the stacking works, but at least it's not codified by the VFS layer - it's
> codified by nothing more special than one more random filesystem
> implementation, and you could in fact have _different_ semantics depending
> on what the user wants.
> 
> Same end result, different "mindset". I think this is what FiST actually 
> does, but I could be wrong.
Yes, to the VFS FiST behaves as a file system.  To the file system it is
stacked on, it behaves like the VFS.

> What's the difference? If the VFS layer supports it (like the VFS supports 
> regular "opaque" overmounting), you'd just do
> 
> 	mount -o stack /dir /otherdir
> 
> the same way you do "bind" mounts. In contrast, if you do it as a 
> third-party "view" filesystem, it would be more like
> 
> 	mount -t stackfs -o under=/usr,over=/usr/local /mnt
> 
> where /usr and /usr/local would be the same old underlying filesystems, 
> and you'd see a totally new filesystem at /mnt.
That is very close to how a FiST file system works.  However, I think
you are more closely describing a Union mount that would provide a
merged view of "/user" and "/usr/local" (with /usr/local taking
precedence over /user).

More concretely, something like:
mount -t gzipfs -o dir=/compressed none /mnt

Would transparently compress data written to /mnt/a, and physically
store it in /compressed/a.  When reading /mnt/a, the data is
decompressed and returned to the user-space.

This a real example that is actually included as part of FiST.

> Btw, this kind of non-VFS-supported filesystem would not necessarily have 
> coherency guarantees: if you changed /usr or /usr/local directly (rather 
> than through the stackable /mnt), you could confuse stackfs. Kind of the 
> same way you can confuse a NFSv2 client by operating directly on the 
> server (ie I'm not saying "confuse" in a "blow up the kernel" kind of 
> way, but in a "would return -ESTALE or similar").
With a simple stacking file system, one way to avoid this is an overlay
mount.

mount -t cryptfs -o dir=/home/cpwright none /home/cpwright

There are some disadvantages to hiding the lower level (e.g., with
Cryptfs you want to backup the encrypted data, not the decrypted data).

Charles