Re: overlayfs: issue with a replaced lower squashfs with export-table

[Amir Goldstein <amir73il@gmail.com>]

On Wed, Jul 8, 2020 at 11:00 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> On Wed, Jul 8, 2020 at 8:55 AM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > On Wed, Jul 8, 2020 at 12:53 AM Vivek Goyal <vgoyal@redhat.com> wrote:
> > >
> > > On Tue, Jul 07, 2020 at 08:41:20PM +0300, Amir Goldstein wrote:
> > > > > > Miklos,
> > > > > >
> > > > > > At first glance I did not understand how changing lower file handles causes
> > > > > > failure to ovl_verify_inode().
> > > > > > To complete the picture, here is the explanation.
> > > > > >
> > > > > > Upper file A was copied up from lower file with inode 10 in old squashfs
> > > > > > and the "origin" file handle composed of the inode number 10 is recorded
> > > > > > in upper file A.
> > > > > >
> > > > > > With newly formatted lower, lower A has inode 11 and lower B has inode 10.
> > > > > > Upper file B is copied from lower file B with inode 10 in new squashfs and
> > > > > > the "origin" file handle composed of the inode number 10 is recorded
> > > > > > in upper file B.
> > > > > > Now we have two upper files with the same "origin" that are not hardlinks.
> > > > > >
> > > > > > On lookup of both overlay files A and B, ovl_check_origin() decodes lower
> > > > > > file B (inode 10) as the lower inode.
> > > > > > This lower inode is used to get the overlay inode number (10) and as
> > > > > > the key to hash overlay inode in inode cache.
> > > > > >
> > > > > > Suppose A is looked up first and it's inode is hashed.
> > > > > > Then B is looked up and in ovl_get_inode() it finds the inode hashed
> > > > > > by the same lower inode in inode cache, but fails ovl_verify_inode()
> > > > > > because:
> > > > > > d_inode(upperdentry) /* B */ != ovl_inode_upper(inode) /* A */
> > > > > >
> > > > > > This can also happen when copying overlay layers to a new
> > > > > > fs tree and carrying over the old "origin" xattr.
> > > > > > In practice, the UUID part of the stored "origin" xattr is meant to
> > > > > > protect against decoding lower fh when migrating to another
> > > > > > filesystem, but layers could be migrated inside the same filesystem.
> > > > > > Since squashfs does not have a UUID, re-creating sqhashfs is similar
> > > > > > to migrating layers inside the same filesystem.
>
> No, I think copying layers is a different issue: it's not possible to
> get the same file handle for A and B since they are
>
>  - either on different filesystems and uuid is different
>  - on the same filesystem, hence fh must be unique for the lifetime of the fs
>
> > Now, suggestions for work arounds:
> >
> > 1. Don't follow with lower null uuid (patch posted) - no caveats
>
> We could add some sort of "overlay version"  to origin to be able to
> trust null uuid, but only if created in *this* instance of overlayfs.
> That way we retain some of the advantages without any of the
> drawbacks.   This does not seem too complex theoretically, but not
> sure if it's worth it.

Don't think it is worth it.
It is a corner case of a special case.

>
> > 2. Opt-out of following origin with explicit option e.g. "index=nofollow"
> > 3. Don't follow origin unless one of the following opt-in features:
> >     metacopy,index,xino
>
> Maybe, if 1) is problematic for some reason.
>

1) is not problematic IMO and the simple patch I posted may be applied
for fixing the reported issue, but it only solved the special case of null uuid.
The problem still exists with re-creating lower on xfs/ext4, e.g. by
rm -rf and unpacking image tar.

So either we solve only the special case and wait for someone to complain
on the non-null uuid case or we try to make the code more robust.

Suggesting improvement for auto detecting re-created lower:
- On mount, even with index disabled, do ovl_verify_origin() of lower root
- On lookup of dir, do ovl_verify_origin() regardless of ovl_verify_lower()
- When ovl_verify_origin() on some dir fails and index/metacopy are disabled,
  warn and set ofs->bad_origin to disable ovl_check_origin() from here on
- Maybe also set null fh in ovl_set_origin() from here on?
- In ovl_dentry_*revalidate(), if non-dir has lower and ofs->bad_origin,
  invalidate the dentry
- Can also set ofs->bad_origin on ovl_verify_inode() for conflicting
  upper inode as reported in the bug

Thanks,
Amir.