stat inconsistency with overlayfs

stat inconsistency with overlayfs

[Atom2]

I am using find with its printf "%D" option (provides the same 
information as stats device information "%d" - device number in decimal) 
to figure out whether a file system entry resides in the r/o lowerdir or 
the r/w upperdir of an overlayfs mounted filesystem. I distinguish 
between the two by getting the device number from a (plain) file know to 
be in the upperdir.

The use case behind that is to be able to backup only files from the 
upperdir for several systems sharing a common lowerdir filesystem. I 
have used that (scripted approach via rsync) now for quiet some time and 
a few kernels back and it seemed to have worked very well.

Currently I am using kernel 3.17.7 on gentoo and I seem to observe a 
strange behaviour (which I do not recall to have seen before on 3.13 and 
3.11) with my approach as follows:

.) plain files still work and the device number is correct
.) directories, however, always seem to reside in the lowerdir - even 
thoguh they do not exist there; in fact there's not a single file in the 
whole filesystem hierarchy that, according to stat/find, seems to reside 
in the upperdir:

please see the stat output for a file and a directory, both residing in 
the same (parent) directory which is completely located in the upperdir 
(and does not at all exist in the lowerdir):
# stat serial
   File: ‘serial’
   Size: 17              Blocks: 8          IO Block: 4096   regular file
Device: ca03h/51715d    Inode: 88          Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-02-19 19:01:56.278161346 +0100
Modify: 2014-02-19 19:01:56.278161346 +0100
Change: 2014-02-19 19:01:56.278161346 +0100
  Birth: -
#
# stat certs/
   File: ‘certs/’
   Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: dh/13d  Inode: 331140      Links: 2
Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-04-26 14:08:47.337968562 +0200
Modify: 2014-02-19 18:55:58.458161346 +0100
Change: 2014-02-19 18:55:58.458161346 +0100
  Birth: -

For comparision, please see the stat of /bin which only resides in the 
lowerdir and does not exist in the upperdir:
# stat /bin
   File: ‘/bin’
   Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: dh/13d  Inode: 401777      Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-02-20 17:50:24.066368607 +0100
Modify: 2015-02-09 17:51:18.000000000 +0100
Change: 2015-02-09 23:58:06.011825328 +0100
  Birth: -

I do not think that this is the expected behaviour and I am pretty 
confident that this was different on older kernels - or am I missing 
anything/doing anything wrong here?

Thanks and regards Atom2
--
To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: stat inconsistency with overlayfs

[hujianyang]

Hi Atom2,

I didn't notice this before, but I had reproduced the situation as you
described. I'm not sure if it is a real problem.

The performing of 'stat' in Overlayfs are different between files and
directories. Files directly call upper/lower getattr function but
directories will set @dev and @ino by Overlayfs superblock itself.

See line 135 in fs/overlayfs/dir.c

"""
static int ovl_dir_getattr(struct vfsmount *mnt, struct dentry *dentry,
                         struct kstat *stat)
{
        int err;
        enum ovl_path_type type;
        struct path realpath;

        type = ovl_path_real(dentry, &realpath);
        err = vfs_getattr(&realpath, stat);
        if (err)
                return err;

        stat->dev = dentry->d_sb->s_dev;        // I think it's the cause.
        stat->ino = dentry->d_inode->i_ino;

        /*
         * It's probably not worth it to count subdirs to get the
         * correct link count.  nlink=1 seems to pacify 'find' and
         * other utilities.
         */
        if (OVL_TYPE_MERGE(type))
                stat->nlink = 1;

        return 0;
}
"""

I don't have the Overlayfs code on 3.11 or 3.13. I've lookup the v11
code in Miklos's git tree:

git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git

and found the code had been changed since the earliest code I could
get. So I don't know the reason of this behavior. I guess it is used
for filesystem consistence: directories in the same superblock have
same device number?

On 2015/2/21 2:15, Atom2 wrote:
> I am using find with its printf "%D" option (provides the same information as stats device information "%d" - device number in decimal) to figure out whether a file system entry resides in the r/o lowerdir or the r/w upperdir of an overlayfs mounted filesystem. I distinguish between the two by getting the device number from a (plain) file know to be in the upperdir.
> 
> The use case behind that is to be able to backup only files from the upperdir for several systems sharing a common lowerdir filesystem. I have used that (scripted approach via rsync) now for quiet some time and a few kernels back and it seemed to have worked very well.
> 

I think your requirement need to be reconsidering. Maybe we could keep
the @dev for a lower-only directory?

Add Cc Miklos.

Thanks,
Hu

> Currently I am using kernel 3.17.7 on gentoo and I seem to observe a strange behaviour (which I do not recall to have seen before on 3.13 and 3.11) with my approach as follows:
> 
> .) plain files still work and the device number is correct
> .) directories, however, always seem to reside in the lowerdir - even thoguh they do not exist there; in fact there's not a single file in the whole filesystem hierarchy that, according to stat/find, seems to reside in the upperdir:
> 
> please see the stat output for a file and a directory, both residing in the same (parent) directory which is completely located in the upperdir (and does not at all exist in the lowerdir):
> # stat serial
>   File: ‘serial’
>   Size: 17              Blocks: 8          IO Block: 4096   regular file
> Device: ca03h/51715d    Inode: 88          Links: 1
> Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2014-02-19 19:01:56.278161346 +0100
> Modify: 2014-02-19 19:01:56.278161346 +0100
> Change: 2014-02-19 19:01:56.278161346 +0100
>  Birth: -
> #
> # stat certs/
>   File: ‘certs/’
>   Size: 4096            Blocks: 8          IO Block: 4096   directory
> Device: dh/13d  Inode: 331140      Links: 2
> Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2014-04-26 14:08:47.337968562 +0200
> Modify: 2014-02-19 18:55:58.458161346 +0100
> Change: 2014-02-19 18:55:58.458161346 +0100
>  Birth: -
> 
> For comparision, please see the stat of /bin which only resides in the lowerdir and does not exist in the upperdir:
> # stat /bin
>   File: ‘/bin’
>   Size: 4096            Blocks: 8          IO Block: 4096   directory
> Device: dh/13d  Inode: 401777      Links: 2
> Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2015-02-20 17:50:24.066368607 +0100
> Modify: 2015-02-09 17:51:18.000000000 +0100
> Change: 2015-02-09 23:58:06.011825328 +0100
>  Birth: -
> 
> I do not think that this is the expected behaviour and I am pretty confident that this was different on older kernels - or am I missing anything/doing anything wrong here?
> 
> Thanks and regards Atom2
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> .
> 


--
To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: stat inconsistency with overlayfs

[Xu Wang]

Hi, Hu, Atom2,

> I don't have the Overlayfs code on 3.11 or 3.13. I've lookup the v11
> code in Miklos's git tree:
Fortunately I got the overlay fs code of kernel V3.18, which is mostly
same to v11 code in Miklos's git.

135 static int ovl_dir_getattr(struct vfsmount *mnt, struct dentry *dentry,
136              struct kstat *stat)
137 {
138     int err;
139     enum ovl_path_type type;
140     struct path realpath;
141 
142     type = ovl_path_real(dentry, &realpath);
143     err = vfs_getattr(&realpath, stat);
144     if (err)
145         return err;
146 
147     stat->dev = dentry->d_sb->s_dev;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here set dev number as Hu mentioned
148     stat->ino = dentry->d_inode->i_ino;
149 
150     /*
151      * It's probably not worth it to count subdirs to get the
152      * correct link count.  nlink=1 seems to pacify 'find' and
153      * other utilities.
154      */
155     if (type == OVL_PATH_MERGE)
156         stat->nlink = 1;
157 
158     return 0;
159 }

And the d_sb->s_dev is coming from the super_block of overlayfs, which is fake.
The super block s_dev is initialized by the call trace 
"mount_nodev->set_anon_super-->get_anon_bdev".

Either the dir exists in lower or exists in upper, the overlay fs fakes it exits in
overlay, which holds the fake device number.

Thanks,

George
-- 
George Wang 王旭

Kernel Quantity Engineer
Red Hat Software (Beijing) Co.,Ltd
IRC:xuw
Tel:+86-010-62608041
Phone:15901231579
9/F, Tower C, Raycom
--
To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: stat inconsistency with overlayfs

[Atom2]

Hi Hu,
thanks for looking into this.

Am 26.02.15 um 06:25 schrieb hujianyang:
> Hi Atom2,
>
> I didn't notice this before, but I had reproduced the situation as you
> described. I'm not sure if it is a real problem.
My take on this is that stat, under all circumstances, should reliably 
provide information about the device the queried entry resides on. In 
case the entry only exists on the upperdir-device, overlayfs should not 
fool the user that it instead resides in the r/o lowerdir when in fact 
no such entry exists there.

In my view that's the only viable option to back-up (empty) directories 
(from a running system) which only reside in the upperdir provided that 
no backup is requested for the r/o (lower) file layer.
>
> The performing of 'stat' in Overlayfs are different between files and
> directories. Files directly call upper/lower getattr function but
> directories will set @dev and @ino by Overlayfs superblock itself.
In my view that then seems to defeat the purpose of the device field for 
the stat sys-calls for overlayfs.
>
> See line 135 in fs/overlayfs/dir.c
>
> """
> static int ovl_dir_getattr(struct vfsmount *mnt, struct dentry *dentry,
>                           struct kstat *stat)
> {
>          int err;
>          enum ovl_path_type type;
>          struct path realpath;
>
>          type = ovl_path_real(dentry, &realpath);
>          err = vfs_getattr(&realpath, stat);
>          if (err)
>                  return err;
>
>          stat->dev = dentry->d_sb->s_dev;        // I think it's the cause.
>          stat->ino = dentry->d_inode->i_ino;
>
>          /*
>           * It's probably not worth it to count subdirs to get the
>           * correct link count.  nlink=1 seems to pacify 'find' and
>           * other utilities.
>           */
>          if (OVL_TYPE_MERGE(type))
>                  stat->nlink = 1;
>
>          return 0;
> }
> """
>
> I don't have the Overlayfs code on 3.11 or 3.13. I've lookup the v11
> code in Miklos's git tree:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git
>
> and found the code had been changed since the earliest code I could
> get. So I don't know the reason of this behavior. I guess it is used
> for filesystem consistence: directories in the same superblock have
> same device number?
Following that argument, why would that then be different for plain 
files? In my understanding a directory in linux is not really different 
from a file (both are just an abstraction of blocks from the underlying 
block device) with the main difference being that the former's content 
is interpreted by the file system code to allow traversal of 
directories. So all operations that can be applied likewise to both 
should provide consistent results between files and directories.

Probably there's a reason behind that difference, but it currently is 
beyond me.

Thanks and regards Atom2
>
> On 2015/2/21 2:15, Atom2 wrote:
>> I am using find with its printf "%D" option (provides the same information as stats device information "%d" - device number in decimal) to figure out whether a file system entry resides in the r/o lowerdir or the r/w upperdir of an overlayfs mounted filesystem. I distinguish between the two by getting the device number from a (plain) file know to be in the upperdir.
>>
>> The use case behind that is to be able to backup only files from the upperdir for several systems sharing a common lowerdir filesystem. I have used that (scripted approach via rsync) now for quiet some time and a few kernels back and it seemed to have worked very well.
>>
>
> I think your requirement need to be reconsidering. Maybe we could keep
> the @dev for a lower-only directory?
>
> Add Cc Miklos.
>
> Thanks,
> Hu
>
>> Currently I am using kernel 3.17.7 on gentoo and I seem to observe a strange behaviour (which I do not recall to have seen before on 3.13 and 3.11) with my approach as follows:
>>
>> .) plain files still work and the device number is correct
>> .) directories, however, always seem to reside in the lowerdir - even thoguh they do not exist there; in fact there's not a single file in the whole filesystem hierarchy that, according to stat/find, seems to reside in the upperdir:
>>
>> please see the stat output for a file and a directory, both residing in the same (parent) directory which is completely located in the upperdir (and does not at all exist in the lowerdir):
>> # stat serial
>>    File: ‘serial’
>>    Size: 17              Blocks: 8          IO Block: 4096   regular file
>> Device: ca03h/51715d    Inode: 88          Links: 1
>> Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
>> Access: 2014-02-19 19:01:56.278161346 +0100
>> Modify: 2014-02-19 19:01:56.278161346 +0100
>> Change: 2014-02-19 19:01:56.278161346 +0100
>>   Birth: -
>> #
>> # stat certs/
>>    File: ‘certs/’
>>    Size: 4096            Blocks: 8          IO Block: 4096   directory
>> Device: dh/13d  Inode: 331140      Links: 2
>> Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
>> Access: 2014-04-26 14:08:47.337968562 +0200
>> Modify: 2014-02-19 18:55:58.458161346 +0100
>> Change: 2014-02-19 18:55:58.458161346 +0100
>>   Birth: -
>>
>> For comparision, please see the stat of /bin which only resides in the lowerdir and does not exist in the upperdir:
>> # stat /bin
>>    File: ‘/bin’
>>    Size: 4096            Blocks: 8          IO Block: 4096   directory
>> Device: dh/13d  Inode: 401777      Links: 2
>> Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
>> Access: 2015-02-20 17:50:24.066368607 +0100
>> Modify: 2015-02-09 17:51:18.000000000 +0100
>> Change: 2015-02-09 23:58:06.011825328 +0100
>>   Birth: -
>>
>> I do not think that this is the expected behaviour and I am pretty confident that this was different on older kernels - or am I missing anything/doing anything wrong here?
--
To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: stat inconsistency with overlayfs

[Atom2]

Hi Xu,
thanks for your follow-up.
Am 26.02.15 um 07:45 schrieb Xu Wang:
> Hi, Hu, Atom2,
>
>> I don't have the Overlayfs code on 3.11 or 3.13. I've lookup the v11
>> code in Miklos's git tree:
> Fortunately I got the overlay fs code of kernel V3.18, which is mostly
> same to v11 code in Miklos's git.
>
> 135 static int ovl_dir_getattr(struct vfsmount *mnt, struct dentry *dentry,
> 136              struct kstat *stat)
> 137 {
> 138     int err;
> 139     enum ovl_path_type type;
> 140     struct path realpath;
> 141
> 142     type = ovl_path_real(dentry, &realpath);
> 143     err = vfs_getattr(&realpath, stat);
> 144     if (err)
> 145         return err;
> 146
> 147     stat->dev = dentry->d_sb->s_dev;
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here set dev number as Hu mentioned
> 148     stat->ino = dentry->d_inode->i_ino;
> 149
> 150     /*
> 151      * It's probably not worth it to count subdirs to get the
> 152      * correct link count.  nlink=1 seems to pacify 'find' and
> 153      * other utilities.
> 154      */
> 155     if (type == OVL_PATH_MERGE)
> 156         stat->nlink = 1;
> 157
> 158     return 0;
> 159 }
>
> And the d_sb->s_dev is coming from the super_block of overlayfs, which is fake.
> The super block s_dev is initialized by the call trace
> "mount_nodev->set_anon_super-->get_anon_bdev".
>
> Either the dir exists in lower or exists in upper, the overlay fs fakes it exits in
> overlay, which holds the fake device number.
I see what you are saying and code-wise that seems to be the cause of 
the behaviour I am seeing. I am, however, still not convinced at all 
that this is the correct/to be expected behaviour.

Giving this some further thought, might this not create more issues 
further down the line: In my (granted: limited) understanding the 
device/i-number combination must be unique across all mounted 
file-systems. Now does this also mean that the i-number is faked as 
well? In other words what is going to happen if the i-number of the 
directory residing in the upperdir (but wrongly attributed to the 
lowerdir) is identical to an i-number of another existing entry in the 
lowerdir?

Finally, stat, as I understand it, amongst other pieces of information 
also provides information about the device an entry resides on and that 
information, IMHO, simply is plain wrong for directories only existing 
on the upperdir.

Thanks Atom2

Re: stat inconsistency with overlayfs

[Miklos Szeredi]

Atom2,

> The use case behind that is to be able to backup only files from the
> upperdir for several systems sharing a common lowerdir filesystem. I have
> used that (scripted approach via rsync) now for quiet some time and a few
> kernels back and it seemed to have worked very well.

Why don't you just back up the upper directory itself instead of
messing around with device numbers?

Thanks,
Miklos

Re: stat inconsistency with overlayfs

[Atom2]

Mikolos,
thanks for joining the party.

Am 03.03.15 um 16:14 schrieb Miklos Szeredi:
> Atom2,
>
>> The use case behind that is to be able to backup only files from the
>> upperdir for several systems sharing a common lowerdir filesystem. I have
>> used that (scripted approach via rsync) now for quiet some time and a few
>> kernels back and it seemed to have worked very well.
>
> Why don't you just back up the upper directory itself instead of
> messing around with device numbers?

I am not aware of any such option, but if you could explain that a 
little bit more in detail, I am happy to explore other options.

Just for everybody to be on the same page: In my use case the r/o 
lowerdir's root directory is also the root of my file system (which is 
shared across many systems) from which (all) the systems boot. Using an 
initramfs during the boot process, the upperdir file system (which is 
different per system) is then r/w overlay-mounted over the common r/o 
lowerdir's root directory in order to catch any r/w access to the file 
system.

So in essence and according to my understanding there's no direct access 
to (only) the r/w upperdir (or only the lowerdir) from within any of the 
running systems other than using the device field to see where a file 
system entry actually exists - at least that's my current understanding.

But again, I am happy to learn ...

Regards Atom2

Re: stat inconsistency with overlayfs

[hujianyang]

On 2015/3/4 1:12, Atom2 wrote:
> Mikolos,
> thanks for joining the party.
> 
> Am 03.03.15 um 16:14 schrieb Miklos Szeredi:
>> Atom2,
>>
>>> The use case behind that is to be able to backup only files from the
>>> upperdir for several systems sharing a common lowerdir filesystem. I have
>>> used that (scripted approach via rsync) now for quiet some time and a few
>>> kernels back and it seemed to have worked very well.
>>
>> Why don't you just back up the upper directory itself instead of
>> messing around with device numbers?
> 
> I am not aware of any such option, but if you could explain that a little bit more in detail, I am happy to explore other options.
> 
> Just for everybody to be on the same page: In my use case the r/o lowerdir's root directory is also the root of my file system (which is shared across many systems) from which (all) the systems boot. Using an initramfs during the boot process, the upperdir file system (which is different per system) is then r/w overlay-mounted over the common r/o lowerdir's root directory in order to catch any r/w access to the file system.
> 
> So in essence and according to my understanding there's no direct access to (only) the r/w upperdir (or only the lowerdir) from within any of the running systems other than using the device field to see where a file system entry actually exists - at least that's my current understanding.
> 
> But again, I am happy to learn ...
> 
> Regards Atom2
> 
> 

Seems not all upper directories are accessible.

Miklos, I didn't find the change log about device number changing, Maybe
there are some reasons?

For Atom2's requirement, I have some thoughts:

The device number of Overlayfs are now different between files and
directories, My plan is showing Overlayfs device number for upper and
upper-merged(upper and several lower) file/dir, using

"stat->dev = dentry->d_sb->s_dev"

and showing lowerfs device number for a lower only file/dir.

For upper non-directory files, report Overlayfs device number.
Do not change the stat of a lower non-directory file.

For upper and upper-merged directories, report Overlayfs device number as
before. But for lower only directories, report lowerfs device number.

After this, Atom2 could backup the files and directories have a device
number equal to Overlayfs mount device number. But as Xu mentioned, this
device number is just fake, so I don't know if there are other problems.

And for multi-lower mount, an Overlayfs directory may mixed up by several
lower directories, Atom2 seems don't want to backup this kind of directories,
we couldn't set the device number to Overlayfs device number, maybe we
could use the left most lowerdir device number, but that may confused with
the lower only directories.

Thanks,
Hu

Re: stat inconsistency with overlayfs

[Miklos Szeredi]

On Tue, Mar 3, 2015 at 6:12 PM, Atom2 <ariel.atom2@web2web.at> wrote:
> Mikolos,
> thanks for joining the party.
>
> Am 03.03.15 um 16:14 schrieb Miklos Szeredi:
>
>> Atom2,
>>
>>> The use case behind that is to be able to backup only files from the
>>> upperdir for several systems sharing a common lowerdir filesystem. I have
>>> used that (scripted approach via rsync) now for quiet some time and a few
>>> kernels back and it seemed to have worked very well.
>>
>>
>> Why don't you just back up the upper directory itself instead of
>> messing around with device numbers?
>
>
> I am not aware of any such option, but if you could explain that a little
> bit more in detail, I am happy to explore other options.
>
> Just for everybody to be on the same page: In my use case the r/o lowerdir's
> root directory is also the root of my file system (which is shared across
> many systems) from which (all) the systems boot. Using an initramfs during
> the boot process, the upperdir file system (which is different per system)
> is then r/w overlay-mounted over the common r/o lowerdir's root directory in
> order to catch any r/w access to the file system.
>
> So in essence and according to my understanding there's no direct access to
> (only) the r/w upperdir (or only the lowerdir) from within any of the
> running systems other than using the device field to see where a file system
> entry actually exists - at least that's my current understanding.

Having access to the upperdir is just your own choice.

Without knowing the details of how you set up the mounts I can't give
you a concrete example, but see for example the pivot_root(8) man page
on how to get access to your old mountpoint.  You can also use bind
mounts to clone the upperdir into the overlay namespace.

Thanks,
Miklos