btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Chris Murphy]

Hi,

The primary problem is Bolt (Thunderbolt 3) tests that are
experiencing a regression when run in a container using overlayfs,
failing at:

Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty:
'empty' should be FALSE

https://gitlab.freedesktop.org/bolt/bolt/-/issues/171#note_872119

I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
approximately the same time I see one, sometimes more, kernel
messages:

[ 6295.379283] overlayfs: upper fs does not support xattr, falling
back to index=off and metacopy=off.

But I don't know if that kernel message relates to the bolt test failure.

If I run the test outside of a container, it doesn't fail. If I run
the test in a podman container using the btrfs driver instead of the
overlay driver, it doesn't fail. So it seems like this is an overlayfs
bug, but could be some kind of overlayfs+btrfs interaction.

Could this be related and just not yet merged?
https://lore.kernel.org/linux-unionfs/20210309162654.243184-1-amir73il@gmail.com/

Thanks,

-- 
Chris Murphy

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Amir Goldstein]

On Sat, Apr 10, 2021 at 1:04 AM Chris Murphy <lists@colorremedies.com> wrote:
>
> Hi,
>
> The primary problem is Bolt (Thunderbolt 3) tests that are
> experiencing a regression when run in a container using overlayfs,
> failing at:
>
> Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty:
> 'empty' should be FALSE
>
> https://gitlab.freedesktop.org/bolt/bolt/-/issues/171#note_872119
>

To summarize, the test case is:
- create empty dir
- open empty dir
- getdents => (".", "..")
- create file at (dirfd, "a",
- lseek to offset 0 on dirfd
- getdents => (".", "..") FAIL to see "a"

It looks like a bug in ovl readdir cache invalidation only there is
not supposed to be any caching of pure upper dir.

Once thing I noticed is that ovl_dentry_version_inc() is inconsistent
with ovl_dir_is_real() - the latter checks whether readdir caching would
be used and the former checks whether invalidating readdir cache is
needed. We need to change ovl_dentry_version_inc() test to:
    if (ovl_test_flag(OVL_WHITEOUTS, dir) || impurity)
Or better yet:
    if (!ovl_dir_is_real() || impurity)

But this still doesn't explain the reported issue.

The OVL_WHITEOUTS inode flag is set in ovl_get_inode() in several
cases including:
    ovl_check_origin_xattr(ofs, upperdentry)

So now we are getting closer to something that sounds related to the
reported issue...

ovl_check_origin_xattr() would return true if
    vfs_getxattr(upperdentry, "trusted.overlay.origin", NULL, 0)
would return 0 instead of -ENODATA for some reason even though that
xattr does not exist.

But we happen to be missing a pr_debug() in  ovl_do_getxattr(), so
it's hard to say what's going on.

Chris,

As the first step, can you try the suggested fix to ovl_dentry_version_inc()
and/or adding the missing pr_debug() and including those prints in
your report?

> I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
> approximately the same time I see one, sometimes more, kernel
> messages:
>
> [ 6295.379283] overlayfs: upper fs does not support xattr, falling
> back to index=off and metacopy=off.
>

Can you say why there is no xattr support?
Is the overlayfs mount executed without privileges to create trusted.* xattrs?
The answer to that may be the key to understanding the bug.

> But I don't know if that kernel message relates to the bolt test failure.
>
> If I run the test outside of a container, it doesn't fail. If I run
> the test in a podman container using the btrfs driver instead of the
> overlay driver, it doesn't fail. So it seems like this is an overlayfs
> bug, but could be some kind of overlayfs+btrfs interaction.
>

My guess is it has to do with changes related to mounting overlayfs
inside userns, but I couldn't find any immediate suspects.

Do you have any idea since when the regression appeared?
A bisect would have been helpful here.

> Could this be related and just not yet merged?
> https://lore.kernel.org/linux-unionfs/20210309162654.243184-1-amir73il@gmail.com/
>

Not likely.
If you want to be sure do:
echo N > /sys/module/overlay/parameters/xino_auto
Before starting the container.
Above commit only matters for xino_auto = Y.

Thanks,
Amir.

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Chris Murphy]

I can reproduce the bolt testcase problem in a podman container, with
overlay driver, using ext4, xfs, and btrfs. So I think I can drop
linux-btrfs@ from this thread.

Also I can reproduce the title of this thread simply by 'podman system
reset' and see the kernel messages before doing the actual reset. I
have a strace here of what it's doing:

https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing

It may be something intentional. The failing testcase,
:../tests/test-common.c:1413:test_io_dir_is_empty also has more
instances of this line, but I don't know they are related. So I'll
keep looking into that.


On Sat, Apr 10, 2021 at 2:04 AM Amir Goldstein <amir73il@gmail.com> wrote:

> As the first step, can you try the suggested fix to ovl_dentry_version_inc()
> and/or adding the missing pr_debug() and including those prints in
> your report?

I'll work with bolt upstream and try to further narrow down when it is
and isn't happening.

> > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
> > approximately the same time I see one, sometimes more, kernel
> > messages:
> >
> > [ 6295.379283] overlayfs: upper fs does not support xattr, falling
> > back to index=off and metacopy=off.
> >
>
> Can you say why there is no xattr support?

I'm not sure. It could be podman specific or fuse-overlayfs related.
Maybe something is using /tmp in one case and not another for some
reason?

> Is the overlayfs mount executed without privileges to create trusted.* xattrs?
> The answer to that may be the key to understanding the bug.

Yep. I think tmpfs supports xattr but not user xattr? And this example
is rootless podman, so it's all unprivileged.


> My guess is it has to do with changes related to mounting overlayfs
> inside userns, but I couldn't find any immediate suspects.
>
> Do you have any idea since when the regression appeared?
> A bisect would have been helpful here.

Yep. All good ideas. Thanks for the fast reply. I'll report back once
this has been narrowed down futher.


-- 
Chris Murphy

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Amir Goldstein]

On Sat, Apr 10, 2021 at 8:36 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> I can reproduce the bolt testcase problem in a podman container, with
> overlay driver, using ext4, xfs, and btrfs. So I think I can drop
> linux-btrfs@ from this thread.
>
> Also I can reproduce the title of this thread simply by 'podman system
> reset' and see the kernel messages before doing the actual reset. I
> have a strace here of what it's doing:
>
> https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing
>

I'm confused. The error in the title of the page is from overlayfs mount().
I see no mount in the strace.
I feel that I am missing some info.
Can you provide the overlayfs mount arguments
and more information about the underlying layers?

> It may be something intentional. The failing testcase,
> :../tests/test-common.c:1413:test_io_dir_is_empty also has more
> instances of this line, but I don't know they are related. So I'll
> keep looking into that.
>
>
> On Sat, Apr 10, 2021 at 2:04 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> > As the first step, can you try the suggested fix to ovl_dentry_version_inc()
> > and/or adding the missing pr_debug() and including those prints in
> > your report?
>
> I'll work with bolt upstream and try to further narrow down when it is
> and isn't happening.
>
> > > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
> > > approximately the same time I see one, sometimes more, kernel
> > > messages:
> > >
> > > [ 6295.379283] overlayfs: upper fs does not support xattr, falling
> > > back to index=off and metacopy=off.
> > >
> >
> > Can you say why there is no xattr support?
>
> I'm not sure. It could be podman specific or fuse-overlayfs related.

Not sure how fuse-overlayfs is related.
This is a message from overlayfs kernel driver.

> Maybe something is using /tmp in one case and not another for some
> reason?
>
> > Is the overlayfs mount executed without privileges to create trusted.* xattrs?
> > The answer to that may be the key to understanding the bug.
>
> Yep. I think tmpfs supports xattr but not user xattr? And this example
> is rootless podman, so it's all unprivileged.
>

OK, so unprivileged overlayfs mount support was added in v5.11
and it requires opt-in with mount option "userxattr", which could
explain the problem if tmpfs is used as upper layer.

Do you know if that is the case?
I sounds to me like it may not be a kernel regression per-se,
but a regression in the container runtime that started to use
a new kernel feature?
Need more context to understand.

Perhaps the solution will be to add user xattr support to tmpfs..

Thanks,
Amir.

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Chris Murphy]

On Sat, Apr 10, 2021 at 11:55 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Sat, Apr 10, 2021 at 8:36 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > I can reproduce the bolt testcase problem in a podman container, with
> > overlay driver, using ext4, xfs, and btrfs. So I think I can drop
> > linux-btrfs@ from this thread.
> >
> > Also I can reproduce the title of this thread simply by 'podman system
> > reset' and see the kernel messages before doing the actual reset. I
> > have a strace here of what it's doing:
> >
> > https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing
> >
>
> I'm confused. The error in the title of the page is from overlayfs mount().
> I see no mount in the strace.
> I feel that I am missing some info.
> Can you provide the overlayfs mount arguments
> and more information about the underlying layers?

Not really? There are none if a container isn't running, and in this
case no containers are running, in fact there are no upper or lower
dirs because I had already reset podman before doing 'strace podman
system reset' - I get the kernel message twice every time I merely do
'podman system reset'

overlayfs: upper fs does not support xattr, falling back to index=off
and metacopy=off
overlayfs: upper fs does not support xattr, falling back to index=off
and metacopy=off

This part of the issue might be something of a goose chase. I don't
know if it's relevant or distracting.


> > Yep. I think tmpfs supports xattr but not user xattr? And this example
> > is rootless podman, so it's all unprivileged.
> >
>
> OK, so unprivileged overlayfs mount support was added in v5.11
> and it requires opt-in with mount option "userxattr", which could
> explain the problem if tmpfs is used as upper layer.
>
> Do you know if that is the case?
> I sounds to me like it may not be a kernel regression per-se,
> but a regression in the container runtime that started to use
> a new kernel feature?
> Need more context to understand.
>
> Perhaps the solution will be to add user xattr support to tmpfs..

$ sudo mount -o remount,userxattr /home
mount: /home: mount point not mounted or bad option.

[   92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr'

/home is effectively a bind mount because it is backed by a btrfs subvolume...

/dev/sda6 on /home type btrfs
(rw,noatime,seclabel,compress=zstd:1,ssd,space_cache=v2,subvolid=586,subvol=/home)

...which is mounted via fstab using -o subvol=home

Is it supported to remount,userxattr? If not then maybe this is needed:

rootflags=subvol=root,userxattr


-- 
Chris Murphy

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Chris Murphy]

On Sat, Apr 10, 2021 at 1:36 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> $ sudo mount -o remount,userxattr /home
> mount: /home: mount point not mounted or bad option.
>
> [   92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr'
>

[   63.320831] BTRFS error (device sda6): unrecognized mount option 'user_xattr'

And if I try it with rootflags at boot, boot fails due to mount
failure due to unrecognized mount option.


-- 
Chris Murphy

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Chris Murphy]

On Sat, Apr 10, 2021 at 1:42 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Apr 10, 2021 at 1:36 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > $ sudo mount -o remount,userxattr /home
> > mount: /home: mount point not mounted or bad option.
> >
> > [   92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr'
> >
>
> [   63.320831] BTRFS error (device sda6): unrecognized mount option 'user_xattr'
>
> And if I try it with rootflags at boot, boot fails due to mount
> failure due to unrecognized mount option.

These are all with kernel 5.12-rc6


-- 
Chris Murphy

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Chris Murphy]

On Sat, Apr 10, 2021 at 1:43 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Apr 10, 2021 at 1:42 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > On Sat, Apr 10, 2021 at 1:36 PM Chris Murphy <lists@colorremedies.com> wrote:
> > >
> > > $ sudo mount -o remount,userxattr /home
> > > mount: /home: mount point not mounted or bad option.
> > >
> > > [   92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr'
> > >
> >
> > [   63.320831] BTRFS error (device sda6): unrecognized mount option 'user_xattr'
> >
> > And if I try it with rootflags at boot, boot fails due to mount
> > failure due to unrecognized mount option.
>
> These are all with kernel 5.12-rc6


Ohhh to tmpfs. Hmmm. I have no idea how to do that with this test
suite. I'll ask bolt folks. I'm just good at bumping into walls,
obviously.

-- 
Chris Murphy

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Chris Murphy]

Keeping everything else the same, and only reverting to kernel
5.9.16-200.fc33.x86_64, this kernel message

>overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off

no longer appears when I 'podman system reset' or when 'podman build'
bolt, using the overlay driver.

However, I do still get
Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty:
'empty' should be FALSE


--
Chris Murphy

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Amir Goldstein]

[removing btrfs list]

On Sat, Apr 10, 2021 at 11:03 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> Keeping everything else the same, and only reverting to kernel
> 5.9.16-200.fc33.x86_64, this kernel message
>
> >overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off
>
> no longer appears when I 'podman system reset' or when 'podman build'
> bolt, using the overlay driver.
>

I don't see a change in overlayfs that would explain seeing this warning
in v5.12 and not in v5.9 - on the contrary, in v5.12 the warning is printed
only if index or metacopy features have actually been requested.

So it must be a change in the upper fs, which is tmpfs?
Anyway, I don't have enough information and this seems unrelated
to the test failure, so I'll drop it.

> However, I do still get
> Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty:
> 'empty' should be FALSE
>

Now I'm confused again.

Your reports starts by stating:
"The primary problem is Bolt (Thunderbolt 3) tests that are
experiencing a regression when run in a container using overlayfs,"

But you say that the problem exists with kernel 5.9.
When you say "regression" above, what are you referring to?

Did those tests pass in a previous Bolt version?
Did those tests ever pass in a container using overlayfs?

There is surely a bug in overlayfs, but it's hard to find it without
minimal bisection info. I'll keep looking.

If it's a regression with newer distro, please try to understand
from distro/package managers, what has changed in the container
setup and kernel config w.r.t a container using overlayfs.

Thanks,
Amir.

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Chris Murphy]

On Sat, Apr 10, 2021 at 11:12 PM Amir Goldstein <amir73il@gmail.com> wrote:

> Now I'm confused again.

So am I, and in retrospect I've posted here prematurely.

>
> Your reports starts by stating:
> "The primary problem is Bolt (Thunderbolt 3) tests that are
> experiencing a regression when run in a container using overlayfs,"
>
> But you say that the problem exists with kernel 5.9.
> When you say "regression" above, what are you referring to?

Overlayfs. Now that I've tested 5.9, I'm not so sure it's a kernel regression.

>
> Did those tests pass in a previous Bolt version?
> Did those tests ever pass in a container using overlayfs?

Yes and yes.

> There is surely a bug in overlayfs, but it's hard to find it without
> minimal bisection info. I'll keep looking.
>
> If it's a regression with newer distro, please try to understand
> from distro/package managers, what has changed in the container
> setup and kernel config w.r.t a container using overlayfs.

Exactly. The original report of the problem is Alpine linux, but I
can't reproduce it on Fedora except with podman using an Alpine image
base. As all the other suspects have fallen apart, what remains
untested for regressions is this.


-- 
Chris Murphy

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Amir Goldstein]

On Sun, Apr 11, 2021 at 9:05 AM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Apr 10, 2021 at 11:12 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> > Now I'm confused again.
>
> So am I, and in retrospect I've posted here prematurely.
>
> >
> > Your reports starts by stating:
> > "The primary problem is Bolt (Thunderbolt 3) tests that are
> > experiencing a regression when run in a container using overlayfs,"
> >
> > But you say that the problem exists with kernel 5.9.
> > When you say "regression" above, what are you referring to?
>
> Overlayfs. Now that I've tested 5.9, I'm not so sure it's a kernel regression.
>
> >
> > Did those tests pass in a previous Bolt version?
> > Did those tests ever pass in a container using overlayfs?
>
> Yes and yes.
>
> > There is surely a bug in overlayfs, but it's hard to find it without
> > minimal bisection info. I'll keep looking.
> >
> > If it's a regression with newer distro, please try to understand
> > from distro/package managers, what has changed in the container
> > setup and kernel config w.r.t a container using overlayfs.
>
> Exactly. The original report of the problem is Alpine linux, but I
> can't reproduce it on Fedora except with podman using an Alpine image
> base. As all the other suspects have fallen apart, what remains
> untested for regressions is this.
>

I'm lost in the maze of distros and containers.
Will wait for more info.
In any case, I was able to reproduce the bug in ovl_dir_version_inc()
I will post a fix soon.

But I don't see how the test case you reported can be affected.
The bug I reproduced requires an upper directory that used to be
a merge dir and whose lower dir was removed while overlayfs was
offline.

Thanks,
Amir.

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Miklos Szeredi]

[-- Attachment #1: Type: text/plain, Size: 2964 bytes --]

On Sat, Apr 10, 2021 at 7:55 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Sat, Apr 10, 2021 at 8:36 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > I can reproduce the bolt testcase problem in a podman container, with
> > overlay driver, using ext4, xfs, and btrfs. So I think I can drop
> > linux-btrfs@ from this thread.
> >
> > Also I can reproduce the title of this thread simply by 'podman system
> > reset' and see the kernel messages before doing the actual reset. I
> > have a strace here of what it's doing:
> >
> > https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing
> >
>
> I'm confused. The error in the title of the page is from overlayfs mount().
> I see no mount in the strace.
> I feel that I am missing some info.
> Can you provide the overlayfs mount arguments
> and more information about the underlying layers?
>
> > It may be something intentional. The failing testcase,
> > :../tests/test-common.c:1413:test_io_dir_is_empty also has more
> > instances of this line, but I don't know they are related. So I'll
> > keep looking into that.
> >
> >
> > On Sat, Apr 10, 2021 at 2:04 AM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > > As the first step, can you try the suggested fix to ovl_dentry_version_inc()
> > > and/or adding the missing pr_debug() and including those prints in
> > > your report?
> >
> > I'll work with bolt upstream and try to further narrow down when it is
> > and isn't happening.
> >
> > > > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
> > > > approximately the same time I see one, sometimes more, kernel
> > > > messages:
> > > >
> > > > [ 6295.379283] overlayfs: upper fs does not support xattr, falling
> > > > back to index=off and metacopy=off.
> > > >
> > >
> > > Can you say why there is no xattr support?
> >
> > I'm not sure. It could be podman specific or fuse-overlayfs related.
>
> Not sure how fuse-overlayfs is related.
> This is a message from overlayfs kernel driver.
>
> > Maybe something is using /tmp in one case and not another for some
> > reason?
> >
> > > Is the overlayfs mount executed without privileges to create trusted.* xattrs?
> > > The answer to that may be the key to understanding the bug.
> >
> > Yep. I think tmpfs supports xattr but not user xattr? And this example
> > is rootless podman, so it's all unprivileged.
> >
>
> OK, so unprivileged overlayfs mount support was added in v5.11
> and it requires opt-in with mount option "userxattr", which could
> explain the problem if tmpfs is used as upper layer.
>
> Do you know if that is the case?
> I sounds to me like it may not be a kernel regression per-se,
> but a regression in the container runtime that started to use
> a new kernel feature?
> Need more context to understand.
>
> Perhaps the solution will be to add user xattr support to tmpfs..

Attached patch.   Tested at some earlier time, since I also bumped
into this issue.

Thanks,
Miklos

[-- Attachment #2: tmpfs-allow-user-xattr.patch --]
[-- Type: text/x-patch, Size: 737 bytes --]

diff --git a/mm/shmem.c b/mm/shmem.c
index d722eb830317..afe59086b3f6 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -3250,6 +3250,12 @@ static const struct xattr_handler shmem_trusted_xattr_handler = {
 	.set = shmem_xattr_handler_set,
 };
 
+static const struct xattr_handler shmem_user_xattr_handler = {
+	.prefix = XATTR_USER_PREFIX,
+	.get = shmem_xattr_handler_get,
+	.set = shmem_xattr_handler_set,
+};
+
 static const struct xattr_handler *shmem_xattr_handlers[] = {
 #ifdef CONFIG_TMPFS_POSIX_ACL
 	&posix_acl_access_xattr_handler,
@@ -3257,6 +3263,7 @@ static const struct xattr_handler *shmem_xattr_handlers[] = {
 #endif
 	&shmem_security_xattr_handler,
 	&shmem_trusted_xattr_handler,
+	&shmem_user_xattr_handler,
 	NULL
 };
 

Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

[Chris Murphy]

On Sun, Apr 11, 2021 at 1:29 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> I'm lost in the maze of distros and containers.
> Will wait for more info.

The bolt test suite failure looks like it fuse-overlayfs related. It
happens with (rootless) podman but not when using sudo podman.

https://github.com/containers/fuse-overlayfs/issues/287


-- 
Chris Murphy