* btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
@ 2021-04-09 22:03 Chris Murphy
2021-04-10 8:03 ` Amir Goldstein
0 siblings, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2021-04-09 22:03 UTC (permalink / raw)
To: Btrfs BTRFS, linux-unionfs
Hi,
The primary problem is Bolt (Thunderbolt 3) tests that are
experiencing a regression when run in a container using overlayfs,
failing at:
Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty:
'empty' should be FALSE
https://gitlab.freedesktop.org/bolt/bolt/-/issues/171#note_872119
I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
approximately the same time I see one, sometimes more, kernel
messages:
[ 6295.379283] overlayfs: upper fs does not support xattr, falling
back to index=off and metacopy=off.
But I don't know if that kernel message relates to the bolt test failure.
If I run the test outside of a container, it doesn't fail. If I run
the test in a podman container using the btrfs driver instead of the
overlay driver, it doesn't fail. So it seems like this is an overlayfs
bug, but could be some kind of overlayfs+btrfs interaction.
Could this be related and just not yet merged?
https://lore.kernel.org/linux-unionfs/20210309162654.243184-1-amir73il@gmail.com/
Thanks,
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-09 22:03 btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off Chris Murphy
@ 2021-04-10 8:03 ` Amir Goldstein
2021-04-10 17:36 ` Chris Murphy
0 siblings, 1 reply; 14+ messages in thread
From: Amir Goldstein @ 2021-04-10 8:03 UTC (permalink / raw)
To: Chris Murphy, Miklos Szeredi; +Cc: Btrfs BTRFS, overlayfs
On Sat, Apr 10, 2021 at 1:04 AM Chris Murphy <lists@colorremedies.com> wrote:
>
> Hi,
>
> The primary problem is Bolt (Thunderbolt 3) tests that are
> experiencing a regression when run in a container using overlayfs,
> failing at:
>
> Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty:
> 'empty' should be FALSE
>
> https://gitlab.freedesktop.org/bolt/bolt/-/issues/171#note_872119
>
To summarize, the test case is:
- create empty dir
- open empty dir
- getdents => (".", "..")
- create file at (dirfd, "a",
- lseek to offset 0 on dirfd
- getdents => (".", "..") FAIL to see "a"
It looks like a bug in ovl readdir cache invalidation only there is
not supposed to be any caching of pure upper dir.
Once thing I noticed is that ovl_dentry_version_inc() is inconsistent
with ovl_dir_is_real() - the latter checks whether readdir caching would
be used and the former checks whether invalidating readdir cache is
needed. We need to change ovl_dentry_version_inc() test to:
if (ovl_test_flag(OVL_WHITEOUTS, dir) || impurity)
Or better yet:
if (!ovl_dir_is_real() || impurity)
But this still doesn't explain the reported issue.
The OVL_WHITEOUTS inode flag is set in ovl_get_inode() in several
cases including:
ovl_check_origin_xattr(ofs, upperdentry)
So now we are getting closer to something that sounds related to the
reported issue...
ovl_check_origin_xattr() would return true if
vfs_getxattr(upperdentry, "trusted.overlay.origin", NULL, 0)
would return 0 instead of -ENODATA for some reason even though that
xattr does not exist.
But we happen to be missing a pr_debug() in ovl_do_getxattr(), so
it's hard to say what's going on.
Chris,
As the first step, can you try the suggested fix to ovl_dentry_version_inc()
and/or adding the missing pr_debug() and including those prints in
your report?
> I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
> approximately the same time I see one, sometimes more, kernel
> messages:
>
> [ 6295.379283] overlayfs: upper fs does not support xattr, falling
> back to index=off and metacopy=off.
>
Can you say why there is no xattr support?
Is the overlayfs mount executed without privileges to create trusted.* xattrs?
The answer to that may be the key to understanding the bug.
> But I don't know if that kernel message relates to the bolt test failure.
>
> If I run the test outside of a container, it doesn't fail. If I run
> the test in a podman container using the btrfs driver instead of the
> overlay driver, it doesn't fail. So it seems like this is an overlayfs
> bug, but could be some kind of overlayfs+btrfs interaction.
>
My guess is it has to do with changes related to mounting overlayfs
inside userns, but I couldn't find any immediate suspects.
Do you have any idea since when the regression appeared?
A bisect would have been helpful here.
> Could this be related and just not yet merged?
> https://lore.kernel.org/linux-unionfs/20210309162654.243184-1-amir73il@gmail.com/
>
Not likely.
If you want to be sure do:
echo N > /sys/module/overlay/parameters/xino_auto
Before starting the container.
Above commit only matters for xino_auto = Y.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-10 8:03 ` Amir Goldstein
@ 2021-04-10 17:36 ` Chris Murphy
2021-04-10 17:55 ` Amir Goldstein
0 siblings, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2021-04-10 17:36 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Chris Murphy, Miklos Szeredi, Btrfs BTRFS, overlayfs
I can reproduce the bolt testcase problem in a podman container, with
overlay driver, using ext4, xfs, and btrfs. So I think I can drop
linux-btrfs@ from this thread.
Also I can reproduce the title of this thread simply by 'podman system
reset' and see the kernel messages before doing the actual reset. I
have a strace here of what it's doing:
https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing
It may be something intentional. The failing testcase,
:../tests/test-common.c:1413:test_io_dir_is_empty also has more
instances of this line, but I don't know they are related. So I'll
keep looking into that.
On Sat, Apr 10, 2021 at 2:04 AM Amir Goldstein <amir73il@gmail.com> wrote:
> As the first step, can you try the suggested fix to ovl_dentry_version_inc()
> and/or adding the missing pr_debug() and including those prints in
> your report?
I'll work with bolt upstream and try to further narrow down when it is
and isn't happening.
> > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
> > approximately the same time I see one, sometimes more, kernel
> > messages:
> >
> > [ 6295.379283] overlayfs: upper fs does not support xattr, falling
> > back to index=off and metacopy=off.
> >
>
> Can you say why there is no xattr support?
I'm not sure. It could be podman specific or fuse-overlayfs related.
Maybe something is using /tmp in one case and not another for some
reason?
> Is the overlayfs mount executed without privileges to create trusted.* xattrs?
> The answer to that may be the key to understanding the bug.
Yep. I think tmpfs supports xattr but not user xattr? And this example
is rootless podman, so it's all unprivileged.
> My guess is it has to do with changes related to mounting overlayfs
> inside userns, but I couldn't find any immediate suspects.
>
> Do you have any idea since when the regression appeared?
> A bisect would have been helpful here.
Yep. All good ideas. Thanks for the fast reply. I'll report back once
this has been narrowed down futher.
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-10 17:36 ` Chris Murphy
@ 2021-04-10 17:55 ` Amir Goldstein
2021-04-10 19:36 ` Chris Murphy
2021-04-12 8:41 ` Miklos Szeredi
0 siblings, 2 replies; 14+ messages in thread
From: Amir Goldstein @ 2021-04-10 17:55 UTC (permalink / raw)
To: Chris Murphy; +Cc: Miklos Szeredi, Btrfs BTRFS, overlayfs
On Sat, Apr 10, 2021 at 8:36 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> I can reproduce the bolt testcase problem in a podman container, with
> overlay driver, using ext4, xfs, and btrfs. So I think I can drop
> linux-btrfs@ from this thread.
>
> Also I can reproduce the title of this thread simply by 'podman system
> reset' and see the kernel messages before doing the actual reset. I
> have a strace here of what it's doing:
>
> https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing
>
I'm confused. The error in the title of the page is from overlayfs mount().
I see no mount in the strace.
I feel that I am missing some info.
Can you provide the overlayfs mount arguments
and more information about the underlying layers?
> It may be something intentional. The failing testcase,
> :../tests/test-common.c:1413:test_io_dir_is_empty also has more
> instances of this line, but I don't know they are related. So I'll
> keep looking into that.
>
>
> On Sat, Apr 10, 2021 at 2:04 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> > As the first step, can you try the suggested fix to ovl_dentry_version_inc()
> > and/or adding the missing pr_debug() and including those prints in
> > your report?
>
> I'll work with bolt upstream and try to further narrow down when it is
> and isn't happening.
>
> > > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
> > > approximately the same time I see one, sometimes more, kernel
> > > messages:
> > >
> > > [ 6295.379283] overlayfs: upper fs does not support xattr, falling
> > > back to index=off and metacopy=off.
> > >
> >
> > Can you say why there is no xattr support?
>
> I'm not sure. It could be podman specific or fuse-overlayfs related.
Not sure how fuse-overlayfs is related.
This is a message from overlayfs kernel driver.
> Maybe something is using /tmp in one case and not another for some
> reason?
>
> > Is the overlayfs mount executed without privileges to create trusted.* xattrs?
> > The answer to that may be the key to understanding the bug.
>
> Yep. I think tmpfs supports xattr but not user xattr? And this example
> is rootless podman, so it's all unprivileged.
>
OK, so unprivileged overlayfs mount support was added in v5.11
and it requires opt-in with mount option "userxattr", which could
explain the problem if tmpfs is used as upper layer.
Do you know if that is the case?
I sounds to me like it may not be a kernel regression per-se,
but a regression in the container runtime that started to use
a new kernel feature?
Need more context to understand.
Perhaps the solution will be to add user xattr support to tmpfs..
Thanks,
Amir.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-10 17:55 ` Amir Goldstein
@ 2021-04-10 19:36 ` Chris Murphy
2021-04-10 19:42 ` Chris Murphy
2021-04-12 8:41 ` Miklos Szeredi
1 sibling, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2021-04-10 19:36 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Chris Murphy, Miklos Szeredi, Btrfs BTRFS, overlayfs
On Sat, Apr 10, 2021 at 11:55 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Sat, Apr 10, 2021 at 8:36 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > I can reproduce the bolt testcase problem in a podman container, with
> > overlay driver, using ext4, xfs, and btrfs. So I think I can drop
> > linux-btrfs@ from this thread.
> >
> > Also I can reproduce the title of this thread simply by 'podman system
> > reset' and see the kernel messages before doing the actual reset. I
> > have a strace here of what it's doing:
> >
> > https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing
> >
>
> I'm confused. The error in the title of the page is from overlayfs mount().
> I see no mount in the strace.
> I feel that I am missing some info.
> Can you provide the overlayfs mount arguments
> and more information about the underlying layers?
Not really? There are none if a container isn't running, and in this
case no containers are running, in fact there are no upper or lower
dirs because I had already reset podman before doing 'strace podman
system reset' - I get the kernel message twice every time I merely do
'podman system reset'
overlayfs: upper fs does not support xattr, falling back to index=off
and metacopy=off
overlayfs: upper fs does not support xattr, falling back to index=off
and metacopy=off
This part of the issue might be something of a goose chase. I don't
know if it's relevant or distracting.
> > Yep. I think tmpfs supports xattr but not user xattr? And this example
> > is rootless podman, so it's all unprivileged.
> >
>
> OK, so unprivileged overlayfs mount support was added in v5.11
> and it requires opt-in with mount option "userxattr", which could
> explain the problem if tmpfs is used as upper layer.
>
> Do you know if that is the case?
> I sounds to me like it may not be a kernel regression per-se,
> but a regression in the container runtime that started to use
> a new kernel feature?
> Need more context to understand.
>
> Perhaps the solution will be to add user xattr support to tmpfs..
$ sudo mount -o remount,userxattr /home
mount: /home: mount point not mounted or bad option.
[ 92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr'
/home is effectively a bind mount because it is backed by a btrfs subvolume...
/dev/sda6 on /home type btrfs
(rw,noatime,seclabel,compress=zstd:1,ssd,space_cache=v2,subvolid=586,subvol=/home)
...which is mounted via fstab using -o subvol=home
Is it supported to remount,userxattr? If not then maybe this is needed:
rootflags=subvol=root,userxattr
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-10 19:36 ` Chris Murphy
@ 2021-04-10 19:42 ` Chris Murphy
2021-04-10 19:43 ` Chris Murphy
0 siblings, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2021-04-10 19:42 UTC (permalink / raw)
To: Chris Murphy; +Cc: Amir Goldstein, Miklos Szeredi, Btrfs BTRFS, overlayfs
On Sat, Apr 10, 2021 at 1:36 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> $ sudo mount -o remount,userxattr /home
> mount: /home: mount point not mounted or bad option.
>
> [ 92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr'
>
[ 63.320831] BTRFS error (device sda6): unrecognized mount option 'user_xattr'
And if I try it with rootflags at boot, boot fails due to mount
failure due to unrecognized mount option.
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-10 19:42 ` Chris Murphy
@ 2021-04-10 19:43 ` Chris Murphy
2021-04-10 19:44 ` Chris Murphy
0 siblings, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2021-04-10 19:43 UTC (permalink / raw)
To: Chris Murphy; +Cc: Amir Goldstein, Miklos Szeredi, Btrfs BTRFS, overlayfs
On Sat, Apr 10, 2021 at 1:42 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Apr 10, 2021 at 1:36 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > $ sudo mount -o remount,userxattr /home
> > mount: /home: mount point not mounted or bad option.
> >
> > [ 92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr'
> >
>
> [ 63.320831] BTRFS error (device sda6): unrecognized mount option 'user_xattr'
>
> And if I try it with rootflags at boot, boot fails due to mount
> failure due to unrecognized mount option.
These are all with kernel 5.12-rc6
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-10 19:43 ` Chris Murphy
@ 2021-04-10 19:44 ` Chris Murphy
2021-04-10 20:03 ` Chris Murphy
0 siblings, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2021-04-10 19:44 UTC (permalink / raw)
To: Chris Murphy; +Cc: Amir Goldstein, Miklos Szeredi, Btrfs BTRFS, overlayfs
On Sat, Apr 10, 2021 at 1:43 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Apr 10, 2021 at 1:42 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > On Sat, Apr 10, 2021 at 1:36 PM Chris Murphy <lists@colorremedies.com> wrote:
> > >
> > > $ sudo mount -o remount,userxattr /home
> > > mount: /home: mount point not mounted or bad option.
> > >
> > > [ 92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr'
> > >
> >
> > [ 63.320831] BTRFS error (device sda6): unrecognized mount option 'user_xattr'
> >
> > And if I try it with rootflags at boot, boot fails due to mount
> > failure due to unrecognized mount option.
>
> These are all with kernel 5.12-rc6
Ohhh to tmpfs. Hmmm. I have no idea how to do that with this test
suite. I'll ask bolt folks. I'm just good at bumping into walls,
obviously.
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-10 19:44 ` Chris Murphy
@ 2021-04-10 20:03 ` Chris Murphy
2021-04-11 5:12 ` Amir Goldstein
0 siblings, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2021-04-10 20:03 UTC (permalink / raw)
To: Chris Murphy; +Cc: Amir Goldstein, Miklos Szeredi, Btrfs BTRFS, overlayfs
Keeping everything else the same, and only reverting to kernel
5.9.16-200.fc33.x86_64, this kernel message
>overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off
no longer appears when I 'podman system reset' or when 'podman build'
bolt, using the overlay driver.
However, I do still get
Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty:
'empty' should be FALSE
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-10 20:03 ` Chris Murphy
@ 2021-04-11 5:12 ` Amir Goldstein
2021-04-11 6:05 ` Chris Murphy
0 siblings, 1 reply; 14+ messages in thread
From: Amir Goldstein @ 2021-04-11 5:12 UTC (permalink / raw)
To: Chris Murphy; +Cc: Miklos Szeredi, overlayfs
[removing btrfs list]
On Sat, Apr 10, 2021 at 11:03 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> Keeping everything else the same, and only reverting to kernel
> 5.9.16-200.fc33.x86_64, this kernel message
>
> >overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off
>
> no longer appears when I 'podman system reset' or when 'podman build'
> bolt, using the overlay driver.
>
I don't see a change in overlayfs that would explain seeing this warning
in v5.12 and not in v5.9 - on the contrary, in v5.12 the warning is printed
only if index or metacopy features have actually been requested.
So it must be a change in the upper fs, which is tmpfs?
Anyway, I don't have enough information and this seems unrelated
to the test failure, so I'll drop it.
> However, I do still get
> Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty:
> 'empty' should be FALSE
>
Now I'm confused again.
Your reports starts by stating:
"The primary problem is Bolt (Thunderbolt 3) tests that are
experiencing a regression when run in a container using overlayfs,"
But you say that the problem exists with kernel 5.9.
When you say "regression" above, what are you referring to?
Did those tests pass in a previous Bolt version?
Did those tests ever pass in a container using overlayfs?
There is surely a bug in overlayfs, but it's hard to find it without
minimal bisection info. I'll keep looking.
If it's a regression with newer distro, please try to understand
from distro/package managers, what has changed in the container
setup and kernel config w.r.t a container using overlayfs.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-11 5:12 ` Amir Goldstein
@ 2021-04-11 6:05 ` Chris Murphy
2021-04-11 7:28 ` Amir Goldstein
0 siblings, 1 reply; 14+ messages in thread
From: Chris Murphy @ 2021-04-11 6:05 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Chris Murphy, Miklos Szeredi, overlayfs
On Sat, Apr 10, 2021 at 11:12 PM Amir Goldstein <amir73il@gmail.com> wrote:
> Now I'm confused again.
So am I, and in retrospect I've posted here prematurely.
>
> Your reports starts by stating:
> "The primary problem is Bolt (Thunderbolt 3) tests that are
> experiencing a regression when run in a container using overlayfs,"
>
> But you say that the problem exists with kernel 5.9.
> When you say "regression" above, what are you referring to?
Overlayfs. Now that I've tested 5.9, I'm not so sure it's a kernel regression.
>
> Did those tests pass in a previous Bolt version?
> Did those tests ever pass in a container using overlayfs?
Yes and yes.
> There is surely a bug in overlayfs, but it's hard to find it without
> minimal bisection info. I'll keep looking.
>
> If it's a regression with newer distro, please try to understand
> from distro/package managers, what has changed in the container
> setup and kernel config w.r.t a container using overlayfs.
Exactly. The original report of the problem is Alpine linux, but I
can't reproduce it on Fedora except with podman using an Alpine image
base. As all the other suspects have fallen apart, what remains
untested for regressions is this.
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-11 6:05 ` Chris Murphy
@ 2021-04-11 7:28 ` Amir Goldstein
2021-04-13 21:50 ` Chris Murphy
0 siblings, 1 reply; 14+ messages in thread
From: Amir Goldstein @ 2021-04-11 7:28 UTC (permalink / raw)
To: Chris Murphy; +Cc: Miklos Szeredi, overlayfs
On Sun, Apr 11, 2021 at 9:05 AM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Apr 10, 2021 at 11:12 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> > Now I'm confused again.
>
> So am I, and in retrospect I've posted here prematurely.
>
> >
> > Your reports starts by stating:
> > "The primary problem is Bolt (Thunderbolt 3) tests that are
> > experiencing a regression when run in a container using overlayfs,"
> >
> > But you say that the problem exists with kernel 5.9.
> > When you say "regression" above, what are you referring to?
>
> Overlayfs. Now that I've tested 5.9, I'm not so sure it's a kernel regression.
>
> >
> > Did those tests pass in a previous Bolt version?
> > Did those tests ever pass in a container using overlayfs?
>
> Yes and yes.
>
> > There is surely a bug in overlayfs, but it's hard to find it without
> > minimal bisection info. I'll keep looking.
> >
> > If it's a regression with newer distro, please try to understand
> > from distro/package managers, what has changed in the container
> > setup and kernel config w.r.t a container using overlayfs.
>
> Exactly. The original report of the problem is Alpine linux, but I
> can't reproduce it on Fedora except with podman using an Alpine image
> base. As all the other suspects have fallen apart, what remains
> untested for regressions is this.
>
I'm lost in the maze of distros and containers.
Will wait for more info.
In any case, I was able to reproduce the bug in ovl_dir_version_inc()
I will post a fix soon.
But I don't see how the test case you reported can be affected.
The bug I reproduced requires an upper directory that used to be
a merge dir and whose lower dir was removed while overlayfs was
offline.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-10 17:55 ` Amir Goldstein
2021-04-10 19:36 ` Chris Murphy
@ 2021-04-12 8:41 ` Miklos Szeredi
1 sibling, 0 replies; 14+ messages in thread
From: Miklos Szeredi @ 2021-04-12 8:41 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Chris Murphy, Btrfs BTRFS, overlayfs
[-- Attachment #1: Type: text/plain, Size: 2964 bytes --]
On Sat, Apr 10, 2021 at 7:55 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Sat, Apr 10, 2021 at 8:36 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > I can reproduce the bolt testcase problem in a podman container, with
> > overlay driver, using ext4, xfs, and btrfs. So I think I can drop
> > linux-btrfs@ from this thread.
> >
> > Also I can reproduce the title of this thread simply by 'podman system
> > reset' and see the kernel messages before doing the actual reset. I
> > have a strace here of what it's doing:
> >
> > https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing
> >
>
> I'm confused. The error in the title of the page is from overlayfs mount().
> I see no mount in the strace.
> I feel that I am missing some info.
> Can you provide the overlayfs mount arguments
> and more information about the underlying layers?
>
> > It may be something intentional. The failing testcase,
> > :../tests/test-common.c:1413:test_io_dir_is_empty also has more
> > instances of this line, but I don't know they are related. So I'll
> > keep looking into that.
> >
> >
> > On Sat, Apr 10, 2021 at 2:04 AM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > > As the first step, can you try the suggested fix to ovl_dentry_version_inc()
> > > and/or adding the missing pr_debug() and including those prints in
> > > your report?
> >
> > I'll work with bolt upstream and try to further narrow down when it is
> > and isn't happening.
> >
> > > > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at
> > > > approximately the same time I see one, sometimes more, kernel
> > > > messages:
> > > >
> > > > [ 6295.379283] overlayfs: upper fs does not support xattr, falling
> > > > back to index=off and metacopy=off.
> > > >
> > >
> > > Can you say why there is no xattr support?
> >
> > I'm not sure. It could be podman specific or fuse-overlayfs related.
>
> Not sure how fuse-overlayfs is related.
> This is a message from overlayfs kernel driver.
>
> > Maybe something is using /tmp in one case and not another for some
> > reason?
> >
> > > Is the overlayfs mount executed without privileges to create trusted.* xattrs?
> > > The answer to that may be the key to understanding the bug.
> >
> > Yep. I think tmpfs supports xattr but not user xattr? And this example
> > is rootless podman, so it's all unprivileged.
> >
>
> OK, so unprivileged overlayfs mount support was added in v5.11
> and it requires opt-in with mount option "userxattr", which could
> explain the problem if tmpfs is used as upper layer.
>
> Do you know if that is the case?
> I sounds to me like it may not be a kernel regression per-se,
> but a regression in the container runtime that started to use
> a new kernel feature?
> Need more context to understand.
>
> Perhaps the solution will be to add user xattr support to tmpfs..
Attached patch. Tested at some earlier time, since I also bumped
into this issue.
Thanks,
Miklos
[-- Attachment #2: tmpfs-allow-user-xattr.patch --]
[-- Type: text/x-patch, Size: 737 bytes --]
diff --git a/mm/shmem.c b/mm/shmem.c
index d722eb830317..afe59086b3f6 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -3250,6 +3250,12 @@ static const struct xattr_handler shmem_trusted_xattr_handler = {
.set = shmem_xattr_handler_set,
};
+static const struct xattr_handler shmem_user_xattr_handler = {
+ .prefix = XATTR_USER_PREFIX,
+ .get = shmem_xattr_handler_get,
+ .set = shmem_xattr_handler_set,
+};
+
static const struct xattr_handler *shmem_xattr_handlers[] = {
#ifdef CONFIG_TMPFS_POSIX_ACL
&posix_acl_access_xattr_handler,
@@ -3257,6 +3263,7 @@ static const struct xattr_handler *shmem_xattr_handlers[] = {
#endif
&shmem_security_xattr_handler,
&shmem_trusted_xattr_handler,
+ &shmem_user_xattr_handler,
NULL
};
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
2021-04-11 7:28 ` Amir Goldstein
@ 2021-04-13 21:50 ` Chris Murphy
0 siblings, 0 replies; 14+ messages in thread
From: Chris Murphy @ 2021-04-13 21:50 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Chris Murphy, Miklos Szeredi, overlayfs
On Sun, Apr 11, 2021 at 1:29 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> I'm lost in the maze of distros and containers.
> Will wait for more info.
The bolt test suite failure looks like it fuse-overlayfs related. It
happens with (rootless) podman but not when using sudo podman.
https://github.com/containers/fuse-overlayfs/issues/287
--
Chris Murphy
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-04-13 21:50 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-09 22:03 btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off Chris Murphy
2021-04-10 8:03 ` Amir Goldstein
2021-04-10 17:36 ` Chris Murphy
2021-04-10 17:55 ` Amir Goldstein
2021-04-10 19:36 ` Chris Murphy
2021-04-10 19:42 ` Chris Murphy
2021-04-10 19:43 ` Chris Murphy
2021-04-10 19:44 ` Chris Murphy
2021-04-10 20:03 ` Chris Murphy
2021-04-11 5:12 ` Amir Goldstein
2021-04-11 6:05 ` Chris Murphy
2021-04-11 7:28 ` Amir Goldstein
2021-04-13 21:50 ` Chris Murphy
2021-04-12 8:41 ` Miklos Szeredi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).