* btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. @ 2021-04-09 22:03 Chris Murphy 2021-04-10 8:03 ` Amir Goldstein 0 siblings, 1 reply; 14+ messages in thread From: Chris Murphy @ 2021-04-09 22:03 UTC (permalink / raw) To: Btrfs BTRFS, linux-unionfs Hi, The primary problem is Bolt (Thunderbolt 3) tests that are experiencing a regression when run in a container using overlayfs, failing at: Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty: 'empty' should be FALSE https://gitlab.freedesktop.org/bolt/bolt/-/issues/171#note_872119 I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at approximately the same time I see one, sometimes more, kernel messages: [ 6295.379283] overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. But I don't know if that kernel message relates to the bolt test failure. If I run the test outside of a container, it doesn't fail. If I run the test in a podman container using the btrfs driver instead of the overlay driver, it doesn't fail. So it seems like this is an overlayfs bug, but could be some kind of overlayfs+btrfs interaction. Could this be related and just not yet merged? https://lore.kernel.org/linux-unionfs/20210309162654.243184-1-amir73il@gmail.com/ Thanks, -- Chris Murphy ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-09 22:03 btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off Chris Murphy @ 2021-04-10 8:03 ` Amir Goldstein 2021-04-10 17:36 ` Chris Murphy 0 siblings, 1 reply; 14+ messages in thread From: Amir Goldstein @ 2021-04-10 8:03 UTC (permalink / raw) To: Chris Murphy, Miklos Szeredi; +Cc: Btrfs BTRFS, overlayfs On Sat, Apr 10, 2021 at 1:04 AM Chris Murphy <lists@colorremedies.com> wrote: > > Hi, > > The primary problem is Bolt (Thunderbolt 3) tests that are > experiencing a regression when run in a container using overlayfs, > failing at: > > Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty: > 'empty' should be FALSE > > https://gitlab.freedesktop.org/bolt/bolt/-/issues/171#note_872119 > To summarize, the test case is: - create empty dir - open empty dir - getdents => (".", "..") - create file at (dirfd, "a", - lseek to offset 0 on dirfd - getdents => (".", "..") FAIL to see "a" It looks like a bug in ovl readdir cache invalidation only there is not supposed to be any caching of pure upper dir. Once thing I noticed is that ovl_dentry_version_inc() is inconsistent with ovl_dir_is_real() - the latter checks whether readdir caching would be used and the former checks whether invalidating readdir cache is needed. We need to change ovl_dentry_version_inc() test to: if (ovl_test_flag(OVL_WHITEOUTS, dir) || impurity) Or better yet: if (!ovl_dir_is_real() || impurity) But this still doesn't explain the reported issue. The OVL_WHITEOUTS inode flag is set in ovl_get_inode() in several cases including: ovl_check_origin_xattr(ofs, upperdentry) So now we are getting closer to something that sounds related to the reported issue... ovl_check_origin_xattr() would return true if vfs_getxattr(upperdentry, "trusted.overlay.origin", NULL, 0) would return 0 instead of -ENODATA for some reason even though that xattr does not exist. But we happen to be missing a pr_debug() in ovl_do_getxattr(), so it's hard to say what's going on. Chris, As the first step, can you try the suggested fix to ovl_dentry_version_inc() and/or adding the missing pr_debug() and including those prints in your report? > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at > approximately the same time I see one, sometimes more, kernel > messages: > > [ 6295.379283] overlayfs: upper fs does not support xattr, falling > back to index=off and metacopy=off. > Can you say why there is no xattr support? Is the overlayfs mount executed without privileges to create trusted.* xattrs? The answer to that may be the key to understanding the bug. > But I don't know if that kernel message relates to the bolt test failure. > > If I run the test outside of a container, it doesn't fail. If I run > the test in a podman container using the btrfs driver instead of the > overlay driver, it doesn't fail. So it seems like this is an overlayfs > bug, but could be some kind of overlayfs+btrfs interaction. > My guess is it has to do with changes related to mounting overlayfs inside userns, but I couldn't find any immediate suspects. Do you have any idea since when the regression appeared? A bisect would have been helpful here. > Could this be related and just not yet merged? > https://lore.kernel.org/linux-unionfs/20210309162654.243184-1-amir73il@gmail.com/ > Not likely. If you want to be sure do: echo N > /sys/module/overlay/parameters/xino_auto Before starting the container. Above commit only matters for xino_auto = Y. Thanks, Amir. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-10 8:03 ` Amir Goldstein @ 2021-04-10 17:36 ` Chris Murphy 2021-04-10 17:55 ` Amir Goldstein 0 siblings, 1 reply; 14+ messages in thread From: Chris Murphy @ 2021-04-10 17:36 UTC (permalink / raw) To: Amir Goldstein; +Cc: Chris Murphy, Miklos Szeredi, Btrfs BTRFS, overlayfs I can reproduce the bolt testcase problem in a podman container, with overlay driver, using ext4, xfs, and btrfs. So I think I can drop linux-btrfs@ from this thread. Also I can reproduce the title of this thread simply by 'podman system reset' and see the kernel messages before doing the actual reset. I have a strace here of what it's doing: https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing It may be something intentional. The failing testcase, :../tests/test-common.c:1413:test_io_dir_is_empty also has more instances of this line, but I don't know they are related. So I'll keep looking into that. On Sat, Apr 10, 2021 at 2:04 AM Amir Goldstein <amir73il@gmail.com> wrote: > As the first step, can you try the suggested fix to ovl_dentry_version_inc() > and/or adding the missing pr_debug() and including those prints in > your report? I'll work with bolt upstream and try to further narrow down when it is and isn't happening. > > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at > > approximately the same time I see one, sometimes more, kernel > > messages: > > > > [ 6295.379283] overlayfs: upper fs does not support xattr, falling > > back to index=off and metacopy=off. > > > > Can you say why there is no xattr support? I'm not sure. It could be podman specific or fuse-overlayfs related. Maybe something is using /tmp in one case and not another for some reason? > Is the overlayfs mount executed without privileges to create trusted.* xattrs? > The answer to that may be the key to understanding the bug. Yep. I think tmpfs supports xattr but not user xattr? And this example is rootless podman, so it's all unprivileged. > My guess is it has to do with changes related to mounting overlayfs > inside userns, but I couldn't find any immediate suspects. > > Do you have any idea since when the regression appeared? > A bisect would have been helpful here. Yep. All good ideas. Thanks for the fast reply. I'll report back once this has been narrowed down futher. -- Chris Murphy ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-10 17:36 ` Chris Murphy @ 2021-04-10 17:55 ` Amir Goldstein 2021-04-10 19:36 ` Chris Murphy 2021-04-12 8:41 ` Miklos Szeredi 0 siblings, 2 replies; 14+ messages in thread From: Amir Goldstein @ 2021-04-10 17:55 UTC (permalink / raw) To: Chris Murphy; +Cc: Miklos Szeredi, Btrfs BTRFS, overlayfs On Sat, Apr 10, 2021 at 8:36 PM Chris Murphy <lists@colorremedies.com> wrote: > > I can reproduce the bolt testcase problem in a podman container, with > overlay driver, using ext4, xfs, and btrfs. So I think I can drop > linux-btrfs@ from this thread. > > Also I can reproduce the title of this thread simply by 'podman system > reset' and see the kernel messages before doing the actual reset. I > have a strace here of what it's doing: > > https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing > I'm confused. The error in the title of the page is from overlayfs mount(). I see no mount in the strace. I feel that I am missing some info. Can you provide the overlayfs mount arguments and more information about the underlying layers? > It may be something intentional. The failing testcase, > :../tests/test-common.c:1413:test_io_dir_is_empty also has more > instances of this line, but I don't know they are related. So I'll > keep looking into that. > > > On Sat, Apr 10, 2021 at 2:04 AM Amir Goldstein <amir73il@gmail.com> wrote: > > > As the first step, can you try the suggested fix to ovl_dentry_version_inc() > > and/or adding the missing pr_debug() and including those prints in > > your report? > > I'll work with bolt upstream and try to further narrow down when it is > and isn't happening. > > > > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at > > > approximately the same time I see one, sometimes more, kernel > > > messages: > > > > > > [ 6295.379283] overlayfs: upper fs does not support xattr, falling > > > back to index=off and metacopy=off. > > > > > > > Can you say why there is no xattr support? > > I'm not sure. It could be podman specific or fuse-overlayfs related. Not sure how fuse-overlayfs is related. This is a message from overlayfs kernel driver. > Maybe something is using /tmp in one case and not another for some > reason? > > > Is the overlayfs mount executed without privileges to create trusted.* xattrs? > > The answer to that may be the key to understanding the bug. > > Yep. I think tmpfs supports xattr but not user xattr? And this example > is rootless podman, so it's all unprivileged. > OK, so unprivileged overlayfs mount support was added in v5.11 and it requires opt-in with mount option "userxattr", which could explain the problem if tmpfs is used as upper layer. Do you know if that is the case? I sounds to me like it may not be a kernel regression per-se, but a regression in the container runtime that started to use a new kernel feature? Need more context to understand. Perhaps the solution will be to add user xattr support to tmpfs.. Thanks, Amir. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-10 17:55 ` Amir Goldstein @ 2021-04-10 19:36 ` Chris Murphy 2021-04-10 19:42 ` Chris Murphy 2021-04-12 8:41 ` Miklos Szeredi 1 sibling, 1 reply; 14+ messages in thread From: Chris Murphy @ 2021-04-10 19:36 UTC (permalink / raw) To: Amir Goldstein; +Cc: Chris Murphy, Miklos Szeredi, Btrfs BTRFS, overlayfs On Sat, Apr 10, 2021 at 11:55 AM Amir Goldstein <amir73il@gmail.com> wrote: > > On Sat, Apr 10, 2021 at 8:36 PM Chris Murphy <lists@colorremedies.com> wrote: > > > > I can reproduce the bolt testcase problem in a podman container, with > > overlay driver, using ext4, xfs, and btrfs. So I think I can drop > > linux-btrfs@ from this thread. > > > > Also I can reproduce the title of this thread simply by 'podman system > > reset' and see the kernel messages before doing the actual reset. I > > have a strace here of what it's doing: > > > > https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing > > > > I'm confused. The error in the title of the page is from overlayfs mount(). > I see no mount in the strace. > I feel that I am missing some info. > Can you provide the overlayfs mount arguments > and more information about the underlying layers? Not really? There are none if a container isn't running, and in this case no containers are running, in fact there are no upper or lower dirs because I had already reset podman before doing 'strace podman system reset' - I get the kernel message twice every time I merely do 'podman system reset' overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off This part of the issue might be something of a goose chase. I don't know if it's relevant or distracting. > > Yep. I think tmpfs supports xattr but not user xattr? And this example > > is rootless podman, so it's all unprivileged. > > > > OK, so unprivileged overlayfs mount support was added in v5.11 > and it requires opt-in with mount option "userxattr", which could > explain the problem if tmpfs is used as upper layer. > > Do you know if that is the case? > I sounds to me like it may not be a kernel regression per-se, > but a regression in the container runtime that started to use > a new kernel feature? > Need more context to understand. > > Perhaps the solution will be to add user xattr support to tmpfs.. $ sudo mount -o remount,userxattr /home mount: /home: mount point not mounted or bad option. [ 92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr' /home is effectively a bind mount because it is backed by a btrfs subvolume... /dev/sda6 on /home type btrfs (rw,noatime,seclabel,compress=zstd:1,ssd,space_cache=v2,subvolid=586,subvol=/home) ...which is mounted via fstab using -o subvol=home Is it supported to remount,userxattr? If not then maybe this is needed: rootflags=subvol=root,userxattr -- Chris Murphy ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-10 19:36 ` Chris Murphy @ 2021-04-10 19:42 ` Chris Murphy 2021-04-10 19:43 ` Chris Murphy 0 siblings, 1 reply; 14+ messages in thread From: Chris Murphy @ 2021-04-10 19:42 UTC (permalink / raw) To: Chris Murphy; +Cc: Amir Goldstein, Miklos Szeredi, Btrfs BTRFS, overlayfs On Sat, Apr 10, 2021 at 1:36 PM Chris Murphy <lists@colorremedies.com> wrote: > > $ sudo mount -o remount,userxattr /home > mount: /home: mount point not mounted or bad option. > > [ 92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr' > [ 63.320831] BTRFS error (device sda6): unrecognized mount option 'user_xattr' And if I try it with rootflags at boot, boot fails due to mount failure due to unrecognized mount option. -- Chris Murphy ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-10 19:42 ` Chris Murphy @ 2021-04-10 19:43 ` Chris Murphy 2021-04-10 19:44 ` Chris Murphy 0 siblings, 1 reply; 14+ messages in thread From: Chris Murphy @ 2021-04-10 19:43 UTC (permalink / raw) To: Chris Murphy; +Cc: Amir Goldstein, Miklos Szeredi, Btrfs BTRFS, overlayfs On Sat, Apr 10, 2021 at 1:42 PM Chris Murphy <lists@colorremedies.com> wrote: > > On Sat, Apr 10, 2021 at 1:36 PM Chris Murphy <lists@colorremedies.com> wrote: > > > > $ sudo mount -o remount,userxattr /home > > mount: /home: mount point not mounted or bad option. > > > > [ 92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr' > > > > [ 63.320831] BTRFS error (device sda6): unrecognized mount option 'user_xattr' > > And if I try it with rootflags at boot, boot fails due to mount > failure due to unrecognized mount option. These are all with kernel 5.12-rc6 -- Chris Murphy ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-10 19:43 ` Chris Murphy @ 2021-04-10 19:44 ` Chris Murphy 2021-04-10 20:03 ` Chris Murphy 0 siblings, 1 reply; 14+ messages in thread From: Chris Murphy @ 2021-04-10 19:44 UTC (permalink / raw) To: Chris Murphy; +Cc: Amir Goldstein, Miklos Szeredi, Btrfs BTRFS, overlayfs On Sat, Apr 10, 2021 at 1:43 PM Chris Murphy <lists@colorremedies.com> wrote: > > On Sat, Apr 10, 2021 at 1:42 PM Chris Murphy <lists@colorremedies.com> wrote: > > > > On Sat, Apr 10, 2021 at 1:36 PM Chris Murphy <lists@colorremedies.com> wrote: > > > > > > $ sudo mount -o remount,userxattr /home > > > mount: /home: mount point not mounted or bad option. > > > > > > [ 92.573364] BTRFS error (device sda6): unrecognized mount option 'userxattr' > > > > > > > [ 63.320831] BTRFS error (device sda6): unrecognized mount option 'user_xattr' > > > > And if I try it with rootflags at boot, boot fails due to mount > > failure due to unrecognized mount option. > > These are all with kernel 5.12-rc6 Ohhh to tmpfs. Hmmm. I have no idea how to do that with this test suite. I'll ask bolt folks. I'm just good at bumping into walls, obviously. -- Chris Murphy ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-10 19:44 ` Chris Murphy @ 2021-04-10 20:03 ` Chris Murphy 2021-04-11 5:12 ` Amir Goldstein 0 siblings, 1 reply; 14+ messages in thread From: Chris Murphy @ 2021-04-10 20:03 UTC (permalink / raw) To: Chris Murphy; +Cc: Amir Goldstein, Miklos Szeredi, Btrfs BTRFS, overlayfs Keeping everything else the same, and only reverting to kernel 5.9.16-200.fc33.x86_64, this kernel message >overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off no longer appears when I 'podman system reset' or when 'podman build' bolt, using the overlay driver. However, I do still get Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty: 'empty' should be FALSE -- Chris Murphy ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-10 20:03 ` Chris Murphy @ 2021-04-11 5:12 ` Amir Goldstein 2021-04-11 6:05 ` Chris Murphy 0 siblings, 1 reply; 14+ messages in thread From: Amir Goldstein @ 2021-04-11 5:12 UTC (permalink / raw) To: Chris Murphy; +Cc: Miklos Szeredi, overlayfs [removing btrfs list] On Sat, Apr 10, 2021 at 11:03 PM Chris Murphy <lists@colorremedies.com> wrote: > > Keeping everything else the same, and only reverting to kernel > 5.9.16-200.fc33.x86_64, this kernel message > > >overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off > > no longer appears when I 'podman system reset' or when 'podman build' > bolt, using the overlay driver. > I don't see a change in overlayfs that would explain seeing this warning in v5.12 and not in v5.9 - on the contrary, in v5.12 the warning is printed only if index or metacopy features have actually been requested. So it must be a change in the upper fs, which is tmpfs? Anyway, I don't have enough information and this seems unrelated to the test failure, so I'll drop it. > However, I do still get > Bail out! ERROR:../tests/test-common.c:1413:test_io_dir_is_empty: > 'empty' should be FALSE > Now I'm confused again. Your reports starts by stating: "The primary problem is Bolt (Thunderbolt 3) tests that are experiencing a regression when run in a container using overlayfs," But you say that the problem exists with kernel 5.9. When you say "regression" above, what are you referring to? Did those tests pass in a previous Bolt version? Did those tests ever pass in a container using overlayfs? There is surely a bug in overlayfs, but it's hard to find it without minimal bisection info. I'll keep looking. If it's a regression with newer distro, please try to understand from distro/package managers, what has changed in the container setup and kernel config w.r.t a container using overlayfs. Thanks, Amir. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-11 5:12 ` Amir Goldstein @ 2021-04-11 6:05 ` Chris Murphy 2021-04-11 7:28 ` Amir Goldstein 0 siblings, 1 reply; 14+ messages in thread From: Chris Murphy @ 2021-04-11 6:05 UTC (permalink / raw) To: Amir Goldstein; +Cc: Chris Murphy, Miklos Szeredi, overlayfs On Sat, Apr 10, 2021 at 11:12 PM Amir Goldstein <amir73il@gmail.com> wrote: > Now I'm confused again. So am I, and in retrospect I've posted here prematurely. > > Your reports starts by stating: > "The primary problem is Bolt (Thunderbolt 3) tests that are > experiencing a regression when run in a container using overlayfs," > > But you say that the problem exists with kernel 5.9. > When you say "regression" above, what are you referring to? Overlayfs. Now that I've tested 5.9, I'm not so sure it's a kernel regression. > > Did those tests pass in a previous Bolt version? > Did those tests ever pass in a container using overlayfs? Yes and yes. > There is surely a bug in overlayfs, but it's hard to find it without > minimal bisection info. I'll keep looking. > > If it's a regression with newer distro, please try to understand > from distro/package managers, what has changed in the container > setup and kernel config w.r.t a container using overlayfs. Exactly. The original report of the problem is Alpine linux, but I can't reproduce it on Fedora except with podman using an Alpine image base. As all the other suspects have fallen apart, what remains untested for regressions is this. -- Chris Murphy ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-11 6:05 ` Chris Murphy @ 2021-04-11 7:28 ` Amir Goldstein 2021-04-13 21:50 ` Chris Murphy 0 siblings, 1 reply; 14+ messages in thread From: Amir Goldstein @ 2021-04-11 7:28 UTC (permalink / raw) To: Chris Murphy; +Cc: Miklos Szeredi, overlayfs On Sun, Apr 11, 2021 at 9:05 AM Chris Murphy <lists@colorremedies.com> wrote: > > On Sat, Apr 10, 2021 at 11:12 PM Amir Goldstein <amir73il@gmail.com> wrote: > > > Now I'm confused again. > > So am I, and in retrospect I've posted here prematurely. > > > > > Your reports starts by stating: > > "The primary problem is Bolt (Thunderbolt 3) tests that are > > experiencing a regression when run in a container using overlayfs," > > > > But you say that the problem exists with kernel 5.9. > > When you say "regression" above, what are you referring to? > > Overlayfs. Now that I've tested 5.9, I'm not so sure it's a kernel regression. > > > > > Did those tests pass in a previous Bolt version? > > Did those tests ever pass in a container using overlayfs? > > Yes and yes. > > > There is surely a bug in overlayfs, but it's hard to find it without > > minimal bisection info. I'll keep looking. > > > > If it's a regression with newer distro, please try to understand > > from distro/package managers, what has changed in the container > > setup and kernel config w.r.t a container using overlayfs. > > Exactly. The original report of the problem is Alpine linux, but I > can't reproduce it on Fedora except with podman using an Alpine image > base. As all the other suspects have fallen apart, what remains > untested for regressions is this. > I'm lost in the maze of distros and containers. Will wait for more info. In any case, I was able to reproduce the bug in ovl_dir_version_inc() I will post a fix soon. But I don't see how the test case you reported can be affected. The bug I reproduced requires an upper directory that used to be a merge dir and whose lower dir was removed while overlayfs was offline. Thanks, Amir. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-11 7:28 ` Amir Goldstein @ 2021-04-13 21:50 ` Chris Murphy 0 siblings, 0 replies; 14+ messages in thread From: Chris Murphy @ 2021-04-13 21:50 UTC (permalink / raw) To: Amir Goldstein; +Cc: Chris Murphy, Miklos Szeredi, overlayfs On Sun, Apr 11, 2021 at 1:29 AM Amir Goldstein <amir73il@gmail.com> wrote: > > I'm lost in the maze of distros and containers. > Will wait for more info. The bolt test suite failure looks like it fuse-overlayfs related. It happens with (rootless) podman but not when using sudo podman. https://github.com/containers/fuse-overlayfs/issues/287 -- Chris Murphy ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off. 2021-04-10 17:55 ` Amir Goldstein 2021-04-10 19:36 ` Chris Murphy @ 2021-04-12 8:41 ` Miklos Szeredi 1 sibling, 0 replies; 14+ messages in thread From: Miklos Szeredi @ 2021-04-12 8:41 UTC (permalink / raw) To: Amir Goldstein; +Cc: Chris Murphy, Btrfs BTRFS, overlayfs [-- Attachment #1: Type: text/plain, Size: 2964 bytes --] On Sat, Apr 10, 2021 at 7:55 PM Amir Goldstein <amir73il@gmail.com> wrote: > > On Sat, Apr 10, 2021 at 8:36 PM Chris Murphy <lists@colorremedies.com> wrote: > > > > I can reproduce the bolt testcase problem in a podman container, with > > overlay driver, using ext4, xfs, and btrfs. So I think I can drop > > linux-btrfs@ from this thread. > > > > Also I can reproduce the title of this thread simply by 'podman system > > reset' and see the kernel messages before doing the actual reset. I > > have a strace here of what it's doing: > > > > https://drive.google.com/file/d/1L9lEm5n4-d9qemgCq3ijqoBstM-PP1By/view?usp=sharing > > > > I'm confused. The error in the title of the page is from overlayfs mount(). > I see no mount in the strace. > I feel that I am missing some info. > Can you provide the overlayfs mount arguments > and more information about the underlying layers? > > > It may be something intentional. The failing testcase, > > :../tests/test-common.c:1413:test_io_dir_is_empty also has more > > instances of this line, but I don't know they are related. So I'll > > keep looking into that. > > > > > > On Sat, Apr 10, 2021 at 2:04 AM Amir Goldstein <amir73il@gmail.com> wrote: > > > > > As the first step, can you try the suggested fix to ovl_dentry_version_inc() > > > and/or adding the missing pr_debug() and including those prints in > > > your report? > > > > I'll work with bolt upstream and try to further narrow down when it is > > and isn't happening. > > > > > > I can reproduce this with 5.12.0-0.rc6.184.fc35.x86_64+debug and at > > > > approximately the same time I see one, sometimes more, kernel > > > > messages: > > > > > > > > [ 6295.379283] overlayfs: upper fs does not support xattr, falling > > > > back to index=off and metacopy=off. > > > > > > > > > > Can you say why there is no xattr support? > > > > I'm not sure. It could be podman specific or fuse-overlayfs related. > > Not sure how fuse-overlayfs is related. > This is a message from overlayfs kernel driver. > > > Maybe something is using /tmp in one case and not another for some > > reason? > > > > > Is the overlayfs mount executed without privileges to create trusted.* xattrs? > > > The answer to that may be the key to understanding the bug. > > > > Yep. I think tmpfs supports xattr but not user xattr? And this example > > is rootless podman, so it's all unprivileged. > > > > OK, so unprivileged overlayfs mount support was added in v5.11 > and it requires opt-in with mount option "userxattr", which could > explain the problem if tmpfs is used as upper layer. > > Do you know if that is the case? > I sounds to me like it may not be a kernel regression per-se, > but a regression in the container runtime that started to use > a new kernel feature? > Need more context to understand. > > Perhaps the solution will be to add user xattr support to tmpfs.. Attached patch. Tested at some earlier time, since I also bumped into this issue. Thanks, Miklos [-- Attachment #2: tmpfs-allow-user-xattr.patch --] [-- Type: text/x-patch, Size: 737 bytes --] diff --git a/mm/shmem.c b/mm/shmem.c index d722eb830317..afe59086b3f6 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3250,6 +3250,12 @@ static const struct xattr_handler shmem_trusted_xattr_handler = { .set = shmem_xattr_handler_set, }; +static const struct xattr_handler shmem_user_xattr_handler = { + .prefix = XATTR_USER_PREFIX, + .get = shmem_xattr_handler_get, + .set = shmem_xattr_handler_set, +}; + static const struct xattr_handler *shmem_xattr_handlers[] = { #ifdef CONFIG_TMPFS_POSIX_ACL &posix_acl_access_xattr_handler, @@ -3257,6 +3263,7 @@ static const struct xattr_handler *shmem_xattr_handlers[] = { #endif &shmem_security_xattr_handler, &shmem_trusted_xattr_handler, + &shmem_user_xattr_handler, NULL }; ^ permalink raw reply related [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-04-13 21:50 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-04-09 22:03 btrfs+overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off Chris Murphy 2021-04-10 8:03 ` Amir Goldstein 2021-04-10 17:36 ` Chris Murphy 2021-04-10 17:55 ` Amir Goldstein 2021-04-10 19:36 ` Chris Murphy 2021-04-10 19:42 ` Chris Murphy 2021-04-10 19:43 ` Chris Murphy 2021-04-10 19:44 ` Chris Murphy 2021-04-10 20:03 ` Chris Murphy 2021-04-11 5:12 ` Amir Goldstein 2021-04-11 6:05 ` Chris Murphy 2021-04-11 7:28 ` Amir Goldstein 2021-04-13 21:50 ` Chris Murphy 2021-04-12 8:41 ` Miklos Szeredi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).