Re: [kernel]:overlayfs: invalid origin (root/bmcweb_persistent_data.json, ftype=8000, origin ftype=4000)

Re: [kernel]:overlayfs: invalid origin (root/bmcweb_persistent_data.json, ftype=8000, origin ftype=4000)

[Amir Goldstein]

On Thu, Apr 15, 2021 at 2:03 PM www <ouyangxuan10@163.com> wrote:
>
> Dear Amir, kevin, miklos,
>
> An open source project uses overlay filesystem and systemd. when the system first time boot, there is an error in the file system:
> kernel: overlayfs: invalid origin (root/bmcweb_persistent_data.json, ftype=8000, origin ftype=4000)
>
> What's the reason for this? Can you give some help and suggestions to solve this problem?
>
> The details are as follows:
> 1. linux kernel version:
> root@N/A:~# uname -a Linux N/A 5.1.5-yocto-s-dirty-1605363 #1 Tue Apr 13 01:13:48 UTC 2021 armv6l GNU/Linux
>
> 2. mount command
> mount -t overlay -o lowerdir=$rodir,upperdir=$upper,workdir=$work cow /root
> (rodir=run/initramfs/ro, upper=run/initramfs/rw/cow, work=rwdir=run/initramfs/rw/work)
>
> 3. mountinfo
> root@N/A:~# cat /proc/self/mountinfo
> 13 21 0:6 / /dev rw,relatime shared:8 - devtmpfs dev rw,size=220800k,nr_inodes=55200,mode=755
> 14 21 0:13 / /sys rw,relatime shared:3 - sysfs sys rw
> 15 21 0:4 / /proc rw,relatime shared:2 - proc proc rw
> 16 21 0:14 / /run rw,nosuid,nodev shared:11 - tmpfs tmpfs rw,mode=755
> 17 16 7:0 / /run/initramfs/ro ro,relatime shared:12 - squashfs /dev/loop0 ro
> 18 16 31:5 / /run/initramfs/rw rw,relatime shared:13 - jffs2 /dev/mtdblock5 rw
> 21 0 0:15 / / rw,relatime shared:1 - overlay cow rw,lowerdir=run/initramfs/ro,upperdir=run/initramfs/rw/cow,workdir=run/initramfs/rw/work
> 22 13 0:17 / /dev/shm rw,nosuid,nodev shared:9 - tmpfs tmpfs rw
> 23 13 0:18 / /dev/pts rw,relatime shared:10 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
> 24 14 0:19 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:4 - tmpfs tmpfs ro,mode=755
> 25 24 0:20 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:5 - cgroup2 cgroup2 rw,nsdelegate
> 26 24 0:21 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:6 - cgroup cgroup rw,xattr,name=systemd
> 27 14 0:22 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:7 - bpf bpf rw,mode=700
> 28 14 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:14 - debugfs debugfs rw
> 29 14 0:12 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:15 - configfs configfs rw
> 30 21 0:23 / /tmp rw,nosuid,nodev shared:16 - tmpfs tmpfs rw
> 31 21 0:24 / /var/volatile rw,relatime shared:17 - tmpfs tmpfs rw
>
> 4. xino_auto
>
> 5. file in ro is correct
> 6. file in rw is correct
> 7. but the file in overlayfs is  incorrect
>
> What's the reason for this?

I suppose lower squashfs was recreated and mounted with an existing
upper dir that already contains upper files with origin references to the
old squashfs lower fs.

> Can you give some help and suggestions to solve this problem?

Cherry-pick this fix to your kernel:
a888db310195 ovl: fix regression with re-formatted lower squashfs

OR as a workaround, build the lower squashfs with the "-no-exports"
mksquashfs option.

Thanks,
Amir.

Re: Re:Re: [kernel]:overlayfs: invalid origin (root/bmcweb_persistent_data.json, ftype=8000, origin ftype=4000)

[Amir Goldstein]

On Tue, Apr 27, 2021 at 11:40 AM www <ouyangxuan10@163.com> wrote:
>
> Hi Amir,
>
> To solve this problem, we are trying to upgrade the kernel version or upgrade the file system separately, but because the kernel version we are using is quite different from the one that solves the problem. So I would like to ask you a few questions:
>
> 1. What is the root cause of this problem?

As I wrote:
I suppose lower squashfs was recreated and mounted with an existing
upper dir that already contains upper files with origin references to the
old squashfs lower fs.

>
> 2. Can we just upgrade overlay fs to solve this problem without upgrading the kernel or all file systems?
>

No.

> 3. If we upgrade overlayfs separately, we are not very good at verifying that we have solved this problem, because the recurrence probability of this problem is very low. So I want to ask, how can we quickly reproduce this problem?

Re-creating a lower squashfs after files have been copied to upper should
reproduce the problem quite often.

>
> 4. Do you have any good suggestions?
>

Is the creation of the lower image under your control?
Did you try the workaround I suggested to create lower squashfs
with the "-no-exports" mksquashfs option?

Try this patch:

diff --git a/fs/overlayfs/namei.c b/fs/overlayfs/namei.c
index 71e264e2f16b..850c0a37f1f0 100644
--- a/fs/overlayfs/namei.c
+++ b/fs/overlayfs/namei.c
@@ -392,7 +392,7 @@ int ovl_check_origin_fh(struct ovl_fs *ofs, struct
ovl_fh *fh, bool connected,
                            upperdentry, d_inode(upperdentry)->i_mode & S_IFMT,
                            d_inode(origin)->i_mode & S_IFMT);
        dput(origin);
-       return -EIO;
+       return -EINVAL;
 }

 static int ovl_check_origin(struct ovl_fs *ofs, struct dentry *upperdentry,
@@ -408,7 +408,7 @@ static int ovl_check_origin(struct ovl_fs *ofs,
struct dentry *upperdentry,
        kfree(fh);

        if (err) {
-               if (err == -ESTALE)
+               if (err == -ESTALE || err == -EINVAL)
                        return 0;
                return err;
        }

It should apply to any kernel and if not it's quite easy to manually apply it.
This fix will not get rid of the warning printed to kmsg, but it will hide the
error from userspace.

If it fixes your problem I think we should apply it to upstream and stable
kernels as well.

Thanks,
Amir.