overlayfs 4.19: unable to make backup link: Invalid cross-device link

overlayfs 4.19: unable to make backup link: Invalid cross-device link

[Theodore Y. Ts'o]

I was rebuilding my dropbox Docker container, and it failed in 4.19
with the following error:

dpkg: error processing archive /var/cache/apt/archives/perl-base_5.20.2-3+deb8u11_amd64.deb (--unpack):
 unable to make backup link of `./usr/bin/perl5.20.2' before installing new version: Invalid cross-device link
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
E: Sub-process /usr/bin/dpkg returned an error code (2)

When I booted the 4.18.0-2 kernel from Debian, the exact same Docker
file worked just fine.  With 4.19 it fails reliably.

To reproduce:

git clone https://github.com/tytso/docker-dropbox
cd docker-dropbox
docker build --no-cache .

I'm using a Debian testing (buster) userspace, using the docker-ce
from dropbox.com, version 18.03.1~ce-0~debian, using overlay2 storage
driverh.

Does this ring a bell at all?  Can someone give me some tips on how to
troubleshoot this?

Thanks!!

     	       	      	       	   - Ted

Re: overlayfs 4.19: unable to make backup link: Invalid cross-device link

[Amir Goldstein]

On Sat, Oct 27, 2018 at 11:05 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> I was rebuilding my dropbox Docker container, and it failed in 4.19
> with the following error:
>
> dpkg: error processing archive /var/cache/apt/archives/perl-base_5.20.2-3+deb8u11_amd64.deb (--unpack):
>  unable to make backup link of `./usr/bin/perl5.20.2' before installing new version: Invalid cross-device link
> dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
> dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
> E: Sub-process /usr/bin/dpkg returned an error code (2)
>

When that happens can you please 'stat /usr/bin/perl{5.20.2}', in
lower upper and overlay.
Assuming link does not exist in neither lower nor upper, please try to create
/usr/bin/perl5.20.2 link manually from /usr/bin/perl on overlay.

> When I booted the 4.18.0-2 kernel from Debian, the exact same Docker
> file worked just fine.  With 4.19 it fails reliably.
>

Strange, I expected many things to behave differently with v4.19 due to
stacked f_ops . link() was not one of them...

> To reproduce:
>
> git clone https://github.com/tytso/docker-dropbox
> cd docker-dropbox
> docker build --no-cache .
>
> I'm using a Debian testing (buster) userspace, using the docker-ce
> from dropbox.com, version 18.03.1~ce-0~debian, using overlay2 storage
> driverh.
>

What file system are you using for /var/lib/docker? ext4?
Is your docker overlay2 driver maybe configured to limit container
size with project quotas?
I don't know if there is a default for that, but anyway, there is no
ext4 project quota
support in docker AFAIK (only xfs project quotas).

> Does this ring a bell at all?  Can someone give me some tips on how to
> troubleshoot this?
>

I tried to reproduce on a xfstests-bld image with chroot and dpkg, but
didn't get far.
I get this warning with or without overlayfs:
update-alternatives: warning: forcing reinstallation of alternative
/usr/bin/prename
because link group rename is broken
I'll try to setup a new vm to test your reproducer.

Thanks,
Amir.

Re: overlayfs 4.19: unable to make backup link: Invalid cross-device link

[Miklos Szeredi]

On Sat, Oct 27, 2018 at 10:04 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> I was rebuilding my dropbox Docker container, and it failed in 4.19
> with the following error:
>
> dpkg: error processing archive /var/cache/apt/archives/perl-base_5.20.2-3+deb8u11_amd64.deb (--unpack):
>  unable to make backup link of `./usr/bin/perl5.20.2' before installing new version: Invalid cross-device link
> dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
> dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
> E: Sub-process /usr/bin/dpkg returned an error code (2)
>
> When I booted the 4.18.0-2 kernel from Debian, the exact same Docker
> file worked just fine.  With 4.19 it fails reliably.

Can you please post

  grep OVERLAY_FS .config

?

I'd suspect METACOPY being enabled and regressing (since it's a new feature).

Thanks,
Mikos

Re: overlayfs 4.19: unable to make backup link: Invalid cross-device link

[Theodore Y. Ts'o]

On Mon, Oct 29, 2018 at 09:31:58AM +0100, Miklos Szeredi wrote:
> Can you please post
> 
>   grep OVERLAY_FS .config

Sure, here you go:

% zgrep OVERLAY_FS /proc/config.gz
CONFIG_OVERLAY_FS=m
CONFIG_OVERLAY_FS_REDIRECT_DIR=y
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y                                        
CONFIG_OVERLAY_FS_INDEX=y
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
CONFIG_OVERLAY_FS_METACOPY=y

And this was the config on the working 4.18 kernel:

% grep OVERLAY_FS /boot/config-4.18.0-2-amd64 
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set

The underlying file system is ext4.

Thanks!

					- Ted

Re: overlayfs 4.19: unable to make backup link: Invalid cross-device link

[Amir Goldstein]

On Tue, Oct 30, 2018 at 12:05 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Mon, Oct 29, 2018 at 09:31:58AM +0100, Miklos Szeredi wrote:
> > Can you please post
> >
> >   grep OVERLAY_FS .config
>
> Sure, here you go:
>
> % zgrep OVERLAY_FS /proc/config.gz
> CONFIG_OVERLAY_FS=m
> CONFIG_OVERLAY_FS_REDIRECT_DIR=y
> CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
> CONFIG_OVERLAY_FS_INDEX=y
> # CONFIG_OVERLAY_FS_XINO_AUTO is not set
> CONFIG_OVERLAY_FS_METACOPY=y
>
> And this was the config on the working 4.18 kernel:
>
> % grep OVERLAY_FS /boot/config-4.18.0-2-amd64
> CONFIG_OVERLAY_FS=m
> # CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
> CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
> # CONFIG_OVERLAY_FS_INDEX is not set
> # CONFIG_OVERLAY_FS_XINO_AUTO is not set
>

So let's try to bisect the offending feature first.
If we can't find an offending feature, then there may be an offending
commit in 4.18..4.19

With kernel 4.19, please try to reproduce after:
echo N > /sys/module/overlay/parameters/metacopy
and then after:
echo N > /sys/module/overlay/parameters/index
and then after:
echo N > /sys/module/overlay/parameters/redirect_dir

With kernel 4.18, please try to reproduce after:
echo Y > /sys/module/overlay/parameters/redirect_dir
and then after:
echo Y > /sys/module/overlay/parameters/index

My immediate suspect w.r.t hardlink issues would be the index feature.

FYI, I had already verified that unionmount testsuite passes on kernel 4.19
with same overlay config as your kernel 4.19.

Thanks,
Amir.

Re: overlayfs 4.19: unable to make backup link: Invalid cross-device link

[Amir Goldstein]

On Tue, Oct 30, 2018 at 6:44 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
...
>
> So let's try to bisect the offending feature first.
> If we can't find an offending feature, then there may be an offending
> commit in 4.18..4.19
>
> With kernel 4.19, please try to reproduce after:
> echo N > /sys/module/overlay/parameters/metacopy
> and then after:

Miklos, Vivek,

FYI, Maciej has reported that he reproduced the same issue and that
disabling metacopy fixes the problem.

He offered to provide more information if requested.

Thanks,
Amir.

Re: overlayfs 4.19: unable to make backup link: Invalid cross-device link

[Miklos Szeredi]

On Wed, Nov 14, 2018 at 10:46 AM, Amir Goldstein <amir73il@gmail.com> wrote:
> On Tue, Oct 30, 2018 at 6:44 AM Amir Goldstein <amir73il@gmail.com> wrote:
>>
> ...
>>
>> So let's try to bisect the offending feature first.
>> If we can't find an offending feature, then there may be an offending
>> commit in 4.18..4.19
>>
>> With kernel 4.19, please try to reproduce after:
>> echo N > /sys/module/overlay/parameters/metacopy
>> and then after:
>
> Miklos, Vivek,
>
> FYI, Maciej has reported that he reproduced the same issue and that
> disabling metacopy fixes the problem.
>
> He offered to provide more information if requested.

Maciej,

Can you please try reproduce with overlay debugging turned on:

    # echo "file fs/overlayfs/* +p" > <debugfs>/dynamic_debug/control

and post the relevant part of dmesg output?

Thanks,
Miklos


>
> Thanks,
> Amir.

Re: overlayfs 4.19: unable to make backup link: Invalid cross-device link

[Amir Goldstein]

On Wed, Nov 14, 2018 at 12:28 PM Maciej Zięba <maciekz82@gmail.com> wrote:
>
> Hi :)
>
> I'm not sure which part is relevant, so I'm sending the entire dmesg output as well as the docker messages. I hope it will be useful.
>

Thanks for the info!

Problem is ovl_set_redirect() inside ovl_link() is missing
ovl_override_creds().

Can you try the patch below?

Reason that unionmount-testsuite didn't catch this is because
it is run as root.

Vivek, Mark,

Either of you care to implement dropping privileges for the duration
of unionmount-testsuite test cases (in between mounts and unmounts)?

Thanks,
Amir.

---

diff --git a/fs/overlayfs/dir.c b/fs/overlayfs/dir.c
index c6289147c787..82c129bfe58d 100644
--- a/fs/overlayfs/dir.c
+++ b/fs/overlayfs/dir.c
@@ -651,6 +651,18 @@ static int ovl_symlink(struct inode *dir, struct
dentry *dentry,
        return ovl_create_object(dentry, S_IFLNK, 0, link);
 }

+static int ovl_set_link_redirect(struct dentry *dentry)
+{
+       const struct cred *old_cred;
+       int err;
+
+       old_cred = ovl_override_creds(dentry->d_sb);
+       err = ovl_set_redirect(dentry, false);
+       revert_creds(old_cred);
+
+       return err;
+}
+
 static int ovl_link(struct dentry *old, struct inode *newdir,
                    struct dentry *new)
 {
@@ -670,7 +682,7 @@ static int ovl_link(struct dentry *old, struct
inode *newdir,
                goto out_drop_write;

        if (ovl_is_metacopy_dentry(old)) {
-               err = ovl_set_redirect(old, false);
+               err = ovl_set_link_redirect(old);
                if (err)
                        goto out_drop_write;
        }

Re: overlayfs 4.19: unable to make backup link: Invalid cross-device link

[Vivek Goyal]

On Wed, Nov 14, 2018 at 02:09:01PM +0200, Amir Goldstein wrote:

[..]
> Vivek, Mark,
> 
> Either of you care to implement dropping privileges for the duration
> of unionmount-testsuite test cases (in between mounts and unmounts)?

Hi Amir,

I think it will be good to drop priviliges while running
unionmount-testsuite.

BTW, docker build used to run as root/priviliged user. Is that not the case
anymore? Why did it fail to begin with.

Thanks
Vivek

> 
> Thanks,
> Amir.
> 
> ---
> 
> diff --git a/fs/overlayfs/dir.c b/fs/overlayfs/dir.c
> index c6289147c787..82c129bfe58d 100644
> --- a/fs/overlayfs/dir.c
> +++ b/fs/overlayfs/dir.c
> @@ -651,6 +651,18 @@ static int ovl_symlink(struct inode *dir, struct
> dentry *dentry,
>         return ovl_create_object(dentry, S_IFLNK, 0, link);
>  }
> 
> +static int ovl_set_link_redirect(struct dentry *dentry)
> +{
> +       const struct cred *old_cred;
> +       int err;
> +
> +       old_cred = ovl_override_creds(dentry->d_sb);
> +       err = ovl_set_redirect(dentry, false);
> +       revert_creds(old_cred);
> +
> +       return err;
> +}
> +
>  static int ovl_link(struct dentry *old, struct inode *newdir,
>                     struct dentry *new)
>  {
> @@ -670,7 +682,7 @@ static int ovl_link(struct dentry *old, struct
> inode *newdir,
>                 goto out_drop_write;
> 
>         if (ovl_is_metacopy_dentry(old)) {
> -               err = ovl_set_redirect(old, false);
> +               err = ovl_set_link_redirect(old);
>                 if (err)
>                         goto out_drop_write;
>         }