ovl: renames in overlayfs over tmpfs returns ESTALE in 4.4

ovl: renames in overlayfs over tmpfs returns ESTALE in 4.4

[Mikhail Afanasyev]

Hello,

I believe the  4.4.y kernel has broken overlayfs: removing the file 
which is present in lower layer, then renaming a file from upper layer 
to have the same name fails with ESTALE. This pattern is used quite 
often, for example by 'depmod' and 'apt-get update'. The bug only seems 
to happen when some of the underlying directories are on tmpfs.

The test script (below) works fine on Ubuntu kernel 4.4.0-34-generic or 
earlier, and it fails in Ubuntu kernel 4.4.0-36-generic. This 
corresponds to Linux 4.4.16.

The error is:
mv: cannot move 'ovf/file.new' to 'ovf/file': Stale file handle

While I have not performed the full bisect, there were only 3 patches 
touching 'fs/overlayfs' which went into the tree between the two revisions:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/log/?id=b275624b3b4303d21eefd78c5b3bf04c34267e4e

ovl: verify upper dentry before unlink and rename
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?id=a4452ab01db4780937d4e19861d3b841c373c3b5

ovl: verify upper dentry in ovl_remove_and_whiteout()
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?id=90089541cf2982cff69724c34f7ae9a088df9c2f

ovl: Copy up underlying inode's ->i_mode to overlay inode
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?id=8326f3ef42b56d97d1060adff6d4ab35818e3b9a

Of them, the first two add a return with ESTALE error and would be the 
prime suspects.
I also understand that 4.4.16 is a somewhat old kernel, but I believe 
the bug may still be present in the latest linux-4.4.y  -- there were no 
changes to the relevant file (fs/overlayfs/dir.c) since these patches.



Mikhail Afanasyev

~~~~~~~~ TEST SCRIPT ~~~~~~
#!/bin/sh -ex
mkdir -p /tmp/ovf-bug-demo
cd /tmp/ovf-bug-demo

umount ./tmpfs/ovf || true
umount ./tmpfs || true
mkdir -p tmpfs
mount -t tmpfs none tmpfs
cd tmpfs

mkdir -p upper lower ovf work
touch lower/file
mount -t overlayfs none -o 
rw,noatime,lowerdir=$PWD/lower,upperdir=$PWD/upper,workdir=$PWD/work 
$PWD/ovf
touch ovf/file.new

rm ovf/file
mv ovf/file.new ovf/file

~~~~~~~~~ END ~~~~~~~~~~~~

Re: ovl: renames in overlayfs over tmpfs returns ESTALE in 4.4

[Greg Kroah-Hartman]

On Tue, Sep 13, 2016 at 01:48:31PM -0400, Mikhail Afanasyev wrote:
> Hello,
> 
> I believe the  4.4.y kernel has broken overlayfs: removing the file which is
> present in lower layer, then renaming a file from upper layer to have the
> same name fails with ESTALE. This pattern is used quite often, for example
> by 'depmod' and 'apt-get update'. The bug only seems to happen when some of
> the underlying directories are on tmpfs.
> 
> The test script (below) works fine on Ubuntu kernel 4.4.0-34-generic or
> earlier, and it fails in Ubuntu kernel 4.4.0-36-generic. This corresponds to
> Linux 4.4.16.

Ubuntu adds a few hundred other patches to their kernel as well, can you
try a "real" 4.4-stable release please?

> The error is:
> mv: cannot move 'ovf/file.new' to 'ovf/file': Stale file handle
> 
> While I have not performed the full bisect, there were only 3 patches
> touching 'fs/overlayfs' which went into the tree between the two revisions:
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/log/?id=b275624b3b4303d21eefd78c5b3bf04c34267e4e
> 
> ovl: verify upper dentry before unlink and rename
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?id=a4452ab01db4780937d4e19861d3b841c373c3b5
> 
> ovl: verify upper dentry in ovl_remove_and_whiteout()
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?id=90089541cf2982cff69724c34f7ae9a088df9c2f
> 
> ovl: Copy up underlying inode's ->i_mode to overlay inode
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?id=8326f3ef42b56d97d1060adff6d4ab35818e3b9a
> 
> Of them, the first two add a return with ESTALE error and would be the prime
> suspects.
> I also understand that 4.4.16 is a somewhat old kernel, but I believe the
> bug may still be present in the latest linux-4.4.y  -- there were no changes
> to the relevant file (fs/overlayfs/dir.c) since these patches.

In the 4.4 kernel to be released tomorrow, there are a number of ovfs
patches, can you test that out?

thanks,

greg k-h

Re: ovl: renames in overlayfs over tmpfs returns ESTALE in 4.4

[Stefan Bader]

[-- Attachment #1: Type: text/plain, Size: 2584 bytes --]

On 13.09.2016 19:48, Mikhail Afanasyev wrote:
> Hello,
> 
> I believe the  4.4.y kernel has broken overlayfs: removing the file which is
> present in lower layer, then renaming a file from upper layer to have the same
> name fails with ESTALE. This pattern is used quite often, for example by
> 'depmod' and 'apt-get update'. The bug only seems to happen when some of the
> underlying directories are on tmpfs.

This was a regression related to compat code which is only in the Ubuntu kernels
and will be fixed in the next update. See:

http://bugs.launchpad.net/bugs/1618572

-Stefan

> 
> The test script (below) works fine on Ubuntu kernel 4.4.0-34-generic or earlier,
> and it fails in Ubuntu kernel 4.4.0-36-generic. This corresponds to Linux 4.4.16.
> 
> The error is:
> mv: cannot move 'ovf/file.new' to 'ovf/file': Stale file handle
> 
> While I have not performed the full bisect, there were only 3 patches touching
> 'fs/overlayfs' which went into the tree between the two revisions:
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/log/?id=b275624b3b4303d21eefd78c5b3bf04c34267e4e
> 
> 
> ovl: verify upper dentry before unlink and rename
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?id=a4452ab01db4780937d4e19861d3b841c373c3b5
> 
> 
> ovl: verify upper dentry in ovl_remove_and_whiteout()
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?id=90089541cf2982cff69724c34f7ae9a088df9c2f
> 
> 
> ovl: Copy up underlying inode's ->i_mode to overlay inode
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/?id=8326f3ef42b56d97d1060adff6d4ab35818e3b9a
> 
> 
> Of them, the first two add a return with ESTALE error and would be the prime
> suspects.
> I also understand that 4.4.16 is a somewhat old kernel, but I believe the bug
> may still be present in the latest linux-4.4.y  -- there were no changes to the
> relevant file (fs/overlayfs/dir.c) since these patches.
> 
> 
> 
> Mikhail Afanasyev
> 
> ~~~~~~~~ TEST SCRIPT ~~~~~~
> #!/bin/sh -ex
> mkdir -p /tmp/ovf-bug-demo
> cd /tmp/ovf-bug-demo
> 
> umount ./tmpfs/ovf || true
> umount ./tmpfs || true
> mkdir -p tmpfs
> mount -t tmpfs none tmpfs
> cd tmpfs
> 
> mkdir -p upper lower ovf work
> touch lower/file
> mount -t overlayfs none -o
> rw,noatime,lowerdir=$PWD/lower,upperdir=$PWD/upper,workdir=$PWD/work $PWD/ovf
> touch ovf/file.new
> 
> rm ovf/file
> mv ovf/file.new ovf/file
> 
> ~~~~~~~~~ END ~~~~~~~~~~~~



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]