flock fails in overlay nfs-exported file

flock fails in overlay nfs-exported file

[Eddie Horng]

Hello Miklos,
I'd like to report a flock(2) problem to overlay nfs-exported files.
The error return from flock(2) is "Bad file descriptor".

Environment:
OS: Ubuntu 14.04.2 LTS
Kernel: 4.16.0-041600rc4-generic (from
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4/)

Reproduce step:
(nfs server side)
mount -t overlay
-orw,lowerdir=/mnt/ro,upperdir=/mnt/u,workdir=/mnt/w,nfs_export=on,index=on
none /mnt/m
touch /mnt/m/foo
(nfs client side)
mount server:/mnt/m /mnt/n

flock /mnt/n/foo
failed to lock file '/mnt/n/foo': Bad file descriptor

The simple flock(2) wrap program is:
int main(int argc, void **argv) {
char *filename=argv[1];
int fd = open(filename, O_RDONLY);;
int flock_result = flock(fd, LOCK_EX | LOCK_NB);
if (flock_result != 0) {
      printf("failed to lock file '%s': %s\n", filename, strerror(errno));
      goto out;
    }
    printf("flock succeed\n");
out:
    close(fd);
}

Thanks!

Re: flock fails in overlay nfs-exported file

[Amir Goldstein]

On Mon, Mar 12, 2018 at 9:38 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> Hello Miklos,
> I'd like to report a flock(2) problem to overlay nfs-exported files.
> The error return from flock(2) is "Bad file descriptor".
>
> Environment:
> OS: Ubuntu 14.04.2 LTS
> Kernel: 4.16.0-041600rc4-generic (from
> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4/)
>
> Reproduce step:
> (nfs server side)
> mount -t overlay
> -orw,lowerdir=/mnt/ro,upperdir=/mnt/u,workdir=/mnt/w,nfs_export=on,index=on
> none /mnt/m
> touch /mnt/m/foo
> (nfs client side)
> mount server:/mnt/m /mnt/n
>
> flock /mnt/n/foo
> failed to lock file '/mnt/n/foo': Bad file descriptor
>

Does not reproduce on my end. I am using v4.16-rc5, but I don't think
any of the fixes there are relevant to this failure.

This is what I have for underlying fs, overlay and nfs mount options
(index and nfs_export are on by default in my kernel):

/dev/mapper/storage-lower_layer on /base type xfs
(rw,relatime,attr2,inode64,noquota)
share on /share type overlay
(rw,relatime,lowerdir=/base/lower,upperdir=/base/upper/0,workdir=/base/upper/work0)
c800:/share on /mnt/t type nfs
(rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.91.126,mountvers=3,mountport=49494,mountproto=udp,local_lock=none,addr=192.168.91.126)

$ touch /mnt/t/foo
$ flock -x -n /mnt/t/foo echo locked
locked

Please share more information about nfs mount options and underlying filesystem

Please check if you see any relevant errors/warnings in dmesg.

Thanks,
Amir.

Re: flock fails in overlay nfs-exported file

[Eddie Horng]

Hi Amir,
Thanks your prompt response. After compare flock(1) and my flock(2)
test program, it seems open flag makes the result different. strace
result shows open with O_RDONLY flock fails (case A), open with
O_RDWR|O_CREAT|O_NOCTTY flock works (case B) and open local ext4 file
with O_RDONLY flock works too (case C)

case A:
strace myflock /mnt/n/foo
open("/mnt/n/foo", O_RDONLY)            = 3
flock(3, LOCK_EX|LOCK_NB)               = -1 EBADF (Bad file descriptor)

case B:
strace flock -x -n /mnt/n/foo echo locked
open("/mnt/n/foo", O_RDWR|O_CREAT|O_NOCTTY, 0666) = 3
flock(3, LOCK_EX|LOCK_NB)               = 0

case C:
strace myflock /tmp/t
open("/tmp/t", O_RDONLY)                = 3
flock(3, LOCK_EX|LOCK_NB)               = 0

Below is my test configuration of case A:
- underlying filesystem:
ext4
- /proc/mounts:
/dev/disk/by-uuid/a2d5005c-.... / ext4
rw,relatime,errors=remount-ro,data=ordered 0 0
none /share overlay
rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base/work,index=on,nfs_export=on
0 0
localhost:/share /mnt/n nfs4
rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1
0 0
- /etc/exports
/share *(rw,sync,no_subtree_check,no_root_squash,fsid=41)


For dmesg, in case A, there's no any output from dmesg, however in my
applications running with overlay nfs exported files, there are  some
lock related messages. Which lock call triggers it, need more
investigation.
The message from nfs server side is like:
[  872.940080] Leaked POSIX lock on dev=0x0:0x42 ino=0xf5a1
fl_owner=0000000023265f44 fl_flags=0x1 fl_type=0x1 fl_pid=1
[ 1939.829655] Leaked locks on dev=0x0:0x42 ino=0xf5a1:
[ 1939.829659] POSIX: fl_owner=0000000023265f44 fl_flags=0x1
fl_type=0x1 fl_pid=1

thanks,
Eddie

2018-03-12 20:07 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
> On Mon, Mar 12, 2018 at 9:38 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>> Hello Miklos,
>> I'd like to report a flock(2) problem to overlay nfs-exported files.
>> The error return from flock(2) is "Bad file descriptor".
>>
>> Environment:
>> OS: Ubuntu 14.04.2 LTS
>> Kernel: 4.16.0-041600rc4-generic (from
>> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4/)
>>
>> Reproduce step:
>> (nfs server side)
>> mount -t overlay
>> -orw,lowerdir=/mnt/ro,upperdir=/mnt/u,workdir=/mnt/w,nfs_export=on,index=on
>> none /mnt/m
>> touch /mnt/m/foo
>> (nfs client side)
>> mount server:/mnt/m /mnt/n
>>
>> flock /mnt/n/foo
>> failed to lock file '/mnt/n/foo': Bad file descriptor
>>
>
> Does not reproduce on my end. I am using v4.16-rc5, but I don't think
> any of the fixes there are relevant to this failure.
>
> This is what I have for underlying fs, overlay and nfs mount options
> (index and nfs_export are on by default in my kernel):
>
> /dev/mapper/storage-lower_layer on /base type xfs
> (rw,relatime,attr2,inode64,noquota)
> share on /share type overlay
> (rw,relatime,lowerdir=/base/lower,upperdir=/base/upper/0,workdir=/base/upper/work0)
> c800:/share on /mnt/t type nfs
> (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.91.126,mountvers=3,mountport=49494,mountproto=udp,local_lock=none,addr=192.168.91.126)
>
> $ touch /mnt/t/foo
> $ flock -x -n /mnt/t/foo echo locked
> locked
>
> Please share more information about nfs mount options and underlying filesystem
>
> Please check if you see any relevant errors/warnings in dmesg.
>
> Thanks,
> Amir.

Re: flock fails in overlay nfs-exported file

[Amir Goldstein]

[CC some NFS/lock folks (see history below top post)]

On Tue, Mar 13, 2018 at 3:39 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> Hi Amir,
> Thanks your prompt response. After compare flock(1) and my flock(2)
> test program, it seems open flag makes the result different. strace
> result shows open with O_RDONLY flock fails (case A), open with
> O_RDWR|O_CREAT|O_NOCTTY flock works (case B) and open local ext4 file
> with O_RDONLY flock works too (case C)
>
> case A:
> strace myflock /mnt/n/foo
> open("/mnt/n/foo", O_RDONLY)            = 3
> flock(3, LOCK_EX|LOCK_NB)               = -1 EBADF (Bad file descriptor)
>

It looks like flock(1) has special code to handle this case for NFSv4
and fall back to open O_RDRW:
https://github.com/karelzak/util-linux/blob/master/sys-utils/flock.c#L295

Although I tested with NFSv3 and open flags used by flock(1)
where O_RDONLY|O_CREAT|O_NOCTTY

Why do you need to get an exclusive lock on a file that is open for read?
Can you open the file for write and resolve the issue like flock(1) does?

You should know that even if you manage to lock a O_RDONLY fd,
if this file is then open for write by another process, that process will
get a file descriptor pointing to a *different* inode.
This is a long standing issue with overlayfs (inconsistent ro/rw fd),
which is being worked around by some user applications -
i.e. touch the file before first access to avoid applications
getting open file descriptor to lower inode.

Let me know if this answer suffice or if you get this error only
with NFSv4 over overalyfs.

> case B:
> strace flock -x -n /mnt/n/foo echo locked
> open("/mnt/n/foo", O_RDWR|O_CREAT|O_NOCTTY, 0666) = 3
> flock(3, LOCK_EX|LOCK_NB)               = 0
>
> case C:
> strace myflock /tmp/t
> open("/tmp/t", O_RDONLY)                = 3
> flock(3, LOCK_EX|LOCK_NB)               = 0
>

So that presumably works because the test is not over NFS and not
because test is not over NFS+overlayfs, because of no NFSv4 flock
emulation.


> Below is my test configuration of case A:
> - underlying filesystem:
> ext4
> - /proc/mounts:
> /dev/disk/by-uuid/a2d5005c-.... / ext4
> rw,relatime,errors=remount-ro,data=ordered 0 0
> none /share overlay
> rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base/work,index=on,nfs_export=on
> 0 0
> localhost:/share /mnt/n nfs4
> rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1
> 0 0
> - /etc/exports
> /share *(rw,sync,no_subtree_check,no_root_squash,fsid=41)
>
>
> For dmesg, in case A, there's no any output from dmesg, however in my
> applications running with overlay nfs exported files, there are  some
> lock related messages. Which lock call triggers it, need more
> investigation.
> The message from nfs server side is like:
> [  872.940080] Leaked POSIX lock on dev=0x0:0x42 ino=0xf5a1
> fl_owner=0000000023265f44 fl_flags=0x1 fl_type=0x1 fl_pid=1
> [ 1939.829655] Leaked locks on dev=0x0:0x42 ino=0xf5a1:
> [ 1939.829659] POSIX: fl_owner=0000000023265f44 fl_flags=0x1
> fl_type=0x1 fl_pid=1
>

I'm not sure what those mean. Maybe NFS folks can shed some light.

Thanks,
Amir.

>
> 2018-03-12 20:07 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
>> On Mon, Mar 12, 2018 at 9:38 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>>> Hello Miklos,
>>> I'd like to report a flock(2) problem to overlay nfs-exported files.
>>> The error return from flock(2) is "Bad file descriptor".
>>>
>>> Environment:
>>> OS: Ubuntu 14.04.2 LTS
>>> Kernel: 4.16.0-041600rc4-generic (from
>>> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4/)
>>>
>>> Reproduce step:
>>> (nfs server side)
>>> mount -t overlay
>>> -orw,lowerdir=/mnt/ro,upperdir=/mnt/u,workdir=/mnt/w,nfs_export=on,index=on
>>> none /mnt/m
>>> touch /mnt/m/foo
>>> (nfs client side)
>>> mount server:/mnt/m /mnt/n
>>>
>>> flock /mnt/n/foo
>>> failed to lock file '/mnt/n/foo': Bad file descriptor
>>>
>>
>> Does not reproduce on my end. I am using v4.16-rc5, but I don't think
>> any of the fixes there are relevant to this failure.
>>
>> This is what I have for underlying fs, overlay and nfs mount options
>> (index and nfs_export are on by default in my kernel):
>>
>> /dev/mapper/storage-lower_layer on /base type xfs
>> (rw,relatime,attr2,inode64,noquota)
>> share on /share type overlay
>> (rw,relatime,lowerdir=/base/lower,upperdir=/base/upper/0,workdir=/base/upper/work0)
>> c800:/share on /mnt/t type nfs
>> (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.91.126,mountvers=3,mountport=49494,mountproto=udp,local_lock=none,addr=192.168.91.126)
>>
>> $ touch /mnt/t/foo
>> $ flock -x -n /mnt/t/foo echo locked
>> locked
>>
>> Please share more information about nfs mount options and underlying filesystem
>>
>> Please check if you see any relevant errors/warnings in dmesg.
>>
>> Thanks,
>> Amir.

Re: flock fails in overlay nfs-exported file

[Eddie Horng]

Hi Amir,
I've confirmed that the flock scenario works with NFS3:
(caseA, /mnt/n is mount with -overs=3)
open("/mnt/n/foo", O_RDONLY)            = 3
flock(3, LOCK_EX|LOCK_NB)               = 0

I plan to go for NFS3 rather then change open flag, because the
behavior comes from an Android build toolchain:
https://android.googlesource.com/platform/art/+/master/dex2oat/dex2oat.cc,
line 2327. It could take long time to change it.
Thanks your nice advise about inconsistent ro/rw fd issue, I'll watch
out for it.

However, after I switch to NFS3, I got another problem by readdir(3).
readdir(3) returns dirent->d_type = 0 (DT_UNKNOWN) to a NFS3 mount
overlay-nfs-exported dir. Same overlay nfs share dir mount with NFS4
returns correct d_type. Although readdir(3) states that:
Currently, only some filesystems (among them: Btrfs, ext2,
ext3, and ext4) have full support for returning the file type
in d_type.  All applications must properly handle a return of
DT_UNKNOWN.

I tested ext4 shared dir by NFS3 mount, d_type works. Native access to
overlay dir works too.
Do you have idea where could the problem is?

thanks,
Eddie

2018-03-13 14:24 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
> [CC some NFS/lock folks (see history below top post)]
>
> On Tue, Mar 13, 2018 at 3:39 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>> Hi Amir,
>> Thanks your prompt response. After compare flock(1) and my flock(2)
>> test program, it seems open flag makes the result different. strace
>> result shows open with O_RDONLY flock fails (case A), open with
>> O_RDWR|O_CREAT|O_NOCTTY flock works (case B) and open local ext4 file
>> with O_RDONLY flock works too (case C)
>>
>> case A:
>> strace myflock /mnt/n/foo
>> open("/mnt/n/foo", O_RDONLY)            = 3
>> flock(3, LOCK_EX|LOCK_NB)               = -1 EBADF (Bad file descriptor)
>>
>
> It looks like flock(1) has special code to handle this case for NFSv4
> and fall back to open O_RDRW:
> https://github.com/karelzak/util-linux/blob/master/sys-utils/flock.c#L295
>
> Although I tested with NFSv3 and open flags used by flock(1)
> where O_RDONLY|O_CREAT|O_NOCTTY
>
> Why do you need to get an exclusive lock on a file that is open for read?
> Can you open the file for write and resolve the issue like flock(1) does?
>
> You should know that even if you manage to lock a O_RDONLY fd,
> if this file is then open for write by another process, that process will
> get a file descriptor pointing to a *different* inode.
> This is a long standing issue with overlayfs (inconsistent ro/rw fd),
> which is being worked around by some user applications -
> i.e. touch the file before first access to avoid applications
> getting open file descriptor to lower inode.
>
> Let me know if this answer suffice or if you get this error only
> with NFSv4 over overalyfs.
>
>> case B:
>> strace flock -x -n /mnt/n/foo echo locked
>> open("/mnt/n/foo", O_RDWR|O_CREAT|O_NOCTTY, 0666) = 3
>> flock(3, LOCK_EX|LOCK_NB)               = 0
>>
>> case C:
>> strace myflock /tmp/t
>> open("/tmp/t", O_RDONLY)                = 3
>> flock(3, LOCK_EX|LOCK_NB)               = 0
>>
>
> So that presumably works because the test is not over NFS and not
> because test is not over NFS+overlayfs, because of no NFSv4 flock
> emulation.
>
>
>> Below is my test configuration of case A:
>> - underlying filesystem:
>> ext4
>> - /proc/mounts:
>> /dev/disk/by-uuid/a2d5005c-.... / ext4
>> rw,relatime,errors=remount-ro,data=ordered 0 0
>> none /share overlay
>> rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base/work,index=on,nfs_export=on
>> 0 0
>> localhost:/share /mnt/n nfs4
>> rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1
>> 0 0
>> - /etc/exports
>> /share *(rw,sync,no_subtree_check,no_root_squash,fsid=41)
>>
>>
>> For dmesg, in case A, there's no any output from dmesg, however in my
>> applications running with overlay nfs exported files, there are  some
>> lock related messages. Which lock call triggers it, need more
>> investigation.
>> The message from nfs server side is like:
>> [  872.940080] Leaked POSIX lock on dev=0x0:0x42 ino=0xf5a1
>> fl_owner=0000000023265f44 fl_flags=0x1 fl_type=0x1 fl_pid=1
>> [ 1939.829655] Leaked locks on dev=0x0:0x42 ino=0xf5a1:
>> [ 1939.829659] POSIX: fl_owner=0000000023265f44 fl_flags=0x1
>> fl_type=0x1 fl_pid=1
>>
>
> I'm not sure what those mean. Maybe NFS folks can shed some light.
>
> Thanks,
> Amir.
>
>>
>> 2018-03-12 20:07 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
>>> On Mon, Mar 12, 2018 at 9:38 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>>>> Hello Miklos,
>>>> I'd like to report a flock(2) problem to overlay nfs-exported files.
>>>> The error return from flock(2) is "Bad file descriptor".
>>>>
>>>> Environment:
>>>> OS: Ubuntu 14.04.2 LTS
>>>> Kernel: 4.16.0-041600rc4-generic (from
>>>> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4/)
>>>>
>>>> Reproduce step:
>>>> (nfs server side)
>>>> mount -t overlay
>>>> -orw,lowerdir=/mnt/ro,upperdir=/mnt/u,workdir=/mnt/w,nfs_export=on,index=on
>>>> none /mnt/m
>>>> touch /mnt/m/foo
>>>> (nfs client side)
>>>> mount server:/mnt/m /mnt/n
>>>>
>>>> flock /mnt/n/foo
>>>> failed to lock file '/mnt/n/foo': Bad file descriptor
>>>>
>>>
>>> Does not reproduce on my end. I am using v4.16-rc5, but I don't think
>>> any of the fixes there are relevant to this failure.
>>>
>>> This is what I have for underlying fs, overlay and nfs mount options
>>> (index and nfs_export are on by default in my kernel):
>>>
>>> /dev/mapper/storage-lower_layer on /base type xfs
>>> (rw,relatime,attr2,inode64,noquota)
>>> share on /share type overlay
>>> (rw,relatime,lowerdir=/base/lower,upperdir=/base/upper/0,workdir=/base/upper/work0)
>>> c800:/share on /mnt/t type nfs
>>> (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.91.126,mountvers=3,mountport=49494,mountproto=udp,local_lock=none,addr=192.168.91.126)
>>>
>>> $ touch /mnt/t/foo
>>> $ flock -x -n /mnt/t/foo echo locked
>>> locked
>>>
>>> Please share more information about nfs mount options and underlying filesystem
>>>
>>> Please check if you see any relevant errors/warnings in dmesg.
>>>
>>> Thanks,
>>> Amir.

Re: flock fails in overlay nfs-exported file

[Amir Goldstein]

On Tue, Mar 13, 2018 at 10:40 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> Hi Amir,
> I've confirmed that the flock scenario works with NFS3:
> (caseA, /mnt/n is mount with -overs=3)
> open("/mnt/n/foo", O_RDONLY)            = 3
> flock(3, LOCK_EX|LOCK_NB)               = 0
>
> I plan to go for NFS3 rather then change open flag, because the
> behavior comes from an Android build toolchain:
> https://android.googlesource.com/platform/art/+/master/dex2oat/dex2oat.cc,
> line 2327. It could take long time to change it.
> Thanks your nice advise about inconsistent ro/rw fd issue, I'll watch
> out for it.
>
> However, after I switch to NFS3, I got another problem by readdir(3).
> readdir(3) returns dirent->d_type = 0 (DT_UNKNOWN) to a NFS3 mount
> overlay-nfs-exported dir. Same overlay nfs share dir mount with NFS4
> returns correct d_type. Although readdir(3) states that:
> Currently, only some filesystems (among them: Btrfs, ext2,
> ext3, and ext4) have full support for returning the file type
> in d_type.  All applications must properly handle a return of
> DT_UNKNOWN.
>
> I tested ext4 shared dir by NFS3 mount, d_type works. Native access to
> overlay dir works too.
> Do you have idea where could the problem is?
>

Not really. However, there is no use of DT_UNKNOWN constant
in overlayfs code nor in nfsd code. AFAICT, both overlayfs and nfsd
just pass through the d_type they get from the layer below.
nfs client, on the other hand, does play around with d_type -
see nfs[2|3|4]_decode_dirent(), but I can't say I understand what
these functions do or why the behavior would be affected by
exported overlayfs.
d_type logic seems to be affected by readdirplus, so I would try
mounting with nordirplus/rdirplus and see how they differ.

Thanks,
Amir.

Re: flock fails in overlay nfs-exported file

[Jeff Layton]

On Tue, 2018-03-13 at 08:24 +0200, Amir Goldstein wrote:
> [CC some NFS/lock folks (see history below top post)]
> 
> On Tue, Mar 13, 2018 at 3:39 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> > Hi Amir,
> > Thanks your prompt response. After compare flock(1) and my flock(2)
> > test program, it seems open flag makes the result different. strace
> > result shows open with O_RDONLY flock fails (case A), open with
> > O_RDWR|O_CREAT|O_NOCTTY flock works (case B) and open local ext4 file
> > with O_RDONLY flock works too (case C)
> > 
> > case A:
> > strace myflock /mnt/n/foo
> > open("/mnt/n/foo", O_RDONLY)            = 3
> > flock(3, LOCK_EX|LOCK_NB)               = -1 EBADF (Bad file descriptor)
> > 
> 
> It looks like flock(1) has special code to handle this case for NFSv4
> and fall back to open O_RDRW:
> https://github.com/karelzak/util-linux/blob/master/sys-utils/flock.c#L295
> 
> Although I tested with NFSv3 and open flags used by flock(1)
> where O_RDONLY|O_CREAT|O_NOCTTY
> 
> Why do you need to get an exclusive lock on a file that is open for read?
> Can you open the file for write and resolve the issue like flock(1) does?
> 
> You should know that even if you manage to lock a O_RDONLY fd,
> if this file is then open for write by another process, that process will
> get a file descriptor pointing to a *different* inode.
> This is a long standing issue with overlayfs (inconsistent ro/rw fd),
> which is being worked around by some user applications -
> i.e. touch the file before first access to avoid applications
> getting open file descriptor to lower inode.
> 
> Let me know if this answer suffice or if you get this error only
> with NFSv4 over overalyfs.
>
> > case B:
> > strace flock -x -n /mnt/n/foo echo locked
> > open("/mnt/n/foo", O_RDWR|O_CREAT|O_NOCTTY, 0666) = 3
> > flock(3, LOCK_EX|LOCK_NB)               = 0
> > 
> > case C:
> > strace myflock /tmp/t
> > open("/tmp/t", O_RDONLY)                = 3
> > flock(3, LOCK_EX|LOCK_NB)               = 0
> > 
> 
> So that presumably works because the test is not over NFS and not
> because test is not over NFS+overlayfs, because of no NFSv4 flock
> emulation.
> 

Agreed. The real issue here is that NFSv4 emulates flock locks using
LOCK/LOCKT byte-range locks. The NFSv4 spec does not allow you to set a
write lock on a file open read-only, so that just plain doesn't work on
NFSv4.

> 
> > Below is my test configuration of case A:
> > - underlying filesystem:
> > ext4
> > - /proc/mounts:
> > /dev/disk/by-uuid/a2d5005c-.... / ext4
> > rw,relatime,errors=remount-ro,data=ordered 0 0
> > none /share overlay
> > rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base/work,index=on,nfs_export=on
> > 0 0
> > localhost:/share /mnt/n nfs4
> > rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1
> > 0 0
> > - /etc/exports
> > /share *(rw,sync,no_subtree_check,no_root_squash,fsid=41)
> > 
> > 
> > For dmesg, in case A, there's no any output from dmesg, however in my
> > applications running with overlay nfs exported files, there are  some
> > lock related messages. Which lock call triggers it, need more
> > investigation.
> > The message from nfs server side is like:
> > [  872.940080] Leaked POSIX lock on dev=0x0:0x42 ino=0xf5a1
> > fl_owner=0000000023265f44 fl_flags=0x1 fl_type=0x1 fl_pid=1
> > [ 1939.829655] Leaked locks on dev=0x0:0x42 ino=0xf5a1:
> > [ 1939.829659] POSIX: fl_owner=0000000023265f44 fl_flags=0x1
> > fl_type=0x1 fl_pid=1
> > 
> 
> I'm not sure what those mean. Maybe NFS folks can shed some light.
>

That means that there was a file_lock associated with this struct file
that was left on the POSIX lock list after filp_close. Either it didn't
get released properly or a lock raced onto the list after
locks_remove_posix ran. That should never happen, so this is likely a
bug.

> Thanks,
> Amir.
> 
> > 
> > 2018-03-12 20:07 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
> > > On Mon, Mar 12, 2018 at 9:38 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> > > > Hello Miklos,
> > > > I'd like to report a flock(2) problem to overlay nfs-exported files.
> > > > The error return from flock(2) is "Bad file descriptor".
> > > > 
> > > > Environment:
> > > > OS: Ubuntu 14.04.2 LTS
> > > > Kernel: 4.16.0-041600rc4-generic (from
> > > > http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4/)
> > > > 
> > > > Reproduce step:
> > > > (nfs server side)
> > > > mount -t overlay
> > > > -orw,lowerdir=/mnt/ro,upperdir=/mnt/u,workdir=/mnt/w,nfs_export=on,index=on
> > > > none /mnt/m
> > > > touch /mnt/m/foo
> > > > (nfs client side)
> > > > mount server:/mnt/m /mnt/n
> > > > 
> > > > flock /mnt/n/foo
> > > > failed to lock file '/mnt/n/foo': Bad file descriptor
> > > > 
> > > 
> > > Does not reproduce on my end. I am using v4.16-rc5, but I don't think
> > > any of the fixes there are relevant to this failure.
> > > 
> > > This is what I have for underlying fs, overlay and nfs mount options
> > > (index and nfs_export are on by default in my kernel):
> > > 
> > > /dev/mapper/storage-lower_layer on /base type xfs
> > > (rw,relatime,attr2,inode64,noquota)
> > > share on /share type overlay
> > > (rw,relatime,lowerdir=/base/lower,upperdir=/base/upper/0,workdir=/base/upper/work0)
> > > c800:/share on /mnt/t type nfs
> > > (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.91.126,mountvers=3,mountport=49494,mountproto=udp,local_lock=none,addr=192.168.91.126)
> > > 
> > > $ touch /mnt/t/foo
> > > $ flock -x -n /mnt/t/foo echo locked
> > > locked
> > > 
> > > Please share more information about nfs mount options and underlying filesystem
> > > 
> > > Please check if you see any relevant errors/warnings in dmesg.
> > > 
> > > Thanks,
> > > Amir.

-- 
Jeff Layton <jlayton@kernel.org>

Re: flock fails in overlay nfs-exported file

[Eddie Horng]

Hi Jeff and Amir,
Thanks your clarifying, flock issue has nothing to do with overlay,
the application should change open flag to adapt for NFSv4.
For POSIX dmesg, I will try to find how it is triggered.

thanks,
Eddie

2018-03-13 20:51 GMT+08:00 Jeff Layton <jlayton@kernel.org>:
> On Tue, 2018-03-13 at 08:24 +0200, Amir Goldstein wrote:
>> [CC some NFS/lock folks (see history below top post)]
>>
>> On Tue, Mar 13, 2018 at 3:39 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>> > Hi Amir,
>> > Thanks your prompt response. After compare flock(1) and my flock(2)
>> > test program, it seems open flag makes the result different. strace
>> > result shows open with O_RDONLY flock fails (case A), open with
>> > O_RDWR|O_CREAT|O_NOCTTY flock works (case B) and open local ext4 file
>> > with O_RDONLY flock works too (case C)
>> >
>> > case A:
>> > strace myflock /mnt/n/foo
>> > open("/mnt/n/foo", O_RDONLY)            = 3
>> > flock(3, LOCK_EX|LOCK_NB)               = -1 EBADF (Bad file descriptor)
>> >
>>
>> It looks like flock(1) has special code to handle this case for NFSv4
>> and fall back to open O_RDRW:
>> https://github.com/karelzak/util-linux/blob/master/sys-utils/flock.c#L295
>>
>> Although I tested with NFSv3 and open flags used by flock(1)
>> where O_RDONLY|O_CREAT|O_NOCTTY
>>
>> Why do you need to get an exclusive lock on a file that is open for read?
>> Can you open the file for write and resolve the issue like flock(1) does?
>>
>> You should know that even if you manage to lock a O_RDONLY fd,
>> if this file is then open for write by another process, that process will
>> get a file descriptor pointing to a *different* inode.
>> This is a long standing issue with overlayfs (inconsistent ro/rw fd),
>> which is being worked around by some user applications -
>> i.e. touch the file before first access to avoid applications
>> getting open file descriptor to lower inode.
>>
>> Let me know if this answer suffice or if you get this error only
>> with NFSv4 over overalyfs.
>>
>> > case B:
>> > strace flock -x -n /mnt/n/foo echo locked
>> > open("/mnt/n/foo", O_RDWR|O_CREAT|O_NOCTTY, 0666) = 3
>> > flock(3, LOCK_EX|LOCK_NB)               = 0
>> >
>> > case C:
>> > strace myflock /tmp/t
>> > open("/tmp/t", O_RDONLY)                = 3
>> > flock(3, LOCK_EX|LOCK_NB)               = 0
>> >
>>
>> So that presumably works because the test is not over NFS and not
>> because test is not over NFS+overlayfs, because of no NFSv4 flock
>> emulation.
>>
>
> Agreed. The real issue here is that NFSv4 emulates flock locks using
> LOCK/LOCKT byte-range locks. The NFSv4 spec does not allow you to set a
> write lock on a file open read-only, so that just plain doesn't work on
> NFSv4.
>
>>
>> > Below is my test configuration of case A:
>> > - underlying filesystem:
>> > ext4
>> > - /proc/mounts:
>> > /dev/disk/by-uuid/a2d5005c-.... / ext4
>> > rw,relatime,errors=remount-ro,data=ordered 0 0
>> > none /share overlay
>> > rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base/work,index=on,nfs_export=on
>> > 0 0
>> > localhost:/share /mnt/n nfs4
>> > rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1
>> > 0 0
>> > - /etc/exports
>> > /share *(rw,sync,no_subtree_check,no_root_squash,fsid=41)
>> >
>> >
>> > For dmesg, in case A, there's no any output from dmesg, however in my
>> > applications running with overlay nfs exported files, there are  some
>> > lock related messages. Which lock call triggers it, need more
>> > investigation.
>> > The message from nfs server side is like:
>> > [  872.940080] Leaked POSIX lock on dev=0x0:0x42 ino=0xf5a1
>> > fl_owner=0000000023265f44 fl_flags=0x1 fl_type=0x1 fl_pid=1
>> > [ 1939.829655] Leaked locks on dev=0x0:0x42 ino=0xf5a1:
>> > [ 1939.829659] POSIX: fl_owner=0000000023265f44 fl_flags=0x1
>> > fl_type=0x1 fl_pid=1
>> >
>>
>> I'm not sure what those mean. Maybe NFS folks can shed some light.
>>
>
> That means that there was a file_lock associated with this struct file
> that was left on the POSIX lock list after filp_close. Either it didn't
> get released properly or a lock raced onto the list after
> locks_remove_posix ran. That should never happen, so this is likely a
> bug.
>
>> Thanks,
>> Amir.
>>
>> >
>> > 2018-03-12 20:07 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
>> > > On Mon, Mar 12, 2018 at 9:38 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>> > > > Hello Miklos,
>> > > > I'd like to report a flock(2) problem to overlay nfs-exported files.
>> > > > The error return from flock(2) is "Bad file descriptor".
>> > > >
>> > > > Environment:
>> > > > OS: Ubuntu 14.04.2 LTS
>> > > > Kernel: 4.16.0-041600rc4-generic (from
>> > > > http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4/)
>> > > >
>> > > > Reproduce step:
>> > > > (nfs server side)
>> > > > mount -t overlay
>> > > > -orw,lowerdir=/mnt/ro,upperdir=/mnt/u,workdir=/mnt/w,nfs_export=on,index=on
>> > > > none /mnt/m
>> > > > touch /mnt/m/foo
>> > > > (nfs client side)
>> > > > mount server:/mnt/m /mnt/n
>> > > >
>> > > > flock /mnt/n/foo
>> > > > failed to lock file '/mnt/n/foo': Bad file descriptor
>> > > >
>> > >
>> > > Does not reproduce on my end. I am using v4.16-rc5, but I don't think
>> > > any of the fixes there are relevant to this failure.
>> > >
>> > > This is what I have for underlying fs, overlay and nfs mount options
>> > > (index and nfs_export are on by default in my kernel):
>> > >
>> > > /dev/mapper/storage-lower_layer on /base type xfs
>> > > (rw,relatime,attr2,inode64,noquota)
>> > > share on /share type overlay
>> > > (rw,relatime,lowerdir=/base/lower,upperdir=/base/upper/0,workdir=/base/upper/work0)
>> > > c800:/share on /mnt/t type nfs
>> > > (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.91.126,mountvers=3,mountport=49494,mountproto=udp,local_lock=none,addr=192.168.91.126)
>> > >
>> > > $ touch /mnt/t/foo
>> > > $ flock -x -n /mnt/t/foo echo locked
>> > > locked
>> > >
>> > > Please share more information about nfs mount options and underlying filesystem
>> > >
>> > > Please check if you see any relevant errors/warnings in dmesg.
>> > >
>> > > Thanks,
>> > > Amir.
>
> --
> Jeff Layton <jlayton@kernel.org>

flock fails in overlay nfs-exported file

[Eddie Horng]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

Hello Miklos,
I'd like to report a flock(2) problem to overlay nfs-exported files. The
error return from flock(2) is "Bad file descriptor".

Environment:
OS: Ubuntu 14.04.2 LTS
Kernel: 4.16.0-041600rc4-generic (from
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4/)

Reproduce step:
(nfs server side)
mount -t overlay
-orw,lowerdir=/mnt/ro,upperdir=/mnt/u,workdir=/mnt/w,nfs_export=on,index=on
none /mnt/m
touch /mnt/m/foo
(nfs client side)
mount server:/mnt/m /mnt/n

flock /mnt/n/foo
failed to lock file '/mnt/n/foo': Bad file descriptor

The simple flock(2) wrap program is:
int main(int argc, void **argv) {
char *filename=argv[1];
int fd = open(filename, O_RDONLY);;
int flock_result = flock(fd, LOCK_EX | LOCK_NB);
if (flock_result != 0) {
      printf("failed to lock file '%s': %s\n", filename, strerror(errno));
      goto out;
    }
    printf("flock succeed\n");
out:
    close(fd);
}

Thanks!

[-- Attachment #2: Type: text/html, Size: 1700 bytes --]