[OVERLAYFS] Can I create file in directory on lower layer located on NFS

[OVERLAYFS] Can I create file in directory on lower layer located on NFS

[charles zheng]

Hi Miklos,

We are developing a docker related system that requires to union
directories on NFS and local filesystem.  I successfully created a
overlayfs that has lower layer located on NFS and upper layer located
on the local file system. But when I tried to create new files in the
directory located on NFS, I got error message: "Operation not
supported". Does that mean I can not create new files inside the
directory located on NFS? But base on the documentation of overlayfs,
lower layer does not need to be writable.

Steps to reproduce the problem are (/mnt/nfs is the NFS mount,
/home/zc is the local mount)

1. create directory "/mnt/nfs/lower/lower_dir"

2. create directories "/home/zc/upper/upper_dir", "/home/zc/work" and
"/home/zc/merged"

3. create overlayfs by running command

mount -t overlay overlay -o\
lowerdir=/mnt/nfs/lower,\
upperdir=/home/zc/upper,\
workdir=/home/zc/work \
/home/zc/merged

4. Try to create new file "/merged/lower_dir/test.txt" and get error
message "Operation not supported"

Thanks,
Charles

Re: [OVERLAYFS] Can I create file in directory on lower layer located on NFS

[Amir Goldstein]

On Mon, Oct 16, 2017 at 7:13 PM, charles zheng <czheng2@nd.edu> wrote:
> Hi Miklos,
>
> We are developing a docker related system that requires to union
> directories on NFS and local filesystem.  I successfully created a
> overlayfs that has lower layer located on NFS and upper layer located
> on the local file system. But when I tried to create new files in the
> directory located on NFS, I got error message: "Operation not
> supported". Does that mean I can not create new files inside the
> directory located on NFS? But base on the documentation of overlayfs,
> lower layer does not need to be writable.
>
> Steps to reproduce the problem are (/mnt/nfs is the NFS mount,
> /home/zc is the local mount)
>
> 1. create directory "/mnt/nfs/lower/lower_dir"
>
> 2. create directories "/home/zc/upper/upper_dir", "/home/zc/work" and
> "/home/zc/merged"
>
> 3. create overlayfs by running command
>
> mount -t overlay overlay -o\
> lowerdir=/mnt/nfs/lower,\
> upperdir=/home/zc/upper,\
> workdir=/home/zc/work \
> /home/zc/merged
>
> 4. Try to create new file "/merged/lower_dir/test.txt" and get error
> message "Operation not supported"
>

Hi Charles,

This should definitely work.
Please specify with kernel version/commit you are testing with
Please attach relevant overlayfs logs from dmesg when you get the
error and from overlayfs mount time.
Please attach content of /proc/self/mountinfo after overlay is mounted

Thanks,
Amir.

Re: [OVERLAYFS] Can I create file in directory on lower layer located on NFS

[charles zheng]

Hi Amir,

Thanks for the prompt reply. The Linux distribution is RHEL 7.4 and
the kernel version is
3.10.0-693.2.2.el7.x86_64

Following is the dmesg output

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[ 1341.548099] TECH PREVIEW: Overlay filesystem may not be fully supported.
Please review provided documentation for limitations.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Following is the log message from /proc/self/mountinfo

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
112 62 0:40 / /mnt/nfs rw,relatime shared:63 - nfs4
10.32.8.113:/var/nfs
rw,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.32.8.113,local_lock=none,addr=10.32.8.113
113 62 0:39 / /home/czheng2/test/merged rw,relatime shared:65 -
overlay overlay
rw,lowerdir=/mnt/nfs/lower,upperdir=/home/czheng2/test/upper,workdir=/home/czheng2/test/work
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I also tried on Fedora 26 with kernel version
4.13.5-200.fc26.x86_64

I got same error, but this time, I cannot find overlay related log from dmesg

Following is the log message from /proc/self/mountinfo

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
202 66 0:54 / /mnt/nfs rw,relatime shared:145 - nfs4
192.168.10.123:/var/nfs
rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.10.123,local_lock=none,addr=192.168.10.123
205 84 0:47 / /home/zc/test/merged rw,relatime shared:152 - overlay
overlay rw,seclabel,lowerdir=/mnt/nfs/lower,upperdir=/home/zc/test/upper,workdir=/home/zc/test/work
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Best,
Charles

On Mon, Oct 16, 2017 at 4:41 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> On Mon, Oct 16, 2017 at 7:13 PM, charles zheng <czheng2@nd.edu> wrote:
>> Hi Miklos,
>>
>> We are developing a docker related system that requires to union
>> directories on NFS and local filesystem.  I successfully created a
>> overlayfs that has lower layer located on NFS and upper layer located
>> on the local file system. But when I tried to create new files in the
>> directory located on NFS, I got error message: "Operation not
>> supported". Does that mean I can not create new files inside the
>> directory located on NFS? But base on the documentation of overlayfs,
>> lower layer does not need to be writable.
>>
>> Steps to reproduce the problem are (/mnt/nfs is the NFS mount,
>> /home/zc is the local mount)
>>
>> 1. create directory "/mnt/nfs/lower/lower_dir"
>>
>> 2. create directories "/home/zc/upper/upper_dir", "/home/zc/work" and
>> "/home/zc/merged"
>>
>> 3. create overlayfs by running command
>>
>> mount -t overlay overlay -o\
>> lowerdir=/mnt/nfs/lower,\
>> upperdir=/home/zc/upper,\
>> workdir=/home/zc/work \
>> /home/zc/merged
>>
>> 4. Try to create new file "/merged/lower_dir/test.txt" and get error
>> message "Operation not supported"
>>
>
> Hi Charles,
>
> This should definitely work.
> Please specify with kernel version/commit you are testing with
> Please attach relevant overlayfs logs from dmesg when you get the
> error and from overlayfs mount time.
> Please attach content of /proc/self/mountinfo after overlay is mounted
>
> Thanks,
> Amir.

Re: [OVERLAYFS] Can I create file in directory on lower layer located on NFS

[Amir Goldstein]

On Tue, Oct 17, 2017 at 1:10 AM, charles zheng <czheng2@nd.edu> wrote:
> Hi Amir,
>
> Thanks for the prompt reply. The Linux distribution is RHEL 7.4 and
> the kernel version is
> 3.10.0-693.2.2.el7.x86_64
>
> Following is the dmesg output
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> [ 1341.548099] TECH PREVIEW: Overlay filesystem may not be fully supported.
> Please review provided documentation for limitations.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Following is the log message from /proc/self/mountinfo
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 112 62 0:40 / /mnt/nfs rw,relatime shared:63 - nfs4
> 10.32.8.113:/var/nfs
> rw,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.32.8.113,local_lock=none,addr=10.32.8.113
> 113 62 0:39 / /home/czheng2/test/merged rw,relatime shared:65 -
> overlay overlay
> rw,lowerdir=/mnt/nfs/lower,upperdir=/home/czheng2/test/upper,workdir=/home/czheng2/test/work
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> I also tried on Fedora 26 with kernel version
> 4.13.5-200.fc26.x86_64
>
> I got same error, but this time, I cannot find overlay related log from dmesg
>
> Following is the log message from /proc/self/mountinfo
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 202 66 0:54 / /mnt/nfs rw,relatime shared:145 - nfs4
> 192.168.10.123:/var/nfs
> rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.10.123,local_lock=none,addr=192.168.10.123
> 205 84 0:47 / /home/zc/test/merged rw,relatime shared:152 - overlay
> overlay rw,seclabel,lowerdir=/mnt/nfs/lower,upperdir=/home/zc/test/upper,workdir=/home/zc/test/work
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>

Since the error happens on such as old kernel as well as a new one, I suspect
copy up fails on copying extended attributes of lower_dir.
I suspect there are some NFS related xattr on lower dir?
and perhaps those cannot be set on your upper fs?
You did not reveal in mountinfo snippet what you local upper fs is
and what mount options it uses.

Please try to check if lower dir contains any xattr and
if your upper fs supports xattr.

Amir.


>
> On Mon, Oct 16, 2017 at 4:41 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>> On Mon, Oct 16, 2017 at 7:13 PM, charles zheng <czheng2@nd.edu> wrote:
>>> Hi Miklos,
>>>
>>> We are developing a docker related system that requires to union
>>> directories on NFS and local filesystem.  I successfully created a
>>> overlayfs that has lower layer located on NFS and upper layer located
>>> on the local file system. But when I tried to create new files in the
>>> directory located on NFS, I got error message: "Operation not
>>> supported". Does that mean I can not create new files inside the
>>> directory located on NFS? But base on the documentation of overlayfs,
>>> lower layer does not need to be writable.
>>>
>>> Steps to reproduce the problem are (/mnt/nfs is the NFS mount,
>>> /home/zc is the local mount)
>>>
>>> 1. create directory "/mnt/nfs/lower/lower_dir"
>>>
>>> 2. create directories "/home/zc/upper/upper_dir", "/home/zc/work" and
>>> "/home/zc/merged"
>>>
>>> 3. create overlayfs by running command
>>>
>>> mount -t overlay overlay -o\
>>> lowerdir=/mnt/nfs/lower,\
>>> upperdir=/home/zc/upper,\
>>> workdir=/home/zc/work \
>>> /home/zc/merged
>>>
>>> 4. Try to create new file "/merged/lower_dir/test.txt" and get error
>>> message "Operation not supported"
>>>
>>
>> Hi Charles,
>>
>> This should definitely work.
>> Please specify with kernel version/commit you are testing with
>> Please attach relevant overlayfs logs from dmesg when you get the
>> error and from overlayfs mount time.
>> Please attach content of /proc/self/mountinfo after overlay is mounted
>>
>> Thanks,
>> Amir.

Re: [OVERLAYFS] Can I create file in directory on lower layer located on NFS

[Miklos Szeredi]

On Tue, Oct 17, 2017 at 7:37 AM, Amir Goldstein <amir73il@gmail.com> wrote:
> Since the error happens on such as old kernel as well as a new one, I suspect
> copy up fails on copying extended attributes of lower_dir.
> I suspect there are some NFS related xattr on lower dir?
> and perhaps those cannot be set on your upper fs?
> You did not reveal in mountinfo snippet what you local upper fs is
> and what mount options it uses.
>
> Please try to check if lower dir contains any xattr and
> if your upper fs supports xattr.

Yes, there's an NFSv4 related splat because overlayfs can't copy up
"system.nfs4_acl" xattr.  The weird but apparently unfixable fact is
that this attribute is there even if the nfs4-acl is equivalent to the
DAC bits (i.e. the common case).

If you don't want to use NFS acls, then just turn them off with
"noacl" mount option on the NFS mount and things should work.

Thanks,
Miklos

Re: [OVERLAYFS] Can I create file in directory on lower layer located on NFS

[charles zheng]

[-- Attachment #1: Type: text/plain, Size: 1230 bytes --]

Hello Amir and Miklos,

I tried NFSv4 with noacl option, and I still got "operation not support". I
also tried NFSv3, and it works as expect. Hope this information is useful
to you.

Best,
Charles

On Tue, Oct 17, 2017 at 3:11 AM, Miklos Szeredi <miklos@szeredi.hu> wrote:

> On Tue, Oct 17, 2017 at 7:37 AM, Amir Goldstein <amir73il@gmail.com>
> wrote:
> > Since the error happens on such as old kernel as well as a new one, I
> suspect
> > copy up fails on copying extended attributes of lower_dir.
> > I suspect there are some NFS related xattr on lower dir?
> > and perhaps those cannot be set on your upper fs?
> > You did not reveal in mountinfo snippet what you local upper fs is
> > and what mount options it uses.
> >
> > Please try to check if lower dir contains any xattr and
> > if your upper fs supports xattr.
>
> Yes, there's an NFSv4 related splat because overlayfs can't copy up
> "system.nfs4_acl" xattr.  The weird but apparently unfixable fact is
> that this attribute is there even if the nfs4-acl is equivalent to the
> DAC bits (i.e. the common case).
>
> If you don't want to use NFS acls, then just turn them off with
> "noacl" mount option on the NFS mount and things should work.
>
> Thanks,
> Miklos
>

[-- Attachment #2: Type: text/html, Size: 2069 bytes --]

[OVERLAYFS] Can I create file in directory on lower layer located on NFS

[charles zheng]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

Hi Miklos,

We are developing a docker related system that requires to union
directories on NFS and local filesystem.  I successfully created a
overlayfs that has lower layer located on NFS and upper layer located on
the local file system. But when I tried to create new files in the
directory located on NFS, I got error message: "Operation not supported".
Does that mean I can not create new files inside the directory located on
NFS? But base on the documentation of overlayfs, lower layer does not need
to be writable.

Steps to reproduce the problem are (/mnt/nfs is the NFS mount, /home/zc is
the local mount)

1. create directory "/mnt/nfs/lower/lower_dir"

2. create directories "/home/zc/upper/upper_dir", "/home/zc/work" and
"/home/zc/merged"

3. create overlayfs by running command

mount -t overlay overlay -o\
lowerdir=/mnt/nfs/lower,\
upperdir=/home/zc/upper,\
workdir=/home/zc/work \
/home/zc/merged

4. Try to create new file "/merged/lower_dir/test.txt" and get error
message "Operation not supported"

Thanks,
Charles

[-- Attachment #2: Type: text/html, Size: 2396 bytes --]