readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Eddie Horng]

Hi Amir,
Since the flock issue is clarified, I would like to start this new
thread to discuss if we can find the cause of d_type=DT_UNKNOWN. First
of all I tried rdirplus and nordirplus mount option but got no
difference. Then I roughly trace call flow of nfs3_decode_dirent( ) as
below:

decode_fattr3()
    fattr->mode = (be32_to_cpup(p++) & ~S_IFMT) | fmode;
    fattr->valid |= NFS_ATTR_FATTR_V3;

decode_post_op_attr()
    p = xdr_inline_decode(xdr, 4);
    if (*p != xdr_zero)
       return decode_fattr3(xdr, fattr);

nfs3_decode_dirent()
    entry->d_type = DT_UNKNOWN;
    if (entry->fattr->valid & NFS_ATTR_FATTR_V3)
        entry->d_type = nfs_umode_to_dtype(entry->fattr->mode);

In overlay exported case, decode_post_op_attr hits *p==xdr_zero, so
decode_fattr3 is not called, then d_type reamins DT_UNKNOWN in
nfs3_decode_dirent. It seems nfs4_decode_dirent has very different way
to decode file attr and maybe the xdr layout is quite different as
well.I have not idea about how the xdr to be filled from overlay and
nfsd's output. Could some of you master give a hint?

thanks,
Eddie

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Amir Goldstein]

On Wed, Mar 14, 2018 at 10:42 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> Hi Amir,
> Since the flock issue is clarified, I would like to start this new
> thread to discuss if we can find the cause of d_type=DT_UNKNOWN. First

Can you provide a simple reproducer? Is DT_UNKNOWN always the case
in readdir or only in lower/upper/merged directories? for all entries?

> of all I tried rdirplus and nordirplus mount option but got no
> difference. Then I roughly trace call flow of nfs3_decode_dirent( ) as
> below:
>
> decode_fattr3()
>     fattr->mode = (be32_to_cpup(p++) & ~S_IFMT) | fmode;
>     fattr->valid |= NFS_ATTR_FATTR_V3;
>
> decode_post_op_attr()
>     p = xdr_inline_decode(xdr, 4);
>     if (*p != xdr_zero)
>        return decode_fattr3(xdr, fattr);
>
> nfs3_decode_dirent()
>     entry->d_type = DT_UNKNOWN;
>     if (entry->fattr->valid & NFS_ATTR_FATTR_V3)
>         entry->d_type = nfs_umode_to_dtype(entry->fattr->mode);
>
> In overlay exported case, decode_post_op_attr hits *p==xdr_zero, so
> decode_fattr3 is not called, then d_type reamins DT_UNKNOWN in
> nfs3_decode_dirent. It seems nfs4_decode_dirent has very different way
> to decode file attr and maybe the xdr layout is quite different as
> well.I have not idea about how the xdr to be filled from overlay and
> nfsd's output. Could some of you master give a hint?
>

Thanks for the detailed analysis.
Seems like server side sets the post op attr in encode_post_op_attr()
probably from nfs3svc_encode_readdirres(). There is a comment
above encode_post_op_attr() that I don't fully understand and not sure
why that would happen in overlayfs.

If you can perform similar tracing on server side and point us at where
the overlay and non-overlay flows diverge this would be helpful.

I also recommend testing with v4.16-rc5, which has some overlayfs NFS
export fixes, although, I am not sure how any of those fixes map to the
problem you reported and I didn't have time to test your report myself yet.

Thanks,
Amir.

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Jeff Layton]

On Wed, 2018-03-14 at 16:42 +0800, Eddie Horng wrote:
> Hi Amir,
> Since the flock issue is clarified, I would like to start this new
> thread to discuss if we can find the cause of d_type=DT_UNKNOWN. First

This sounds like NOTABUG to me. As readdir(3) states:

Currently, only some filesystems (among them: Btrfs, ext2, ext3,
and ext4) have full support  for  returning  the  file  type  in
d_type.   All  applications  must  properly  handle  a return of
DT_UNKNOWN.

Applications that rely solely on d_type are effectively broken. You
always need to be able to follow up with a stat or equivalent.

> of all I tried rdirplus and nordirplus mount option but got no
> difference. Then I roughly trace call flow of nfs3_decode_dirent( ) as
> below:
> 
> decode_fattr3()
>     fattr->mode = (be32_to_cpup(p++) & ~S_IFMT) | fmode;
>     fattr->valid |= NFS_ATTR_FATTR_V3;
> 
> decode_post_op_attr()
>     p = xdr_inline_decode(xdr, 4);
>     if (*p != xdr_zero)
>        return decode_fattr3(xdr, fattr);
> 
> nfs3_decode_dirent()
>     entry->d_type = DT_UNKNOWN;
>     if (entry->fattr->valid & NFS_ATTR_FATTR_V3)
>         entry->d_type = nfs_umode_to_dtype(entry->fattr->mode);
> 
> In overlay exported case, decode_post_op_attr hits *p==xdr_zero, so
> decode_fattr3 is not called, then d_type reamins DT_UNKNOWN in
> nfs3_decode_dirent. It seems nfs4_decode_dirent has very different way
> to decode file attr and maybe the xdr layout is quite different as
> well.I have not idea about how the xdr to be filled from overlay and
> nfsd's output. Could some of you master give a hint?

Very different.

The v3 client will sometimes issue a READDIR (just names and fileids of
dirents) and sometimes a READDIRPLUS (same as READDIR + attributes)
depending on the expected size of the result and what sort of access
pattern the application has. If you get back DT_UNKNOWN, it's quite
possible that it chose to use a READDIR and therefore doesn't have
enough info to fill out the d_type.

Cheers,
-- 
Jeff Layton <jlayton@kernel.org>

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Trond Myklebust]

On Wed, 2018-03-14 at 07:02 -0400, Jeff Layton wrote:
> On Wed, 2018-03-14 at 16:42 +0800, Eddie Horng wrote:
> > Hi Amir,
> > Since the flock issue is clarified, I would like to start this new
> > thread to discuss if we can find the cause of d_type=DT_UNKNOWN.
> > First
> 
> This sounds like NOTABUG to me. As readdir(3) states:
> 
> Currently, only some filesystems (among them: Btrfs, ext2, ext3,
> and ext4) have full support  for  returning  the  file  type  in
> d_type.   All  applications  must  properly  handle  a return of
> DT_UNKNOWN.
> 
> Applications that rely solely on d_type are effectively broken. You
> always need to be able to follow up with a stat or equivalent.
> 

Yes, but one of the main such applications is the "find" utility, which
uses it to avoid calling stat() in order to discover the directories.
For that reason, NFS does try to set the d_type flag when it is using
readdirplus, and the server returns attributes for the entry in
question. Otherwise, it is forced to default to DT_UNKNOWN.

Note that in the cases where the readdir entry has a matching dentry,
we probably could try to do better by doing a d_lookup() and then
filling the d_type. Is that worth doing?

> > of all I tried rdirplus and nordirplus mount option but got no
> > difference. Then I roughly trace call flow of nfs3_decode_dirent( )
> > as
> > below:
> > 
> > decode_fattr3()
> >     fattr->mode = (be32_to_cpup(p++) & ~S_IFMT) | fmode;
> >     fattr->valid |= NFS_ATTR_FATTR_V3;
> > 
> > decode_post_op_attr()
> >     p = xdr_inline_decode(xdr, 4);
> >     if (*p != xdr_zero)
> >        return decode_fattr3(xdr, fattr);
> > 
> > nfs3_decode_dirent()
> >     entry->d_type = DT_UNKNOWN;
> >     if (entry->fattr->valid & NFS_ATTR_FATTR_V3)
> >         entry->d_type = nfs_umode_to_dtype(entry->fattr->mode);
> > 
> > In overlay exported case, decode_post_op_attr hits *p==xdr_zero, so
> > decode_fattr3 is not called, then d_type reamins DT_UNKNOWN in
> > nfs3_decode_dirent. It seems nfs4_decode_dirent has very different
> > way
> > to decode file attr and maybe the xdr layout is quite different as
> > well.I have not idea about how the xdr to be filled from overlay
> > and
> > nfsd's output. Could some of you master give a hint?
> 
> Very different.
> 
> The v3 client will sometimes issue a READDIR (just names and fileids
> of
> dirents) and sometimes a READDIRPLUS (same as READDIR + attributes)
> depending on the expected size of the result and what sort of access
> pattern the application has. If you get back DT_UNKNOWN, it's quite
> possible that it chose to use a READDIR and therefore doesn't have
> enough info to fill out the d_type.
> 
> Cheers,
-- 
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust@primarydata.com

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Jeff Layton]

On Wed, 2018-03-14 at 14:16 +0000, Trond Myklebust wrote:
> On Wed, 2018-03-14 at 07:02 -0400, Jeff Layton wrote:
> > On Wed, 2018-03-14 at 16:42 +0800, Eddie Horng wrote:
> > > Hi Amir,
> > > Since the flock issue is clarified, I would like to start this new
> > > thread to discuss if we can find the cause of d_type=DT_UNKNOWN.
> > > First
> > 
> > This sounds like NOTABUG to me. As readdir(3) states:
> > 
> > Currently, only some filesystems (among them: Btrfs, ext2, ext3,
> > and ext4) have full support  for  returning  the  file  type  in
> > d_type.   All  applications  must  properly  handle  a return of
> > DT_UNKNOWN.
> > 
> > Applications that rely solely on d_type are effectively broken. You
> > always need to be able to follow up with a stat or equivalent.
> > 
> 
> Yes, but one of the main such applications is the "find" utility, which
> uses it to avoid calling stat() in order to discover the directories.
> For that reason, NFS does try to set the d_type flag when it is using
> readdirplus, and the server returns attributes for the entry in
> question. Otherwise, it is forced to default to DT_UNKNOWN.
> 

Yes, didn't mean to imply that we shouldn't try to fill these out where
we can, just that there are situations where we might not be able to do
so without taking a performance hit.

> Note that in the cases where the readdir entry has a matching dentry,
> we probably could try to do better by doing a d_lookup() and then
> filling the d_type. Is that worth doing?
> 

I like that idea. Filling out what info we can from the local cache is
almost always worthwhile.

An inode's d_type can never change, so you can just vet the fileid or fh
in the entry3 vs. the inode that comes back from d_lookup. If they match
then you can reliably fill that out.

> > > of all I tried rdirplus and nordirplus mount option but got no
> > > difference. Then I roughly trace call flow of nfs3_decode_dirent( )
> > > as
> > > below:
> > > 
> > > decode_fattr3()
> > >     fattr->mode = (be32_to_cpup(p++) & ~S_IFMT) | fmode;
> > >     fattr->valid |= NFS_ATTR_FATTR_V3;
> > > 
> > > decode_post_op_attr()
> > >     p = xdr_inline_decode(xdr, 4);
> > >     if (*p != xdr_zero)
> > >        return decode_fattr3(xdr, fattr);
> > > 
> > > nfs3_decode_dirent()
> > >     entry->d_type = DT_UNKNOWN;
> > >     if (entry->fattr->valid & NFS_ATTR_FATTR_V3)
> > >         entry->d_type = nfs_umode_to_dtype(entry->fattr->mode);
> > > 
> > > In overlay exported case, decode_post_op_attr hits *p==xdr_zero, so
> > > decode_fattr3 is not called, then d_type reamins DT_UNKNOWN in
> > > nfs3_decode_dirent. It seems nfs4_decode_dirent has very different
> > > way
> > > to decode file attr and maybe the xdr layout is quite different as
> > > well.I have not idea about how the xdr to be filled from overlay
> > > and
> > > nfsd's output. Could some of you master give a hint?
> > 
> > Very different.
> > 
> > The v3 client will sometimes issue a READDIR (just names and fileids
> > of
> > dirents) and sometimes a READDIRPLUS (same as READDIR + attributes)
> > depending on the expected size of the result and what sort of access
> > pattern the application has. If you get back DT_UNKNOWN, it's quite
> > possible that it chose to use a READDIR and therefore doesn't have
> > enough info to fill out the d_type.
> > 
> > Cheers,
> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer, PrimaryData
> trond.myklebust@primarydata.com

-- 
Jeff Layton <jlayton@kernel.org>

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Amir Goldstein]

On Wed, Mar 14, 2018 at 4:30 PM, Jeff Layton <jlayton@kernel.org> wrote:
> On Wed, 2018-03-14 at 14:16 +0000, Trond Myklebust wrote:
>> On Wed, 2018-03-14 at 07:02 -0400, Jeff Layton wrote:
>> > On Wed, 2018-03-14 at 16:42 +0800, Eddie Horng wrote:
>> > > Hi Amir,
>> > > Since the flock issue is clarified, I would like to start this new
>> > > thread to discuss if we can find the cause of d_type=DT_UNKNOWN.
>> > > First
>> >
>> > This sounds like NOTABUG to me. As readdir(3) states:
>> >
>> > Currently, only some filesystems (among them: Btrfs, ext2, ext3,
>> > and ext4) have full support  for  returning  the  file  type  in
>> > d_type.   All  applications  must  properly  handle  a return of
>> > DT_UNKNOWN.
>> >
>> > Applications that rely solely on d_type are effectively broken. You
>> > always need to be able to follow up with a stat or equivalent.
>> >
>>
>> Yes, but one of the main such applications is the "find" utility, which
>> uses it to avoid calling stat() in order to discover the directories.
>> For that reason, NFS does try to set the d_type flag when it is using
>> readdirplus, and the server returns attributes for the entry in
>> question. Otherwise, it is forced to default to DT_UNKNOWN.
>>
>
> Yes, didn't mean to imply that we shouldn't try to fill these out where
> we can, just that there are situations where we might not be able to do
> so without taking a performance hit.
>
>> Note that in the cases where the readdir entry has a matching dentry,
>> we probably could try to do better by doing a d_lookup() and then
>> filling the d_type. Is that worth doing?
>>
>
> I like that idea. Filling out what info we can from the local cache is
> almost always worthwhile.
>
> An inode's d_type can never change, so you can just vet the fileid or fh
> in the entry3 vs. the inode that comes back from d_lookup. If they match
> then you can reliably fill that out.
>

Ironically, this is where NFS over overlayfs may fail, because in overlayfs
d_ino is not always consistent with st_ino. Since v4.15, d_ino is consistent
with st_ino for the case of all layers on the same filesystem. I already posted
a POC for fixing d_ino/st_ino for non-samefs, but it never got merged.

What puzzles me w.r.t. this "nonbug" report is that I don't understand why
NFS over overlayfs would behave differently vs. NFS over local fs.
I am hoping it does not point to a different problem, so would love to
get a more detailed analysis of what's going on between nfsd and overlayfs.

Thanks,
Amir.

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Trond Myklebust]

On Wed, 2018-03-14 at 17:03 +0200, Amir Goldstein wrote:
> On Wed, Mar 14, 2018 at 4:30 PM, Jeff Layton <jlayton@kernel.org>
> wrote:
> > On Wed, 2018-03-14 at 14:16 +0000, Trond Myklebust wrote:
> > > On Wed, 2018-03-14 at 07:02 -0400, Jeff Layton wrote:
> > > > On Wed, 2018-03-14 at 16:42 +0800, Eddie Horng wrote:
> > > > > Hi Amir,
> > > > > Since the flock issue is clarified, I would like to start
> > > > > this new
> > > > > thread to discuss if we can find the cause of
> > > > > d_type=DT_UNKNOWN.
> > > > > First
> > > > 
> > > > This sounds like NOTABUG to me. As readdir(3) states:
> > > > 
> > > > Currently, only some filesystems (among them: Btrfs, ext2,
> > > > ext3,
> > > > and ext4) have full
> > > > support  for  returning  the  file  type  in
> > > > d_type.   All  applications  must  properly  handle  a return
> > > > of
> > > > DT_UNKNOWN.
> > > > 
> > > > Applications that rely solely on d_type are effectively broken.
> > > > You
> > > > always need to be able to follow up with a stat or equivalent.
> > > > 
> > > 
> > > Yes, but one of the main such applications is the "find" utility,
> > > which
> > > uses it to avoid calling stat() in order to discover the
> > > directories.
> > > For that reason, NFS does try to set the d_type flag when it is
> > > using
> > > readdirplus, and the server returns attributes for the entry in
> > > question. Otherwise, it is forced to default to DT_UNKNOWN.
> > > 
> > 
> > Yes, didn't mean to imply that we shouldn't try to fill these out
> > where
> > we can, just that there are situations where we might not be able
> > to do
> > so without taking a performance hit.
> > 
> > > Note that in the cases where the readdir entry has a matching
> > > dentry,
> > > we probably could try to do better by doing a d_lookup() and then
> > > filling the d_type. Is that worth doing?
> > > 
> > 
> > I like that idea. Filling out what info we can from the local cache
> > is
> > almost always worthwhile.
> > 
> > An inode's d_type can never change, so you can just vet the fileid
> > or fh
> > in the entry3 vs. the inode that comes back from d_lookup. If they
> > match
> > then you can reliably fill that out.
> > 
> 
> Ironically, this is where NFS over overlayfs may fail, because in
> overlayfs
> d_ino is not always consistent with st_ino. Since v4.15, d_ino is
> consistent
> with st_ino for the case of all layers on the same filesystem. I
> already posted
> a POC for fixing d_ino/st_ino for non-samefs, but it never got
> merged.
>
> What puzzles me w.r.t. this "nonbug" report is that I don't
> understand why
> NFS over overlayfs would behave differently vs. NFS over local fs.
> I am hoping it does not point to a different problem, so would love
> to
> get a more detailed analysis of what's going on between nfsd and
> overlayfs.

The behaviour being described is true of the regular NFS client. It has
nothing to do with overlayfs.

-- 
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust@primarydata.com

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Amir Goldstein]

On Wed, Mar 14, 2018 at 5:06 PM, Trond Myklebust
<trondmy@primarydata.com> wrote:
> On Wed, 2018-03-14 at 17:03 +0200, Amir Goldstein wrote:
>> On Wed, Mar 14, 2018 at 4:30 PM, Jeff Layton <jlayton@kernel.org>
>> wrote:
>> > On Wed, 2018-03-14 at 14:16 +0000, Trond Myklebust wrote:
>> > > On Wed, 2018-03-14 at 07:02 -0400, Jeff Layton wrote:
>> > > > On Wed, 2018-03-14 at 16:42 +0800, Eddie Horng wrote:
>> > > > > Hi Amir,
>> > > > > Since the flock issue is clarified, I would like to start
>> > > > > this new
>> > > > > thread to discuss if we can find the cause of
>> > > > > d_type=DT_UNKNOWN.
>> > > > > First
>> > > >
>> > > > This sounds like NOTABUG to me. As readdir(3) states:
>> > > >
>> > > > Currently, only some filesystems (among them: Btrfs, ext2,
>> > > > ext3,
>> > > > and ext4) have full
>> > > > support  for  returning  the  file  type  in
>> > > > d_type.   All  applications  must  properly  handle  a return
>> > > > of
>> > > > DT_UNKNOWN.
>> > > >
>> > > > Applications that rely solely on d_type are effectively broken.
>> > > > You
>> > > > always need to be able to follow up with a stat or equivalent.
>> > > >
>> > >
>> > > Yes, but one of the main such applications is the "find" utility,
>> > > which
>> > > uses it to avoid calling stat() in order to discover the
>> > > directories.
>> > > For that reason, NFS does try to set the d_type flag when it is
>> > > using
>> > > readdirplus, and the server returns attributes for the entry in
>> > > question. Otherwise, it is forced to default to DT_UNKNOWN.
>> > >
>> >
>> > Yes, didn't mean to imply that we shouldn't try to fill these out
>> > where
>> > we can, just that there are situations where we might not be able
>> > to do
>> > so without taking a performance hit.
>> >
>> > > Note that in the cases where the readdir entry has a matching
>> > > dentry,
>> > > we probably could try to do better by doing a d_lookup() and then
>> > > filling the d_type. Is that worth doing?
>> > >
>> >
>> > I like that idea. Filling out what info we can from the local cache
>> > is
>> > almost always worthwhile.
>> >
>> > An inode's d_type can never change, so you can just vet the fileid
>> > or fh
>> > in the entry3 vs. the inode that comes back from d_lookup. If they
>> > match
>> > then you can reliably fill that out.
>> >
>>
>> Ironically, this is where NFS over overlayfs may fail, because in
>> overlayfs
>> d_ino is not always consistent with st_ino. Since v4.15, d_ino is
>> consistent
>> with st_ino for the case of all layers on the same filesystem. I
>> already posted
>> a POC for fixing d_ino/st_ino for non-samefs, but it never got
>> merged.
>>
>> What puzzles me w.r.t. this "nonbug" report is that I don't
>> understand why
>> NFS over overlayfs would behave differently vs. NFS over local fs.
>> I am hoping it does not point to a different problem, so would love
>> to
>> get a more detailed analysis of what's going on between nfsd and
>> overlayfs.
>
> The behaviour being described is true of the regular NFS client. It has
> nothing to do with overlayfs.
>

Here:
https://marc.info/?l=linux-unionfs&m=152093050204357&w=2

Eddie claims that the observed DT_UNKNOWN issue is reproduced
with NFSv3 over overlayfs and NOT with NFSv3 over ext4.

That's the only fact I find puzzling.

But I did not see a confirmation that this is a reproducible problem
with exact same directory and I did not try to reproduce it myself.

Thanks,
Amir.

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Eddie Horng]

This is an always reproduced case for NFSv3 over overlayfs, and each
dir under nfs mount point behaviors the same. The same overlay
exported mount with NFSv4 returns correct d_type. And for ext4
exported mount with NFSv3 returns correct as well.

My simple test program is like:
void main(int argc, void **argv) {
    DIR *dir = opendir(argv[1]);

    if (dir == NULL) {
        fprintf(stderr,
                "ERROR: Could not open dir: %s\n", argv[1]);
        return;
    }
    struct dirent *ent;
    while ((ent = readdir(dir)) != NULL) {
       printf("readdir: %s ino(%x) type(%x)\n", ent->d_name,
ent->d_ino, ent->d_type);
    }
    closedir(dir);

}

output is:
readdir: . ino(e0003) type(0)
readdir: .. ino(e0003) type(0)
readdir: foo ino(e0007) type(0)

2018-03-14 23:14 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
> On Wed, Mar 14, 2018 at 5:06 PM, Trond Myklebust
> <trondmy@primarydata.com> wrote:
>> On Wed, 2018-03-14 at 17:03 +0200, Amir Goldstein wrote:
>>> On Wed, Mar 14, 2018 at 4:30 PM, Jeff Layton <jlayton@kernel.org>
>>> wrote:
>>> > On Wed, 2018-03-14 at 14:16 +0000, Trond Myklebust wrote:
>>> > > On Wed, 2018-03-14 at 07:02 -0400, Jeff Layton wrote:
>>> > > > On Wed, 2018-03-14 at 16:42 +0800, Eddie Horng wrote:
>>> > > > > Hi Amir,
>>> > > > > Since the flock issue is clarified, I would like to start
>>> > > > > this new
>>> > > > > thread to discuss if we can find the cause of
>>> > > > > d_type=DT_UNKNOWN.
>>> > > > > First
>>> > > >
>>> > > > This sounds like NOTABUG to me. As readdir(3) states:
>>> > > >
>>> > > > Currently, only some filesystems (among them: Btrfs, ext2,
>>> > > > ext3,
>>> > > > and ext4) have full
>>> > > > support  for  returning  the  file  type  in
>>> > > > d_type.   All  applications  must  properly  handle  a return
>>> > > > of
>>> > > > DT_UNKNOWN.
>>> > > >
>>> > > > Applications that rely solely on d_type are effectively broken.
>>> > > > You
>>> > > > always need to be able to follow up with a stat or equivalent.
>>> > > >
>>> > >
>>> > > Yes, but one of the main such applications is the "find" utility,
>>> > > which
>>> > > uses it to avoid calling stat() in order to discover the
>>> > > directories.
>>> > > For that reason, NFS does try to set the d_type flag when it is
>>> > > using
>>> > > readdirplus, and the server returns attributes for the entry in
>>> > > question. Otherwise, it is forced to default to DT_UNKNOWN.
>>> > >
>>> >
>>> > Yes, didn't mean to imply that we shouldn't try to fill these out
>>> > where
>>> > we can, just that there are situations where we might not be able
>>> > to do
>>> > so without taking a performance hit.
>>> >
>>> > > Note that in the cases where the readdir entry has a matching
>>> > > dentry,
>>> > > we probably could try to do better by doing a d_lookup() and then
>>> > > filling the d_type. Is that worth doing?
>>> > >
>>> >
>>> > I like that idea. Filling out what info we can from the local cache
>>> > is
>>> > almost always worthwhile.
>>> >
>>> > An inode's d_type can never change, so you can just vet the fileid
>>> > or fh
>>> > in the entry3 vs. the inode that comes back from d_lookup. If they
>>> > match
>>> > then you can reliably fill that out.
>>> >
>>>
>>> Ironically, this is where NFS over overlayfs may fail, because in
>>> overlayfs
>>> d_ino is not always consistent with st_ino. Since v4.15, d_ino is
>>> consistent
>>> with st_ino for the case of all layers on the same filesystem. I
>>> already posted
>>> a POC for fixing d_ino/st_ino for non-samefs, but it never got
>>> merged.
>>>
>>> What puzzles me w.r.t. this "nonbug" report is that I don't
>>> understand why
>>> NFS over overlayfs would behave differently vs. NFS over local fs.
>>> I am hoping it does not point to a different problem, so would love
>>> to
>>> get a more detailed analysis of what's going on between nfsd and
>>> overlayfs.
>>
>> The behaviour being described is true of the regular NFS client. It has
>> nothing to do with overlayfs.
>>
>
> Here:
> https://marc.info/?l=linux-unionfs&m=152093050204357&w=2
>
> Eddie claims that the observed DT_UNKNOWN issue is reproduced
> with NFSv3 over overlayfs and NOT with NFSv3 over ext4.
>
> That's the only fact I find puzzling.
>
> But I did not see a confirmation that this is a reproducible problem
> with exact same directory and I did not try to reproduce it myself.
>
> Thanks,
> Amir.

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Eddie Horng]

[-- Attachment #1: Type: text/plain, Size: 4843 bytes --]

Eddie Horng <eddiehorng.tw@gmail.com> 於 2018年3月14日週三 23:40 寫道：

> This is an always reproduced case for NFSv3 over overlayfs, and each
> dir under nfs mount point behaviors the same. The same overlay
> exported mount with NFSv4 returns correct d_type. And for ext4
> exported mount with NFSv3 returns correct as well.
>
> My simple test program is like:
> void main(int argc, void **argv) {
>     DIR *dir = opendir(argv[1]);
>
>     if (dir == NULL) {
>         fprintf(stderr,
>                 "ERROR: Could not open dir: %s\n", argv[1]);
>         return;
>     }
>     struct dirent *ent;
>     while ((ent = readdir(dir)) != NULL) {
>        printf("readdir: %s ino(%x) type(%x)\n", ent->d_name,
> ent->d_ino, ent->d_type);
>     }
>     closedir(dir);
>
> }
>
> output is:
> readdir: . ino(e0003) type(0)
> readdir: .. ino(e0003) type(0)
> readdir: foo ino(e0007) type(0)
>
> 2018-03-14 23:14 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
> > On Wed, Mar 14, 2018 at 5:06 PM, Trond Myklebust
> > <trondmy@primarydata.com> wrote:
> >> On Wed, 2018-03-14 at 17:03 +0200, Amir Goldstein wrote:
> >>> On Wed, Mar 14, 2018 at 4:30 PM, Jeff Layton <jlayton@kernel.org>
> >>> wrote:
> >>> > On Wed, 2018-03-14 at 14:16 +0000, Trond Myklebust wrote:
> >>> > > On Wed, 2018-03-14 at 07:02 -0400, Jeff Layton wrote:
> >>> > > > On Wed, 2018-03-14 at 16:42 +0800, Eddie Horng wrote:
> >>> > > > > Hi Amir,
> >>> > > > > Since the flock issue is clarified, I would like to start
> >>> > > > > this new
> >>> > > > > thread to discuss if we can find the cause of
> >>> > > > > d_type=DT_UNKNOWN.
> >>> > > > > First
> >>> > > >
> >>> > > > This sounds like NOTABUG to me. As readdir(3) states:
> >>> > > >
> >>> > > > Currently, only some filesystems (among them: Btrfs, ext2,
> >>> > > > ext3,
> >>> > > > and ext4) have full
> >>> > > > support  for  returning  the  file  type  in
> >>> > > > d_type.   All  applications  must  properly  handle  a return
> >>> > > > of
> >>> > > > DT_UNKNOWN.
> >>> > > >
> >>> > > > Applications that rely solely on d_type are effectively broken.
> >>> > > > You
> >>> > > > always need to be able to follow up with a stat or equivalent.
> >>> > > >
> >>> > >
> >>> > > Yes, but one of the main such applications is the "find" utility,
> >>> > > which
> >>> > > uses it to avoid calling stat() in order to discover the
> >>> > > directories.
> >>> > > For that reason, NFS does try to set the d_type flag when it is
> >>> > > using
> >>> > > readdirplus, and the server returns attributes for the entry in
> >>> > > question. Otherwise, it is forced to default to DT_UNKNOWN.
> >>> > >
> >>> >
> >>> > Yes, didn't mean to imply that we shouldn't try to fill these out
> >>> > where
> >>> > we can, just that there are situations where we might not be able
> >>> > to do
> >>> > so without taking a performance hit.
> >>> >
> >>> > > Note that in the cases where the readdir entry has a matching
> >>> > > dentry,
> >>> > > we probably could try to do better by doing a d_lookup() and then
> >>> > > filling the d_type. Is that worth doing?
> >>> > >
> >>> >
> >>> > I like that idea. Filling out what info we can from the local cache
> >>> > is
> >>> > almost always worthwhile.
> >>> >
> >>> > An inode's d_type can never change, so you can just vet the fileid
> >>> > or fh
> >>> > in the entry3 vs. the inode that comes back from d_lookup. If they
> >>> > match
> >>> > then you can reliably fill that out.
> >>> >
> >>>
> >>> Ironically, this is where NFS over overlayfs may fail, because in
> >>> overlayfs
> >>> d_ino is not always consistent with st_ino. Since v4.15, d_ino is
> >>> consistent
> >>> with st_ino for the case of all layers on the same filesystem. I
> >>> already posted
> >>> a POC for fixing d_ino/st_ino for non-samefs, but it never got
> >>> merged.
> >>>
> >>> What puzzles me w.r.t. this "nonbug" report is that I don't
> >>> understand why
> >>> NFS over overlayfs would behave differently vs. NFS over local fs.
> >>> I am hoping it does not point to a different problem, so would love
> >>> to
> >>> get a more detailed analysis of what's going on between nfsd and
> >>> overlayfs.
> >>
> >> The behaviour being described is true of the regular NFS client. It has
> >> nothing to do with overlayfs.
> >>
> >
> > Here:
> > https://marc.info/?l=linux-unionfs&m=152093050204357&w=2
> >
> > Eddie claims that the observed DT_UNKNOWN issue is reproduced
> > with NFSv3 over overlayfs and NOT with NFSv3 over ext4.
> >
> > That's the only fact I find puzzling.
> >
> > But I did not see a confirmation that this is a reproducible problem
> > with exact same directory and I did not try to reproduce it myself.
> >
> > Thanks,
> > Amir.
>

[-- Attachment #2: Type: text/html, Size: 6967 bytes --]

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Eddie Horng]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

I tried to track the difference between overlay-NFSv3 and ext4-NFSv3
of encode_post_op_attr.
mount configuration:
none /share overlay
rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base/work,index=on,nfs_export=on
0 0
localhost:/share /mnt/n nfs
rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
0 0
/dev/loop0 /share2 ext4 rw,relatime,data=ordered 0 0
localhost:/share2 /mnt/n2 nfs
rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
0 0

file tree:
/mnt/n
|-- dirA
|   `-- bar
|-- dirL
|   `-- ro-file
`-- foo

/mnt/n2
`-- lost+found

Attached log are dmesg start from "readdir /mnt/n (n2)" with nfs and
nfsd log are enabled all by sunrpc.nfs(d)_debug. I also add a
dump_stack() in the beginning of encode_post_op_attr.

It seems overlay and ext4 have different call flow after "nfsd:
READDIR+", there is no failure in overlay's encode_post_op_attr, nfsd
does fill the attrs, same as ext4 but it has additional readdir call
to child node of "/". I'm not sure if this is normal to overlay and is
the cause of DT_UNKNOWN.

In additional, overlay returns DT_UNKNOWN to either lower dir or merged dir.

/readdir/a.out /mnt/n
readdir: dirL ino(e007c) type(0)
readdir: . ino(e000c) type(0)
readdir: .. ino(e000c) type(0)
readdir: foo ino(e0005) type(0)
readdir: dirA ino(e0006) type(0)

/readdir/a.out /mnt/n2
readdir: lost+found ino(b) type(4)
readdir: .. ino(2) type(0)
readdir: . ino(2) type(4)

thanks,
Eddie

[-- Attachment #2: ext4_nfs3_readdir.log --]
[-- Type: application/octet-stream, Size: 9561 bytes --]

Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176671] NFS: revalidating (0:41/2)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176676] NFS call  getattr
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176803] nfsd_dispatch: vers 3 proc 1
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176810] nfsd: GETATTR(3)  20: 00060001 a2a26c5f 22448dd1 0ed04dba d211a8f6 00000000
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176814] nfsd: fh_verify(20: 00060001 a2a26c5f 22448dd1 0ed04dba d211a8f6 00000000)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176968] NFS reply getattr: 0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176976] NFS: nfs_update_inode(0:41/2 fh_crc=0x393b721e ct=3 info=0x27e7f)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176979] NFS: (0:41/2) revalidation complete
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176982] NFS: nfs_weak_revalidate: inode 2 is valid
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176988] NFS: permission(0:41/2), mask=0x24, res=0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.176993] NFS: open dir(/)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177125] NFS: readdir(/) starting at cookie 0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177134] NFS call  readdirplus 0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177252] nfsd_dispatch: vers 3 proc 17
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177260] nfsd: READDIR+(3) 20: 00060001 a2a26c5f 22448dd1 0ed04dba d211a8f6 00000000 4096 bytes at 0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177264] nfsd: fh_verify(20: 00060001 a2a26c5f 22448dd1 0ed04dba d211a8f6 00000000)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177274] nfsd: fh_verify(20: 00060001 a2a26c5f 22448dd1 0ed04dba d211a8f6 00000000)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177301] nfsd: fh_compose(exp 07:00/2 /lost+found, ino=11)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177304] encode_post_op_attr, dentry=/lost+found, is_positive=1
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177309] CPU: 0 PID: 1151 Comm: nfsd Not tainted 4.16.0-rc5-nfs #1
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177315] Hardware name: LENOVO 10FVS0XQ00/30BE, BIOS FWKT5AA   09/23/2016
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177317] Call Trace:
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177327]  dump_stack+0x63/0x8b
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177350]  encode_post_op_attr.isra.6+0x40/0xf0 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177355]  ? printk+0x43/0x4b
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177362]  ? exportfs_encode_fh+0x5a/0x70
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177376]  ? _fh_update.isra.14.part.15+0x40/0x60 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177390]  ? fh_compose+0x33d/0x4e0 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177394]  ? _cond_resched+0x1a/0x50
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177398]  ? dput+0x34/0x1f0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177415]  encode_entryplus_baggage+0x11c/0x190 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177431]  encode_entry.isra.12+0x2b2/0x310 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177448]  ? nfs3svc_encode_entry+0x20/0x20 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177463]  nfs3svc_encode_entry_plus+0x14/0x20 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177477]  nfsd_readdir+0x162/0x200 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177490]  ? nfsd_direct_splice_actor+0x20/0x20 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177505]  nfsd3_proc_readdirplus+0xeb/0x250 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177519]  ? nfsd3_proc_readdirplus+0xeb/0x250 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177532]  nfsd_dispatch+0xb3/0x240 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177560]  svc_process_common+0x351/0x650 [sunrpc]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177586]  svc_process+0xfa/0x1c0 [sunrpc]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177599]  nfsd+0xe9/0x150 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177606]  kthread+0x102/0x140
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177618]  ? nfsd_destroy+0x60/0x60 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177624]  ? kthread_associate_blkcg+0xa0/0xa0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177629]  ? do_syscall_64+0x6d/0x120
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177633]  ? SyS_exit_group+0x14/0x20
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177639]  ret_from_fork+0x35/0x40
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177649] nfsd: fh_compose(exp 07:00/2 /, ino=2)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177651] encode_post_op_attr, dentry=/, is_positive=1
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177654] CPU: 0 PID: 1151 Comm: nfsd Not tainted 4.16.0-rc5-nfs #1
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177658] Hardware name: LENOVO 10FVS0XQ00/30BE, BIOS FWKT5AA   09/23/2016
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177659] Call Trace:
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177663]  dump_stack+0x63/0x8b
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177678]  encode_post_op_attr.isra.6+0x40/0xf0 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177681]  ? printk+0x43/0x4b
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177695]  ? fh_compose+0xfd/0x4e0 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177699]  ? _cond_resched+0x1a/0x50
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177702]  ? dput+0x34/0x1f0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177716]  encode_entryplus_baggage+0x11c/0x190 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177729]  encode_entry.isra.12+0x2b2/0x310 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177745]  ? nfs3svc_encode_entry+0x20/0x20 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177759]  nfs3svc_encode_entry_plus+0x14/0x20 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177773]  nfsd_readdir+0x162/0x200 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177786]  ? nfsd_direct_splice_actor+0x20/0x20 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177800]  nfsd3_proc_readdirplus+0xeb/0x250 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177812]  ? nfsd3_proc_readdirplus+0xeb/0x250 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177825]  nfsd_dispatch+0xb3/0x240 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177850]  svc_process_common+0x351/0x650 [sunrpc]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177873]  svc_process+0xfa/0x1c0 [sunrpc]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177886]  nfsd+0xe9/0x150 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177891]  kthread+0x102/0x140
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177904]  ? nfsd_destroy+0x60/0x60 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177909]  ? kthread_associate_blkcg+0xa0/0xa0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177914]  ? do_syscall_64+0x6d/0x120
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177917]  ? SyS_exit_group+0x14/0x20
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177922]  ret_from_fork+0x35/0x40
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177931] encode_post_op_attr, dentry=/, is_positive=1
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177934] CPU: 0 PID: 1151 Comm: nfsd Not tainted 4.16.0-rc5-nfs #1
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177937] Hardware name: LENOVO 10FVS0XQ00/30BE, BIOS FWKT5AA   09/23/2016
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177938] Call Trace:
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177942]  dump_stack+0x63/0x8b
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177956]  encode_post_op_attr.isra.6+0x40/0xf0 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177970]  ? nfsd_readdir+0x19f/0x200 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177983]  ? nfsd_direct_splice_actor+0x20/0x20 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.177997]  ? nfsd3_proc_readdirplus+0xeb/0x250 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178013]  nfs3svc_encode_readdirres+0x23/0xd0 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178026]  nfsd_dispatch+0x18a/0x240 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178049]  svc_process_common+0x351/0x650 [sunrpc]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178072]  svc_process+0xfa/0x1c0 [sunrpc]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178085]  nfsd+0xe9/0x150 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178090]  kthread+0x102/0x140
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178102]  ? nfsd_destroy+0x60/0x60 [nfsd]
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178107]  ? kthread_associate_blkcg+0xa0/0xa0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178112]  ? do_syscall_64+0x6d/0x120
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178115]  ? SyS_exit_group+0x14/0x20
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178120]  ret_from_fork+0x35/0x40
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178264] NFS: nfs_update_inode(0:41/2 fh_crc=0x393b721e ct=3 info=0x27e7f)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178268] NFS reply readdirplus: 348
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178281] NFS: nfs_fhget(0:41/11 fh_crc=0x4d899cb3 ct=1)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178285] NFS: dentry_delete(/lost+found, 20488c)
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178298] NFS: nfs_do_filldir() filling ended @ cookie 9223372036854775807; returning = 0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178300] NFS: readdir(/) returns 0
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178378] NFS: readdir(/) starting at cookie 3
Mar 15 15:49:21 eddie-linuxpc kernel: [ 6648.178381] NFS: readdir(/) returns 0

[-- Attachment #3: overlay_nfs3_readdir.log --]
[-- Type: application/octet-stream, Size: 4049 bytes --]

Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.149932] NFS: revalidating (0:40/917516)
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.149935] NFS call  getattr
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150051] nfsd_dispatch: vers 3 proc 1
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150058] nfsd: GETATTR(3)  8: 00010001 00000026 00000000 00000000 00000000 00000000
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150061] nfsd: fh_verify(8: 00010001 00000026 00000000 00000000 00000000 00000000)
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150200] NFS reply getattr: 0
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150208] NFS: nfs_update_inode(0:40/917516 fh_crc=0xe78dfcb0 ct=3 info=0x27e7f)
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150211] NFS: (0:40/917516) revalidation complete
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150214] NFS: nfs_weak_revalidate: inode 917516 is valid
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150221] NFS: permission(0:40/917516), mask=0x24, res=0
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150225] NFS: open dir(/)
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150368] NFS: readdir(/) starting at cookie 0
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150378] NFS call  readdirplus 0
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150472] nfsd_dispatch: vers 3 proc 17
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150480] nfsd: READDIR+(3) 8: 00010001 00000026 00000000 00000000 00000000 00000000 4096 bytes at 0
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150483] nfsd: fh_verify(8: 00010001 00000026 00000000 00000000 00000000 00000000)
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150495] nfsd: fh_verify(8: 00010001 00000026 00000000 00000000 00000000 00000000)
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150551] encode_post_op_attr, dentry=/, is_positive=1
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150556] CPU: 3 PID: 1151 Comm: nfsd Not tainted 4.16.0-rc5-nfs #1
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150563] Hardware name: LENOVO 10FVS0XQ00/30BE, BIOS FWKT5AA   09/23/2016
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150564] Call Trace:
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150575]  dump_stack+0x63/0x8b
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150599]  encode_post_op_attr.isra.6+0x40/0xf0 [nfsd]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150614]  ? nfsd_readdir+0x19f/0x200 [nfsd]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150628]  ? nfsd_direct_splice_actor+0x20/0x20 [nfsd]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150646]  ? nfsd3_proc_readdirplus+0xeb/0x250 [nfsd]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150663]  nfs3svc_encode_readdirres+0x23/0xd0 [nfsd]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150677]  nfsd_dispatch+0x18a/0x240 [nfsd]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150706]  svc_process_common+0x351/0x650 [sunrpc]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150733]  svc_process+0xfa/0x1c0 [sunrpc]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150747]  nfsd+0xe9/0x150 [nfsd]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150753]  kthread+0x102/0x140
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150766]  ? nfsd_destroy+0x60/0x60 [nfsd]
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150772]  ? kthread_associate_blkcg+0xa0/0xa0
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150777]  ? do_syscall_64+0x6d/0x120
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150781]  ? SyS_exit_group+0x14/0x20
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150787]  ret_from_fork+0x35/0x40
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150917] NFS: nfs_update_inode(0:40/917516 fh_crc=0xe78dfcb0 ct=3 info=0x27e7f)
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150921] NFS reply readdirplus: 188
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150938] NFS: nfs_do_filldir() filling ended @ cookie 5; returning = 0
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.150940] NFS: readdir(/) returns 0
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.151034] NFS: readdir(/) starting at cookie 5
Mar 15 15:42:43 eddie-linuxpc kernel: [ 6250.151038] NFS: readdir(/) returns 0

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Amir Goldstein]

On Thu, Mar 15, 2018 at 11:47 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> I tried to track the difference between overlay-NFSv3 and ext4-NFSv3
> of encode_post_op_attr.
> mount configuration:
> none /share overlay
> rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base/work,index=on,nfs_export=on
> 0 0
> localhost:/share /mnt/n nfs
> rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
> 0 0
> /dev/loop0 /share2 ext4 rw,relatime,data=ordered 0 0
> localhost:/share2 /mnt/n2 nfs
> rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
> 0 0
>
> file tree:
> /mnt/n
> |-- dirA
> |   `-- bar
> |-- dirL
> |   `-- ro-file
> `-- foo
>
> /mnt/n2
> `-- lost+found
>
> Attached log are dmesg start from "readdir /mnt/n (n2)" with nfs and
> nfsd log are enabled all by sunrpc.nfs(d)_debug. I also add a
> dump_stack() in the beginning of encode_post_op_attr.
>
> It seems overlay and ext4 have different call flow after "nfsd:
> READDIR+", there is no failure in overlay's encode_post_op_attr, nfsd
> does fill the attrs, same as ext4 but it has additional readdir call
> to child node of "/". I'm not sure if this is normal to overlay and is
> the cause of DT_UNKNOWN.

I tried to follow nfsd readdir code to see where the difference can come
from, but nothing poped at me so far.

>
> In additional, overlay returns DT_UNKNOWN to either lower dir or merged dir.

Your test is readdir of a merged dir, the behavior could differ when readdir of
an overlayfs non merged dir.

Please send the output and dmesg of readdir of the non-merged lower dir:

/readdir/a.out /mnt/n/dirL

Please mount an NFS share of /base and send the output of the same dir:

/readdir/a.out /mnt/n_base/lower/n/dirL

Right now I speculate that the issue you report is related to
how merged dirs are iterated, but not sure exactly why.

Thanks,
Amir.

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Trond Myklebust]

On Thu, 2018-03-15 at 15:13 +0200, Amir Goldstein wrote:
> On Thu, Mar 15, 2018 at 11:47 AM, Eddie Horng <eddiehorng.tw@gmail.co
> m> wrote:
> > I tried to track the difference between overlay-NFSv3 and ext4-
> > NFSv3
> > of encode_post_op_attr.
> > mount configuration:
> > none /share overlay
> > rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base
> > /work,index=on,nfs_export=on
> > 0 0
> > localhost:/share /mnt/n nfs
> > rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,prot
> > o=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,m
> > ountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
> > 0 0
> > /dev/loop0 /share2 ext4 rw,relatime,data=ordered 0 0
> > localhost:/share2 /mnt/n2 nfs
> > rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,prot
> > o=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,m
> > ountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
> > 0 0
> > 
> > file tree:
> > /mnt/n
> > > -- dirA
> > >   `-- bar
> > > -- dirL
> > >   `-- ro-file
> > 
> > `-- foo
> > 
> > /mnt/n2
> > `-- lost+found
> > 
> > Attached log are dmesg start from "readdir /mnt/n (n2)" with nfs
> > and
> > nfsd log are enabled all by sunrpc.nfs(d)_debug. I also add a
> > dump_stack() in the beginning of encode_post_op_attr.
> > 
> > It seems overlay and ext4 have different call flow after "nfsd:
> > READDIR+", there is no failure in overlay's encode_post_op_attr,
> > nfsd
> > does fill the attrs, same as ext4 but it has additional readdir
> > call
> > to child node of "/". I'm not sure if this is normal to overlay and
> > is
> > the cause of DT_UNKNOWN.
> 
> I tried to follow nfsd readdir code to see where the difference can
> come
> from, but nothing poped at me so far.
> 
> > 
> > In additional, overlay returns DT_UNKNOWN to either lower dir or
> > merged dir.
> 
> Your test is readdir of a merged dir, the behavior could differ when
> readdir of
> an overlayfs non merged dir.
> 
> Please send the output and dmesg of readdir of the non-merged lower
> dir:
> 
> /readdir/a.out /mnt/n/dirL
> 
> Please mount an NFS share of /base and send the output of the same
> dir:
> 
> /readdir/a.out /mnt/n_base/lower/n/dirL
> 
> Right now I speculate that the issue you report is related to
> how merged dirs are iterated, but not sure exactly why.

fs/nfsd has nothing to do with ${SUBJECT}.

If you want to trace where the DT_UNKNOWN is coming from, then you need
to look at the _client_ code in fs/nfs/nfs3xdr.c:nfs3_decode_dirent(). 

-- 
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust@primarydata.com

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Eddie Horng]

Hi Trond,
As previous post I traced nfs3_decode_dirent and found *p==xdr_zero in
decode_post_op_attr so fattr is not actually decoded from xdr. Could
you suggest where to trace the xdr is encoded?

decode_post_op_attr()
    p = xdr_inline_decode(xdr, 4);
    if (*p != xdr_zero)
       return decode_fattr3(xdr, fattr);

nfs3_decode_dirent()
    entry->d_type = DT_UNKNOWN;
    if (entry->fattr->valid & NFS_ATTR_FATTR_V3)
        entry->d_type = nfs_umode_to_dtype(entry->fattr->mode);

thanks,
Eddie

2018-03-15 21:22 GMT+08:00 Trond Myklebust <trondmy@primarydata.com>:
> On Thu, 2018-03-15 at 15:13 +0200, Amir Goldstein wrote:
>> On Thu, Mar 15, 2018 at 11:47 AM, Eddie Horng <eddiehorng.tw@gmail.co
>> m> wrote:
>> > I tried to track the difference between overlay-NFSv3 and ext4-
>> > NFSv3
>> > of encode_post_op_attr.
>> > mount configuration:
>> > none /share overlay
>> > rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base
>> > /work,index=on,nfs_export=on
>> > 0 0
>> > localhost:/share /mnt/n nfs
>> > rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,prot
>> > o=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,m
>> > ountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
>> > 0 0
>> > /dev/loop0 /share2 ext4 rw,relatime,data=ordered 0 0
>> > localhost:/share2 /mnt/n2 nfs
>> > rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,prot
>> > o=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,m
>> > ountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
>> > 0 0
>> >
>> > file tree:
>> > /mnt/n
>> > > -- dirA
>> > >   `-- bar
>> > > -- dirL
>> > >   `-- ro-file
>> >
>> > `-- foo
>> >
>> > /mnt/n2
>> > `-- lost+found
>> >
>> > Attached log are dmesg start from "readdir /mnt/n (n2)" with nfs
>> > and
>> > nfsd log are enabled all by sunrpc.nfs(d)_debug. I also add a
>> > dump_stack() in the beginning of encode_post_op_attr.
>> >
>> > It seems overlay and ext4 have different call flow after "nfsd:
>> > READDIR+", there is no failure in overlay's encode_post_op_attr,
>> > nfsd
>> > does fill the attrs, same as ext4 but it has additional readdir
>> > call
>> > to child node of "/". I'm not sure if this is normal to overlay and
>> > is
>> > the cause of DT_UNKNOWN.
>>
>> I tried to follow nfsd readdir code to see where the difference can
>> come
>> from, but nothing poped at me so far.
>>
>> >
>> > In additional, overlay returns DT_UNKNOWN to either lower dir or
>> > merged dir.
>>
>> Your test is readdir of a merged dir, the behavior could differ when
>> readdir of
>> an overlayfs non merged dir.
>>
>> Please send the output and dmesg of readdir of the non-merged lower
>> dir:
>>
>> /readdir/a.out /mnt/n/dirL
>>
>> Please mount an NFS share of /base and send the output of the same
>> dir:
>>
>> /readdir/a.out /mnt/n_base/lower/n/dirL
>>
>> Right now I speculate that the issue you report is related to
>> how merged dirs are iterated, but not sure exactly why.
>
> fs/nfsd has nothing to do with ${SUBJECT}.
>
> If you want to trace where the DT_UNKNOWN is coming from, then you need
> to look at the _client_ code in fs/nfs/nfs3xdr.c:nfs3_decode_dirent().
>
> --
> Trond Myklebust
> Linux NFS client maintainer, PrimaryData
> trond.myklebust@primarydata.com

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Amir Goldstein]

On Thu, Mar 15, 2018 at 4:30 PM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> Hi Trond,
> As previous post I traced nfs3_decode_dirent and found *p==xdr_zero in
> decode_post_op_attr so fattr is not actually decoded from xdr. Could
> you suggest where to trace the xdr is encoded?
>
> decode_post_op_attr()
>     p = xdr_inline_decode(xdr, 4);
>     if (*p != xdr_zero)
>        return decode_fattr3(xdr, fattr);
>
> nfs3_decode_dirent()
>     entry->d_type = DT_UNKNOWN;
>     if (entry->fattr->valid & NFS_ATTR_FATTR_V3)
>         entry->d_type = nfs_umode_to_dtype(entry->fattr->mode);
>

Eddie,

Please don't "top post".
You are right. Problem is with overlayfs on server side.
See the explanation to the problem below.


>
> 2018-03-15 21:22 GMT+08:00 Trond Myklebust <trondmy@primarydata.com>:
>> On Thu, 2018-03-15 at 15:13 +0200, Amir Goldstein wrote:
>>> On Thu, Mar 15, 2018 at 11:47 AM, Eddie Horng <eddiehorng.tw@gmail.co
>>> m> wrote:
>>> > I tried to track the difference between overlay-NFSv3 and ext4-
>>> > NFSv3
>>> > of encode_post_op_attr.
>>> > mount configuration:
>>> > none /share overlay
>>> > rw,relatime,lowerdir=/base/lower,upperdir=/base/upper,workdir=/base
>>> > /work,index=on,nfs_export=on
>>> > 0 0
>>> > localhost:/share /mnt/n nfs
>>> > rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,prot
>>> > o=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,m
>>> > ountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
>>> > 0 0
>>> > /dev/loop0 /share2 ext4 rw,relatime,data=ordered 0 0
>>> > localhost:/share2 /mnt/n2 nfs
>>> > rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,prot
>>> > o=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,m
>>> > ountport=40931,mountproto=udp,local_lock=none,addr=127.0.0.1
>>> > 0 0
>>> >
>>> > file tree:
>>> > /mnt/n
>>> > > -- dirA
>>> > >   `-- bar
>>> > > -- dirL
>>> > >   `-- ro-file
>>> >
>>> > `-- foo
>>> >
>>> > /mnt/n2
>>> > `-- lost+found
>>> >
>>> > Attached log are dmesg start from "readdir /mnt/n (n2)" with nfs
>>> > and
>>> > nfsd log are enabled all by sunrpc.nfs(d)_debug. I also add a
>>> > dump_stack() in the beginning of encode_post_op_attr.
>>> >
>>> > It seems overlay and ext4 have different call flow after "nfsd:
>>> > READDIR+", there is no failure in overlay's encode_post_op_attr,
>>> > nfsd
>>> > does fill the attrs, same as ext4 but it has additional readdir
>>> > call
>>> > to child node of "/". I'm not sure if this is normal to overlay and
>>> > is
>>> > the cause of DT_UNKNOWN.
>>>
[...]
>>
>> fs/nfsd has nothing to do with ${SUBJECT}.
>>
>> If you want to trace where the DT_UNKNOWN is coming from, then you need
>> to look at the _client_ code in fs/nfs/nfs3xdr.c:nfs3_decode_dirent().
>>

The problem *is* with nfsd+overlayfs, because nfsd verifies
in compose_entry_fh() that (dchild->d_inode->i_ino == ino), but it is not.
In that case, encode_entryplus_baggage() falls back to encoding xdr_zero.
In overlayfs stat.st_ino is consistent with readdir d_ino since kernel version
v4.15 and only for all layers on the same fs.

However, there is no guaranty that inode->i_ino is the same as stat.st_ino.
Overlayfs exposes only stat.st_ino to user (as well as readdir d_ino), but
never (to my knowledge) does it expose inode->i_ino.

There was a nfsd fix for a somewhat similar problem that went into v4.16-rc1:
76c479480b9a nfsd: encode stat->mtime for getattr instead of inode->i_mtime

The solution to the problem is to either convert all references of
nfsd to i_inode
with references to st_ino, or make sure to set inode->i_ino correctly for
overlayfs inodes.

>From first glimps, the change in nfsd looks non trivial.
The change to overlayfs seems doable, but didn't look closely yet.
Will try to come up with test patch for you.

Thanks for reporting and following up with debugging!
Amir.

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Amir Goldstein]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

On Thu, Mar 15, 2018 at 8:40 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> On Thu, Mar 15, 2018 at 4:30 PM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
[...]
> The problem *is* with nfsd+overlayfs, because nfsd verifies
> in compose_entry_fh() that (dchild->d_inode->i_ino == ino), but it is not.
> In that case, encode_entryplus_baggage() falls back to encoding xdr_zero.
> In overlayfs stat.st_ino is consistent with readdir d_ino since kernel version
> v4.15 and only for all layers on the same fs.
>
> However, there is no guaranty that inode->i_ino is the same as stat.st_ino.
> Overlayfs exposes only stat.st_ino to user (as well as readdir d_ino), but
> never (to my knowledge) does it expose inode->i_ino.
>
> There was a nfsd fix for a somewhat similar problem that went into v4.16-rc1:
> 76c479480b9a nfsd: encode stat->mtime for getattr instead of inode->i_mtime
>
> The solution to the problem is to either convert all references of
> nfsd to i_inode
> with references to st_ino, or make sure to set inode->i_ino correctly for
> overlayfs inodes.
>
> From first glimps, the change in nfsd looks non trivial.
> The change to overlayfs seems doable, but didn't look closely yet.
> Will try to come up with test patch for you.
>

Eddie,

Please try this patch. It worked for me.

Thanks,
Amir.

[-- Attachment #2: 0001-ovl-set-i_ino-to-the-value-of-st_ino-for-NFS-export.patch --]
[-- Type: text/x-patch, Size: 4092 bytes --]

From 5dfe75eaf1aec080d110df343aa339ad7b816670 Mon Sep 17 00:00:00 2001
From: Amir Goldstein <amir73il@gmail.com>
Date: Thu, 15 Mar 2018 23:39:01 +0200
Subject: [PATCH] ovl: set i_ino to the value of st_ino for NFS export

When NFS export is enabled and d_ino is consistent with st_ino
(samefs), set the same value to i_ino, because nfsd readdirplus
compares d_ino values to i_ino values of child entries. When called
from ovl_new_inode(), ino arg is 0, so i_ino will be updated to real
upper inode i_ino on ovl_inode_init() or ovl_inode_update().

Reported-by: Eddie Horng <eddiehorng.tw@gmail.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 fs/overlayfs/inode.c | 21 +++++++++++++++++----
 fs/overlayfs/util.c  |  8 +++++++-
 2 files changed, 24 insertions(+), 5 deletions(-)

diff --git a/fs/overlayfs/inode.c b/fs/overlayfs/inode.c
index 3b1bd469accd..4689716f23d8 100644
--- a/fs/overlayfs/inode.c
+++ b/fs/overlayfs/inode.c
@@ -459,9 +459,20 @@ static inline void ovl_lockdep_annotate_inode_mutex_key(struct inode *inode)
 #endif
 }
 
-static void ovl_fill_inode(struct inode *inode, umode_t mode, dev_t rdev)
+static void ovl_fill_inode(struct inode *inode, umode_t mode, dev_t rdev,
+			   unsigned long ino)
 {
-	inode->i_ino = get_next_ino();
+	/*
+	 * When NFS export is enabled and d_ino is consistent with st_ino
+	 * (samefs), set the same value to i_ino, because nfsd readdirplus
+	 * compares d_ino values to i_ino values of child entries. When called
+	 * from ovl_new_inode(), ino arg is 0, so i_ino will be updated to real
+	 * upper inode i_ino on ovl_inode_init() or ovl_inode_update().
+	 */
+	if (inode->i_sb->s_export_op && ovl_same_sb(inode->i_sb))
+		inode->i_ino = ino;
+	else
+		inode->i_ino = get_next_ino();
 	inode->i_mode = mode;
 	inode->i_flags |= S_NOCMTIME;
 #ifdef CONFIG_FS_POSIX_ACL
@@ -597,7 +608,7 @@ struct inode *ovl_new_inode(struct super_block *sb, umode_t mode, dev_t rdev)
 
 	inode = new_inode(sb);
 	if (inode)
-		ovl_fill_inode(inode, mode, rdev);
+		ovl_fill_inode(inode, mode, rdev, 0);
 
 	return inode;
 }
@@ -710,6 +721,7 @@ struct inode *ovl_get_inode(struct super_block *sb, struct dentry *upperdentry,
 	struct inode *inode;
 	bool bylower = ovl_hash_bylower(sb, upperdentry, lowerdentry, index);
 	bool is_dir;
+	unsigned long ino = 0;
 
 	if (!realinode)
 		realinode = d_inode(lowerdentry);
@@ -748,13 +760,14 @@ struct inode *ovl_get_inode(struct super_block *sb, struct dentry *upperdentry,
 		if (!is_dir)
 			nlink = ovl_get_nlink(lowerdentry, upperdentry, nlink);
 		set_nlink(inode, nlink);
+		ino = key->i_ino;
 	} else {
 		/* Lower hardlink that will be broken on copy up */
 		inode = new_inode(sb);
 		if (!inode)
 			goto out_nomem;
 	}
-	ovl_fill_inode(inode, realinode->i_mode, realinode->i_rdev);
+	ovl_fill_inode(inode, realinode->i_mode, realinode->i_rdev, ino);
 	ovl_inode_init(inode, upperdentry, lowerdentry);
 
 	if (upperdentry && ovl_is_impuredir(upperdentry))
diff --git a/fs/overlayfs/util.c b/fs/overlayfs/util.c
index 930784a26623..493f9b76fbf6 100644
--- a/fs/overlayfs/util.c
+++ b/fs/overlayfs/util.c
@@ -279,12 +279,16 @@ void ovl_dentry_set_redirect(struct dentry *dentry, const char *redirect)
 void ovl_inode_init(struct inode *inode, struct dentry *upperdentry,
 		    struct dentry *lowerdentry)
 {
+	struct inode *realinode = d_inode(upperdentry ?: lowerdentry);
+
 	if (upperdentry)
 		OVL_I(inode)->__upperdentry = upperdentry;
 	if (lowerdentry)
 		OVL_I(inode)->lower = igrab(d_inode(lowerdentry));
 
-	ovl_copyattr(d_inode(upperdentry ?: lowerdentry), inode);
+	ovl_copyattr(realinode, inode);
+	if (!inode->i_ino)
+		inode->i_ino = realinode->i_ino;
 }
 
 void ovl_inode_update(struct inode *inode, struct dentry *upperdentry)
@@ -299,6 +303,8 @@ void ovl_inode_update(struct inode *inode, struct dentry *upperdentry)
 	smp_wmb();
 	OVL_I(inode)->__upperdentry = upperdentry;
 	if (inode_unhashed(inode)) {
+		if (!inode->i_ino)
+			inode->i_ino = upperinode->i_ino;
 		inode->i_private = upperinode;
 		__insert_inode_hash(inode, (unsigned long) upperinode);
 	}
-- 
2.7.4

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Eddie Horng]

2018-03-16 5:48 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
> On Thu, Mar 15, 2018 at 8:40 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>> On Thu, Mar 15, 2018 at 4:30 PM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> [...]
>> The problem *is* with nfsd+overlayfs, because nfsd verifies
>> in compose_entry_fh() that (dchild->d_inode->i_ino == ino), but it is not.
>> In that case, encode_entryplus_baggage() falls back to encoding xdr_zero.
>> In overlayfs stat.st_ino is consistent with readdir d_ino since kernel version
>> v4.15 and only for all layers on the same fs.
>>
>> However, there is no guaranty that inode->i_ino is the same as stat.st_ino.
>> Overlayfs exposes only stat.st_ino to user (as well as readdir d_ino), but
>> never (to my knowledge) does it expose inode->i_ino.
>>
>> There was a nfsd fix for a somewhat similar problem that went into v4.16-rc1:
>> 76c479480b9a nfsd: encode stat->mtime for getattr instead of inode->i_mtime
>>
>> The solution to the problem is to either convert all references of
>> nfsd to i_inode
>> with references to st_ino, or make sure to set inode->i_ino correctly for
>> overlayfs inodes.
>>
>> From first glimps, the change in nfsd looks non trivial.
>> The change to overlayfs seems doable, but didn't look closely yet.
>> Will try to come up with test patch for you.
>>
>
> Eddie,
>
> Please try this patch. It worked for me.
>
> Thanks,
> Amir.

Hi Amir,
The patch works like a charm in my first test, but when I switch lower
layer to different fs then upper and merged, problem comes again. The
configuration is:
/dev/loop0 /ro_mount ext4 rw,relatime,data=ordered 0 0
none /share overlay
rw,relatime,lowerdir=/ro_mount,upperdir=/base/upper,workdir=/base/work,index=on,nfs_export=on
0 0
localhost:/share /mnt/n nfs
rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=40970,mountproto=udp,local_lock=none,addr=127.0.0.1
0 0

/ro_mount/
|-- lost+found
|-- ro-dir
`-- ro-file

result:
readdir: lost+found ino(b) type(0)
readdir: ro-file ino(c) type(0)
readdir: .. ino(e0001) type(0)
readdir: ro-dir ino(2003) type(0)
readdir: . ino(e0003) type(0)

ps. /ro_mount is created by:
truncate -s 1g image
mkfs.ext4 image
mount image /ro_mount

thanks,
Eddie

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Amir Goldstein]

On Fri, Mar 16, 2018 at 8:25 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> 2018-03-16 5:48 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
>> On Thu, Mar 15, 2018 at 8:40 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>>> On Thu, Mar 15, 2018 at 4:30 PM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>> [...]
>>> The problem *is* with nfsd+overlayfs, because nfsd verifies
>>> in compose_entry_fh() that (dchild->d_inode->i_ino == ino), but it is not.
>>> In that case, encode_entryplus_baggage() falls back to encoding xdr_zero.
>>> In overlayfs stat.st_ino is consistent with readdir d_ino since kernel version
>>> v4.15 and only for all layers on the same fs.
>>>
>>> However, there is no guaranty that inode->i_ino is the same as stat.st_ino.
>>> Overlayfs exposes only stat.st_ino to user (as well as readdir d_ino), but
>>> never (to my knowledge) does it expose inode->i_ino.
>>>
>>> There was a nfsd fix for a somewhat similar problem that went into v4.16-rc1:
>>> 76c479480b9a nfsd: encode stat->mtime for getattr instead of inode->i_mtime
>>>
>>> The solution to the problem is to either convert all references of
>>> nfsd to i_inode
>>> with references to st_ino, or make sure to set inode->i_ino correctly for
>>> overlayfs inodes.
>>>
>>> From first glimps, the change in nfsd looks non trivial.
>>> The change to overlayfs seems doable, but didn't look closely yet.
>>> Will try to come up with test patch for you.
>>>
>>
>> Eddie,
>>
>> Please try this patch. It worked for me.
>>
>> Thanks,
>> Amir.
>
> Hi Amir,
> The patch works like a charm in my first test, but when I switch lower
> layer to different fs then upper and merged, problem comes again. The

Yes, that is expected. As I explained, overlayfs does NOT guaranty consistency
of d_ino to st_ino with non-samefs and this patch doesn't change that.
Apparently, the consequence is that with NFSv3 you will get DT_UNKNOWN.

If this is really a problem for your use case, I have already posted a solution
for non-samefs while back https://github.com/amir73il/linux/commits/ovl-xino
I can rebase and re-post if you are interested in testing.
Can I assume from your test example that you are interested in the setup of
ext4 as upper/lower (but non samefs)? Because solving the problem for ext4
(32bit ino) is a bit simpler than the general case (64bit ino).

Thanks,
Amir.

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Eddie Horng]

2018-03-16 15:23 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
> On Fri, Mar 16, 2018 at 8:25 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>> 2018-03-16 5:48 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
>>> On Thu, Mar 15, 2018 at 8:40 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>>>> On Thu, Mar 15, 2018 at 4:30 PM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>>> [...]
>>>> The problem *is* with nfsd+overlayfs, because nfsd verifies
>>>> in compose_entry_fh() that (dchild->d_inode->i_ino == ino), but it is not.
>>>> In that case, encode_entryplus_baggage() falls back to encoding xdr_zero.
>>>> In overlayfs stat.st_ino is consistent with readdir d_ino since kernel version
>>>> v4.15 and only for all layers on the same fs.
>>>>
>>>> However, there is no guaranty that inode->i_ino is the same as stat.st_ino.
>>>> Overlayfs exposes only stat.st_ino to user (as well as readdir d_ino), but
>>>> never (to my knowledge) does it expose inode->i_ino.
>>>>
>>>> There was a nfsd fix for a somewhat similar problem that went into v4.16-rc1:
>>>> 76c479480b9a nfsd: encode stat->mtime for getattr instead of inode->i_mtime
>>>>
>>>> The solution to the problem is to either convert all references of
>>>> nfsd to i_inode
>>>> with references to st_ino, or make sure to set inode->i_ino correctly for
>>>> overlayfs inodes.
>>>>
>>>> From first glimps, the change in nfsd looks non trivial.
>>>> The change to overlayfs seems doable, but didn't look closely yet.
>>>> Will try to come up with test patch for you.
>>>>
>>>
>>> Eddie,
>>>
>>> Please try this patch. It worked for me.
>>>
>>> Thanks,
>>> Amir.
>>
>> Hi Amir,
>> The patch works like a charm in my first test, but when I switch lower
>> layer to different fs then upper and merged, problem comes again. The
>
> Yes, that is expected. As I explained, overlayfs does NOT guaranty consistency
> of d_ino to st_ino with non-samefs and this patch doesn't change that.
> Apparently, the consequence is that with NFSv3 you will get DT_UNKNOWN.
>
> If this is really a problem for your use case, I have already posted a solution
> for non-samefs while back https://github.com/amir73il/linux/commits/ovl-xino
> I can rebase and re-post if you are interested in testing.
> Can I assume from your test example that you are interested in the setup of
> ext4 as upper/lower (but non samefs)? Because solving the problem for ext4
> (32bit ino) is a bit simpler than the general case (64bit ino).
>
> Thanks,
> Amir.

My setup for upper is ext4, lower however is a proprietary fs, whose
ino is 32bit.
So far it has identical behavior as ext4 to work with overlay. I'm
glad to test if the solution works
for ext4 or the proprietary fs.
In the other hand, I also try to submit a request to Google to see if
the build tool can handle
DT_UNKNOWN properly, after all the application use d_type should do that.

thanks,
Eddie

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Amir Goldstein]

On Fri, Mar 16, 2018 at 10:06 AM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
> 2018-03-16 15:23 GMT+08:00 Amir Goldstein <amir73il@gmail.com>:
[...]
>>> Hi Amir,
>>> The patch works like a charm in my first test, but when I switch lower
>>> layer to different fs then upper and merged, problem comes again. The
>>
>> Yes, that is expected. As I explained, overlayfs does NOT guaranty consistency
>> of d_ino to st_ino with non-samefs and this patch doesn't change that.
>> Apparently, the consequence is that with NFSv3 you will get DT_UNKNOWN.
>>
>> If this is really a problem for your use case, I have already posted a solution
>> for non-samefs while back https://github.com/amir73il/linux/commits/ovl-xino
>> I can rebase and re-post if you are interested in testing.
>> Can I assume from your test example that you are interested in the setup of
>> ext4 as upper/lower (but non samefs)? Because solving the problem for ext4
>> (32bit ino) is a bit simpler than the general case (64bit ino).
>>
>> Thanks,
>> Amir.
>
> My setup for upper is ext4, lower however is a proprietary fs, whose
> ino is 32bit.
> So far it has identical behavior as ext4 to work with overlay. I'm
> glad to test if the solution works
> for ext4 or the proprietary fs.
> In the other hand, I also try to submit a request to Google to see if
> the build tool can handle
> DT_UNKNOWN properly, after all the application use d_type should do that.
>

Eddie,

I rebased and push branch ovl-xino [1] to my github, it includes the
fix you already
tested.

If your proprietary filesystem does not implement the operation encode_fh()
and uses the default 32INO encoding then you don't have to specify any new
mount option. If it implements its own file handle encoding, you need to specify
overlay mount option -o xino to declare that all inode numbers are
limited to 32bit
(or at least not using 64bit).

Cheers,
Amir.

[1] https://github.com/amir73il/linux/commits/ovl-xino

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Eddie Horng]

> Eddie,
>
> I rebased and push branch ovl-xino [1] to my github, it includes the
> fix you already
> tested.
>
> If your proprietary filesystem does not implement the operation encode_fh()
> and uses the default 32INO encoding then you don't have to specify any new
> mount option. If it implements its own file handle encoding, you need to specify
> overlay mount option -o xino to declare that all inode numbers are
> limited to 32bit
> (or at least not using 64bit).
>
> Cheers,
> Amir.
>
> [1] https://github.com/amir73il/linux/commits/ovl-xino

I tested ovl-xino with my proprietary filesystem and ext4, the result
is great -- readdir returns correct d_type.
Really thanks your effort to find root cause and provide solution so fast.
In addition, do you plan to merge the fix or even the xino feature in 4.16?

Thanks,
Eddie

Re: readdir returns d_type=DT_UNKNOWN to overlay exported dir (NFSv3)

[Amir Goldstein]

On Fri, Mar 16, 2018 at 5:06 PM, Eddie Horng <eddiehorng.tw@gmail.com> wrote:
>> Eddie,
>>
>> I rebased and push branch ovl-xino [1] to my github, it includes the
>> fix you already
>> tested.
>>
>> If your proprietary filesystem does not implement the operation encode_fh()
>> and uses the default 32INO encoding then you don't have to specify any new
>> mount option. If it implements its own file handle encoding, you need to specify
>> overlay mount option -o xino to declare that all inode numbers are
>> limited to 32bit
>> (or at least not using 64bit).
>>
>> Cheers,
>> Amir.
>>
>> [1] https://github.com/amir73il/linux/commits/ovl-xino
>
> I tested ovl-xino with my proprietary filesystem and ext4, the result
> is great -- readdir returns correct d_type.
> Really thanks your effort to find root cause and provide solution so fast.
> In addition, do you plan to merge the fix or even the xino feature in 4.16?
>

I am not the one doing the merging. Miklos does.
I posted the fix for samefs and it could get merged to 4.16 if Miklos
has time to review it. May require some pinging from you if that doesn't
happen within a week.

xino feature is not for 4.16. It needs some more work and then I can
re-post it for 4.17.

Cheers,
Amir.