[ceph-fuse] fuse fails to invalidate dentry when ceph-fuse mounts sub directory as root_directory

[ceph-fuse] fuse fails to invalidate dentry when ceph-fuse mounts sub directory as root_directory

[Zhi Zhang]

Hi cepher,

Steps to reproduce this issue:
1. reduce mds_cache_size to 200 for quick verification
2. mount sub directory as root_directory, e.g, ceph-fuse -r /zhi_test
/mnt/cephfs
3. create a new sub directory under /mnt/cephfs, e.g, /mnt/cephfs/sub_dir
4. write more than 200 files into /mnt/cephfs/sub_dir/ directory
5. client should start to trim_caps, but caps are not released.

ceph version: 0.94.1 with some backports
fuse version: both 2.8.3 or 2.9.2

By further looking at the codes, ceph-fuse wants fuse to invalidate
some dentries by calling Client::_async_dentry_invalidate, which will
call dentry_invalidate_cb in fuse_ll.cc and fuse's API
fuse_lowlevel_notify_delete.

Within these calls, the parent's inode (dirino) is our mount point's
inode (zhi_test), NOT the original default root inode ("/"), the child
inode may be a fake inode or sub directory's inode (sub_dir).

However, fuse's API fuse_lowlevel_notify_delete returns -2, so no
ll_forget will be called on those files under sub_dir and those caps
are not released. MDS will report failing to respond to cache
pressure.

We have changed the parent's inode to the original default root inode
("/") in Client::_async_dentry_invalidate, but no changes on child
inode. After this, fuse can invalidate child dentry (sub_dir)
successfully and ll_forget will be called on those files under
sub_dir.

I didn't find much information from the fuse's doc
https://fossies.org/dox/fuse-2.9.5/fuse__lowlevel_8h.html#a94893b3eb83fe2a218e80e43089670a1.
But it looks like fuse looking for the child from the head of the
tree, can't from an intermediate node? otherwise is it a bug from ceph
side?

Thanks.

Regards,
Zhi Zhang (David)
Contact: zhang.david2011@gmail.com
              zhangz.david@outlook.com

Re: [ceph-fuse] fuse fails to invalidate dentry when ceph-fuse mounts sub directory as root_directory

[Yan, Zheng]

On Mon, Feb 29, 2016 at 4:13 PM, Zhi Zhang <zhang.david2011@gmail.com> wrote:
> Hi cepher,
>
> Steps to reproduce this issue:
> 1. reduce mds_cache_size to 200 for quick verification
> 2. mount sub directory as root_directory, e.g, ceph-fuse -r /zhi_test
> /mnt/cephfs
> 3. create a new sub directory under /mnt/cephfs, e.g, /mnt/cephfs/sub_dir
> 4. write more than 200 files into /mnt/cephfs/sub_dir/ directory
> 5. client should start to trim_caps, but caps are not released.
>
> ceph version: 0.94.1 with some backports
> fuse version: both 2.8.3 or 2.9.2
>
> By further looking at the codes, ceph-fuse wants fuse to invalidate
> some dentries by calling Client::_async_dentry_invalidate, which will
> call dentry_invalidate_cb in fuse_ll.cc and fuse's API
> fuse_lowlevel_notify_delete.
>
> Within these calls, the parent's inode (dirino) is our mount point's
> inode (zhi_test), NOT the original default root inode ("/"), the child
> inode may be a fake inode or sub directory's inode (sub_dir).
>
> However, fuse's API fuse_lowlevel_notify_delete returns -2, so no
> ll_forget will be called on those files under sub_dir and those caps
> are not released. MDS will report failing to respond to cache
> pressure.
>
> We have changed the parent's inode to the original default root inode
> ("/") in Client::_async_dentry_invalidate, but no changes on child
> inode. After this, fuse can invalidate child dentry (sub_dir)
> successfully and ll_forget will be called on those files under
> sub_dir.
>
> I didn't find much information from the fuse's doc
> https://fossies.org/dox/fuse-2.9.5/fuse__lowlevel_8h.html#a94893b3eb83fe2a218e80e43089670a1.
> But it looks like fuse looking for the child from the head of the
> tree, can't from an intermediate node? otherwise is it a bug from ceph
> side?

Using fuse_lowlevel_notify_delete() to invalidate dentry no longer
works for 3.18+ kernel.  See http://tracker.ceph.com/issues/9997 for
detail

Regards
Yan, Zheng


> Thanks.
>
> Regards,
> Zhi Zhang (David)
> Contact: zhang.david2011@gmail.com
>               zhangz.david@outlook.com

Re: [ceph-fuse] fuse fails to invalidate dentry when ceph-fuse mounts sub directory as root_directory

[Zhi Zhang]

Regards,
Zhi Zhang (David)
Contact: zhang.david2011@gmail.com
              zhangz.david@outlook.com


On Thu, Mar 3, 2016 at 8:16 PM, Yan, Zheng <ukernel@gmail.com> wrote:
> On Mon, Feb 29, 2016 at 4:13 PM, Zhi Zhang <zhang.david2011@gmail.com> wrote:
>> Hi cepher,
>>
>> Steps to reproduce this issue:
>> 1. reduce mds_cache_size to 200 for quick verification
>> 2. mount sub directory as root_directory, e.g, ceph-fuse -r /zhi_test
>> /mnt/cephfs
>> 3. create a new sub directory under /mnt/cephfs, e.g, /mnt/cephfs/sub_dir
>> 4. write more than 200 files into /mnt/cephfs/sub_dir/ directory
>> 5. client should start to trim_caps, but caps are not released.
>>
>> ceph version: 0.94.1 with some backports
>> fuse version: both 2.8.3 or 2.9.2
>>
>> By further looking at the codes, ceph-fuse wants fuse to invalidate
>> some dentries by calling Client::_async_dentry_invalidate, which will
>> call dentry_invalidate_cb in fuse_ll.cc and fuse's API
>> fuse_lowlevel_notify_delete.
>>
>> Within these calls, the parent's inode (dirino) is our mount point's
>> inode (zhi_test), NOT the original default root inode ("/"), the child
>> inode may be a fake inode or sub directory's inode (sub_dir).
>>
>> However, fuse's API fuse_lowlevel_notify_delete returns -2, so no
>> ll_forget will be called on those files under sub_dir and those caps
>> are not released. MDS will report failing to respond to cache
>> pressure.
>>
>> We have changed the parent's inode to the original default root inode
>> ("/") in Client::_async_dentry_invalidate, but no changes on child
>> inode. After this, fuse can invalidate child dentry (sub_dir)
>> successfully and ll_forget will be called on those files under
>> sub_dir.
>>
>> I didn't find much information from the fuse's doc
>> https://fossies.org/dox/fuse-2.9.5/fuse__lowlevel_8h.html#a94893b3eb83fe2a218e80e43089670a1.
>> But it looks like fuse looking for the child from the head of the
>> tree, can't from an intermediate node? otherwise is it a bug from ceph
>> side?
>
> Using fuse_lowlevel_notify_delete() to invalidate dentry no longer
> works for 3.18+ kernel.  See http://tracker.ceph.com/issues/9997 for
> detail
>

I am using centos-7 whose kernel is 3.10.x. I also tried to set
client_try_dentry_invalidate as false to force use remount to
invalidate dentry. It works but performance degrades.

fuse_lowlevel_notify_delete() works on kernel 3.10.x. What I found is
that the second args (fuse_ino_t parent) of this function should be
the ceph-fuse's root inode number ("/"), can't be others.


> Regards
> Yan, Zheng
>
>
>> Thanks.
>>
>> Regards,
>> Zhi Zhang (David)
>> Contact: zhang.david2011@gmail.com
>>               zhangz.david@outlook.com

Re: [ceph-fuse] fuse fails to invalidate dentry when ceph-fuse mounts sub directory as root_directory

[Yan, Zheng]

On Mon, Feb 29, 2016 at 4:13 PM, Zhi Zhang <zhang.david2011@gmail.com> wrote:
> Hi cepher,
>
> Steps to reproduce this issue:
> 1. reduce mds_cache_size to 200 for quick verification
> 2. mount sub directory as root_directory, e.g, ceph-fuse -r /zhi_test
> /mnt/cephfs
> 3. create a new sub directory under /mnt/cephfs, e.g, /mnt/cephfs/sub_dir
> 4. write more than 200 files into /mnt/cephfs/sub_dir/ directory
> 5. client should start to trim_caps, but caps are not released.
>
> ceph version: 0.94.1 with some backports
> fuse version: both 2.8.3 or 2.9.2
>
> By further looking at the codes, ceph-fuse wants fuse to invalidate
> some dentries by calling Client::_async_dentry_invalidate, which will
> call dentry_invalidate_cb in fuse_ll.cc and fuse's API
> fuse_lowlevel_notify_delete.
>
> Within these calls, the parent's inode (dirino) is our mount point's
> inode (zhi_test), NOT the original default root inode ("/"), the child
> inode may be a fake inode or sub directory's inode (sub_dir).
>
> However, fuse's API fuse_lowlevel_notify_delete returns -2, so no
> ll_forget will be called on those files under sub_dir and those caps
> are not released. MDS will report failing to respond to cache
> pressure.
>
> We have changed the parent's inode to the original default root inode
> ("/") in Client::_async_dentry_invalidate, but no changes on child
> inode. After this, fuse can invalidate child dentry (sub_dir)
> successfully and ll_forget will be called on those files under
> sub_dir.
>
> I didn't find much information from the fuse's doc
> https://fossies.org/dox/fuse-2.9.5/fuse__lowlevel_8h.html#a94893b3eb83fe2a218e80e43089670a1.
> But it looks like fuse looking for the child from the head of the
> tree, can't from an intermediate node? otherwise is it a bug from ceph
> side?

It's a bug in ceph-fuse. Fuse always uses 1 as root inode number. To
invalidate dentries in root directory, the parent's inode number
should be set to 1.

Regards
Yan, Zheng

> Thanks.
>
> Regards,
> Zhi Zhang (David)
> Contact: zhang.david2011@gmail.com
>               zhangz.david@outlook.com

Re: [ceph-fuse] fuse fails to invalidate dentry when ceph-fuse mounts sub directory as root_directory

[Gregory Farnum]

On Tue, Mar 8, 2016 at 1:06 AM, Yan, Zheng <ukernel@gmail.com> wrote:
> On Mon, Feb 29, 2016 at 4:13 PM, Zhi Zhang <zhang.david2011@gmail.com> wrote:
>> Hi cepher,
>>
>> Steps to reproduce this issue:
>> 1. reduce mds_cache_size to 200 for quick verification
>> 2. mount sub directory as root_directory, e.g, ceph-fuse -r /zhi_test
>> /mnt/cephfs
>> 3. create a new sub directory under /mnt/cephfs, e.g, /mnt/cephfs/sub_dir
>> 4. write more than 200 files into /mnt/cephfs/sub_dir/ directory
>> 5. client should start to trim_caps, but caps are not released.
>>
>> ceph version: 0.94.1 with some backports
>> fuse version: both 2.8.3 or 2.9.2
>>
>> By further looking at the codes, ceph-fuse wants fuse to invalidate
>> some dentries by calling Client::_async_dentry_invalidate, which will
>> call dentry_invalidate_cb in fuse_ll.cc and fuse's API
>> fuse_lowlevel_notify_delete.
>>
>> Within these calls, the parent's inode (dirino) is our mount point's
>> inode (zhi_test), NOT the original default root inode ("/"), the child
>> inode may be a fake inode or sub directory's inode (sub_dir).
>>
>> However, fuse's API fuse_lowlevel_notify_delete returns -2, so no
>> ll_forget will be called on those files under sub_dir and those caps
>> are not released. MDS will report failing to respond to cache
>> pressure.
>>
>> We have changed the parent's inode to the original default root inode
>> ("/") in Client::_async_dentry_invalidate, but no changes on child
>> inode. After this, fuse can invalidate child dentry (sub_dir)
>> successfully and ll_forget will be called on those files under
>> sub_dir.
>>
>> I didn't find much information from the fuse's doc
>> https://fossies.org/dox/fuse-2.9.5/fuse__lowlevel_8h.html#a94893b3eb83fe2a218e80e43089670a1.
>> But it looks like fuse looking for the child from the head of the
>> tree, can't from an intermediate node? otherwise is it a bug from ceph
>> side?
>
> It's a bug in ceph-fuse. Fuse always uses 1 as root inode number. To
> invalidate dentries in root directory, the parent's inode number
> should be set to 1.

For those following along at home, the tracker ticket for this is
http://tracker.ceph.com/issues/15008, and the PR is at
https://github.com/ceph/ceph/pull/7976.
-Greg