nfs hang bugs

nfs hang bugs

[程洋]

Hi. I'm an server engineer who hold gerrit servers. And uses nfs as my harddisk.
Some time I found when I run `git pack-refs`, it will hang, and it causes deadlock in kernel, prevent other fs command to execute like `ls`
Pretty much like this, but not the same issue.  https://about.gitlab.com/blog/2018/11/14/how-we-spent-two-weeks-hunting-an-nfs-bug/

Since I'm new to NFS, I don't know what else can I provide. If there anyway to get more debug information, please feel free to ask me.

My NFS server: Ubuntu 16.04
My NFS client: Ubuntu 18.04


`git pack-refs strace`
write(3, "users/94/1003094\n886741817c1d824"..., 8192) = 8192
write(3, "ers/95/1006795\n7ec95f6ad772f6d63"..., 8192) = 8192
write(3, "s/96/1011196\n5e310bf8491276a6a8d"..., 8192) = 8192
write(3, "98/1000998\n3e1f177f0427e7aa79065"..., 8192) = 8192
write(3, "/1004899\n14e572b05b3bb857d14d990"..., 3726) = 3726
close(3)                                = 0
access("hooks/reference-transaction", X_OK) = -1 ENOENT (No such file or directory)
munmap(0x7f7c2ef5b000, 3903118)         = 0
rename("/home/work/repositories/miui/All-Users.git/./packed-refs.new", "/home/work/repositories/miui/All-Users.git/./packed-refs"

`git pack-refs kernel stack`
[<0>] rpc_wait_bit_killable+0x24/0xa0 [sunrpc]
[<0>] __rpc_wait_for_completion_task+0x2d/0x30 [sunrpc]
[<0>] nfs_rename+0xc5/0x310 [nfs]
[<0>] vfs_rename+0x3dc/0xa80
[<0>] do_renameat2+0x4ca/0x590
[<0>] __x64_sys_rename+0x20/0x30
[<0>] do_syscall_64+0x57/0x190
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9


`ls` strace:
......
.....
mmap(NULL, 1683056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f63251b0000
close(3)                                = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TIOCGWINSZ, {ws_row=33, ws_col=113, ws_xpixel=0, ws_ypixel=0}) = 0
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents(3

`ls kernel stack`:
[<0>] rwsem_down_write_slowpath+0x23d/0x4c0
[<0>] iterate_dir+0x126/0x1b0
[<0>] __x64_sys_getdents+0xab/0x140
[<0>] do_syscall_64+0x57/0x190
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#

Re: nfs hang bugs

[Chuck Lever III]

Hi!

> On Nov 11, 2021, at 11:39 AM, 程洋 <chengyang@xiaomi.com> wrote:
> 
> Hi. I'm an server engineer who hold gerrit servers. And uses nfs as my harddisk.
> Some time I found when I run `git pack-refs`, it will hang, and it causes deadlock in kernel, prevent other fs command to execute like `ls`
> Pretty much like this, but not the same issue.  https://about.gitlab.com/blog/2018/11/14/how-we-spent-two-weeks-hunting-an-nfs-bug/
> 
> Since I'm new to NFS, I don't know what else can I provide. If there anyway to get more debug information, please feel free to ask me.
> 
> My NFS server: Ubuntu 16.04
> My NFS client: Ubuntu 18.04

Just to note that this is a developer mailing list, not
really for user issues. For faster response, it's best
to work directly with your Linux distributor.

Your description sounds like a network or server issue:
the client looks like it's legitimately waiting for a
response rather than stuck somewhere unexpected.

Some other information for folks here to help out:
- Mount options on the client
- Server configuration and unusual log messages
- If you have a reproducer, you could post it here


> `git pack-refs strace`
> write(3, "users/94/1003094\n886741817c1d824"..., 8192) = 8192
> write(3, "ers/95/1006795\n7ec95f6ad772f6d63"..., 8192) = 8192
> write(3, "s/96/1011196\n5e310bf8491276a6a8d"..., 8192) = 8192
> write(3, "98/1000998\n3e1f177f0427e7aa79065"..., 8192) = 8192
> write(3, "/1004899\n14e572b05b3bb857d14d990"..., 3726) = 3726
> close(3)                                = 0
> access("hooks/reference-transaction", X_OK) = -1 ENOENT (No such file or directory)
> munmap(0x7f7c2ef5b000, 3903118)         = 0
> rename("/home/work/repositories/miui/All-Users.git/./packed-refs.new", "/home/work/repositories/miui/All-Users.git/./packed-refs"
> 
> `git pack-refs kernel stack`
> [<0>] rpc_wait_bit_killable+0x24/0xa0 [sunrpc]
> [<0>] __rpc_wait_for_completion_task+0x2d/0x30 [sunrpc]
> [<0>] nfs_rename+0xc5/0x310 [nfs]
> [<0>] vfs_rename+0x3dc/0xa80
> [<0>] do_renameat2+0x4ca/0x590
> [<0>] __x64_sys_rename+0x20/0x30
> [<0>] do_syscall_64+0x57/0x190
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> 
> `ls` strace:
> ......
> .....
> mmap(NULL, 1683056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f63251b0000
> close(3)                                = 0
> ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
> ioctl(1, TIOCGWINSZ, {ws_row=33, ws_col=113, ws_xpixel=0, ws_ypixel=0}) = 0
> openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
> fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> getdents(3
> 
> `ls kernel stack`:
> [<0>] rwsem_down_write_slowpath+0x23d/0x4c0
> [<0>] iterate_dir+0x126/0x1b0
> [<0>] __x64_sys_getdents+0xab/0x140
> [<0>] do_syscall_64+0x57/0x190
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#

--
Chuck Lever