Re: [PATCH 2/2] nfsd: return ENOSPC if unable to allocate a session slot

[Manjunath Patil <manjunath.b.patil@oracle.com>]

Hi Bruce,

ct also had a test setup with Linux nfs server which had 512M RAM and 
they were cloning a VMs for clients. These client had a fstab entry for 
nfs mount. So each start up of the client would mount to NAS. They clone 
and start the VM. They observed that 10th VM clone hung during startup.

This setup was just to see how many client can be used. Note that ct NAS 
didn't have de766e570413 and 44d8660d3bb0. Having these might have 
pushed the no.of clients even further.

I will get back to you on their actual use-case.

-Thanks,
Manjunath
On 6/25/2018 3:04 PM, J. Bruce Fields wrote:
> On Mon, Jun 25, 2018 at 10:17:21AM -0700, Manjunath Patil wrote:
>> Hi Bruce,
>>
>> I could reproduce this issue by lowering the amount of RAM. On my
>> virtual box VM with 176M MB of RAM I can reproduce this with 3
>> clients.
> I know how to reproduce it, I was just wondering what motivated it--were
> customers hitting it (how), was it just artificial testing?
>
> Oh well, it probably needs to be fixed regardless.
>
> --b.
>
>> My kernel didn't have the following fixes -
>>
>>     de766e5 nfsd: give out fewer session slots as limit approaches
>>     44d8660 nfsd: increase DRC cache limit
>>
>> Once I apply these patches, the issue recurs with 10+ clients.
>> Once the mount starts to hang due to this issue, a NFSv4.0 still succeeds.
>>
>> I took the latest mainline kernel [4.18.0-rc1] and made the server
>> return NFS4ERR_DELAY[nfserr_jukebox] if its unable to allocate 50
>> slots[just to accelerate the issue]
>>
>>     -       if (!ca->maxreqs)
>>     +       if (ca->maxreqs < 50) {
>>                ...
>>                     return nfserr_jukebox;
>>
>> Then used the same client[4.18.0-rc1] and observed that mount calls
>> still hangs[indefinitely].
>> Typically the client hangs here - [stack are from oracle kernel] -
>>
>>     [root@OL7U5-work ~]# ps -ef | grep mount
>>     root      2032  1732  0 09:49 pts/0    00:00:00 strace -tttvf -o
>>     /tmp/a.out mount 10.211.47.123:/exports /NFSMNT -vvv -o retry=1
>>     root      2034  2032  0 09:49 pts/0    00:00:00 mount
>>     10.211.47.123:/exports /NFSMNT -vvv -o retry=1
>>     root      2035  2034  0 09:49 pts/0    00:00:00 /sbin/mount.nfs
>>     10.211.47.123:/exports /NFSMNT -v -o rw,retry=1
>>     root      2039  1905  0 09:49 pts/1    00:00:00 grep --color=auto mount
>>     [root@OL7U5-work ~]# cat /proc/2035/stack
>>     [<ffffffffa05204d2>] nfs_wait_client_init_complete+0x52/0xc0 [nfs]
>>     [<ffffffffa05872ed>] nfs41_discover_server_trunking+0x6d/0xb0 [nfsv4]
>>     [<ffffffffa0587802>] nfs4_discover_server_trunking+0x82/0x2e0 [nfsv4]
>>     [<ffffffffa058f8d6>] nfs4_init_client+0x136/0x300 [nfsv4]
>>     [<ffffffffa05210bf>] nfs_get_client+0x24f/0x2f0 [nfs]
>>     [<ffffffffa058eeef>] nfs4_set_client+0x9f/0xf0 [nfsv4]
>>     [<ffffffffa059039e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
>>     [<ffffffffa05881b2>] nfs4_remote_mount+0x32/0x60 [nfsv4]
>>     [<ffffffff8121df3e>] mount_fs+0x3e/0x180
>>     [<ffffffff8123a6db>] vfs_kern_mount+0x6b/0x110
>>     [<ffffffffa05880d6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
>>     [<ffffffffa05884c4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
>>     [<ffffffffa052ed6b>] nfs_fs_mount+0x4cb/0xda0 [nfs]
>>     [<ffffffff8121df3e>] mount_fs+0x3e/0x180
>>     [<ffffffff8123a6db>] vfs_kern_mount+0x6b/0x110
>>     [<ffffffff8123d5c1>] do_mount+0x251/0xcf0
>>     [<ffffffff8123e3a2>] SyS_mount+0xa2/0x110
>>     [<ffffffff81751f4b>] tracesys_phase2+0x6d/0x72
>>     [<ffffffffffffffff>] 0xffffffffffffffff
>>
>>     [root@OL7U5-work ~]# cat /proc/2034/stack
>>     [<ffffffff8108c147>] do_wait+0x217/0x2a0
>>     [<ffffffff8108d360>] do_wait4+0x80/0x110
>>     [<ffffffff8108d40d>] SyS_wait4+0x1d/0x20
>>     [<ffffffff81751f4b>] tracesys_phase2+0x6d/0x72
>>     [<ffffffffffffffff>] 0xffffffffffffffff
>>
>>     [root@OL7U5-work ~]# cat /proc/2032/stack
>>     [<ffffffff8108c147>] do_wait+0x217/0x2a0
>>     [<ffffffff8108d360>] do_wait4+0x80/0x110
>>     [<ffffffff8108d40d>] SyS_wait4+0x1d/0x20
>>     [<ffffffff81751ddc>] system_call_fastpath+0x18/0xd6
>>     [<ffffffffffffffff>] 0xffffffffffffffff
>>
>> -Thanks,
>> Manjunath
>> On 6/24/2018 1:26 PM, J. Bruce Fields wrote:
>>> By the way, could you share some more details with us about the
>>> situation when you (or your customers) are actually hitting this case?
>>>
>>> How many clients, what kind of clients, etc.  And what version of the
>>> server were you seeing the problem on?  (I'm mainly curious whether
>>> de766e570413 and 44d8660d3bb0 were already applied.)
>>>
>>> I'm glad we're thinking about how to handle this case, but my feeling is
>>> that the server is probably just being *much* too conservative about
>>> these allocations, and the most important thing may be to fix that and
>>> make it a lot rarer that we hit this case in the first place.
>>>
>>> --b.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html