Question about nfs in infiniband environment

Question about nfs in infiniband environment

[Volker Lieder]

Hi list,

we have a setup with round about 15 centos 7.5 server.

All are connected via infiniband 56Gbit and installed with new mellanox driver.
One server (4 Core, 8 threads, 16GB) is nfs server for a disk shelf with round about 500TB data.

The server exports 4-6 mounts to each client.

Since we added 3 further nodes to the setup, we recieve following messages:

On nfs-server:
[Tue Aug 28 07:29:33 2018] rpc-srv/tcp: nfsd: sent only 224000 when sending 1048684 bytes - shutting down socket
[Tue Aug 28 07:30:13 2018] rpc-srv/tcp: nfsd: sent only 209004 when sending 1048684 bytes - shutting down socket
[Tue Aug 28 07:30:14 2018] rpc-srv/tcp: nfsd: sent only 204908 when sending 630392 bytes - shutting down socket
[Tue Aug 28 07:32:31 2018] rpc-srv/tcp: nfsd: got error -11 when sending 524396 bytes - shutting down socket
[Tue Aug 28 07:32:33 2018] rpc-srv/tcp: nfsd: got error -11 when sending 308 bytes - shutting down socket
[Tue Aug 28 07:32:35 2018] rpc-srv/tcp: nfsd: got error -11 when sending 172 bytes - shutting down socket
[Tue Aug 28 07:32:53 2018] rpc-srv/tcp: nfsd: got error -11 when sending 164 bytes - shutting down socket
[Tue Aug 28 07:38:52 2018] rpc-srv/tcp: nfsd: sent only 749452 when sending 1048684 bytes - shutting down socket
[Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 244 bytes - shutting down socket
[Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 1048684 bytes - shutting down socket

on nfs-clients:
[229903.273435] nfs: server 172.16.55.221 not responding, still trying
[229903.523455] nfs: server 172.16.55.221 OK
[229939.080276] nfs: server 172.16.55.221 OK
[236527.473064] perf: interrupt took too long (6226 > 6217), lowering kernel.perf_event_max_sample_rate to 32000
[248874.777322] RPC: Could not send backchannel reply error: -105
[249484.823793] RPC: Could not send backchannel reply error: -105
[250382.497448] RPC: Could not send backchannel reply error: -105
[250671.054112] RPC: Could not send backchannel reply error: -105
[251284.622707] RPC: Could not send backchannel reply error: -105

Also file requests or "df -h" ended sometimes in a stale nfs status whcih will be good after a minute.

I googled all messages and tried different things without success.
We are now going on to upgrade cpu power on nfs server. 

Do you also have any hints or points i can look for?

Best regards,
Volker

Re: Question about nfs in infiniband environment

[Volker Lieder]

Hi,

a short update from our site.

We resized CPU and RAM on the nfs server and the performance is good right now and the error messages are gone.

Is there a guide what hardware requirements a fast nfs server has?

Or an information, how many nfs prozesses are needed for x nfs clients?

Best regards,
Volker

> Am 28.08.2018 um 09:45 schrieb Volker Lieder <v.lieder@uvensys.de>:
> 
> Hi list,
> 
> we have a setup with round about 15 centos 7.5 server.
> 
> All are connected via infiniband 56Gbit and installed with new mellanox driver.
> One server (4 Core, 8 threads, 16GB) is nfs server for a disk shelf with round about 500TB data.
> 
> The server exports 4-6 mounts to each client.
> 
> Since we added 3 further nodes to the setup, we recieve following messages:
> 
> On nfs-server:
> [Tue Aug 28 07:29:33 2018] rpc-srv/tcp: nfsd: sent only 224000 when sending 1048684 bytes - shutting down socket
> [Tue Aug 28 07:30:13 2018] rpc-srv/tcp: nfsd: sent only 209004 when sending 1048684 bytes - shutting down socket
> [Tue Aug 28 07:30:14 2018] rpc-srv/tcp: nfsd: sent only 204908 when sending 630392 bytes - shutting down socket
> [Tue Aug 28 07:32:31 2018] rpc-srv/tcp: nfsd: got error -11 when sending 524396 bytes - shutting down socket
> [Tue Aug 28 07:32:33 2018] rpc-srv/tcp: nfsd: got error -11 when sending 308 bytes - shutting down socket
> [Tue Aug 28 07:32:35 2018] rpc-srv/tcp: nfsd: got error -11 when sending 172 bytes - shutting down socket
> [Tue Aug 28 07:32:53 2018] rpc-srv/tcp: nfsd: got error -11 when sending 164 bytes - shutting down socket
> [Tue Aug 28 07:38:52 2018] rpc-srv/tcp: nfsd: sent only 749452 when sending 1048684 bytes - shutting down socket
> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 244 bytes - shutting down socket
> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 1048684 bytes - shutting down socket
> 
> on nfs-clients:
> [229903.273435] nfs: server 172.16.55.221 not responding, still trying
> [229903.523455] nfs: server 172.16.55.221 OK
> [229939.080276] nfs: server 172.16.55.221 OK
> [236527.473064] perf: interrupt took too long (6226 > 6217), lowering kernel.perf_event_max_sample_rate to 32000
> [248874.777322] RPC: Could not send backchannel reply error: -105
> [249484.823793] RPC: Could not send backchannel reply error: -105
> [250382.497448] RPC: Could not send backchannel reply error: -105
> [250671.054112] RPC: Could not send backchannel reply error: -105
> [251284.622707] RPC: Could not send backchannel reply error: -105
> 
> Also file requests or "df -h" ended sometimes in a stale nfs status whcih will be good after a minute.
> 
> I googled all messages and tried different things without success.
> We are now going on to upgrade cpu power on nfs server. 
> 
> Do you also have any hints or points i can look for?
> 
> Best regards,
> Volker

Re: Question about nfs in infiniband environment

[Chuck Lever]

Hi Volker-


> On Aug 28, 2018, at 8:37 AM, Volker Lieder <v.lieder@uvensys.de> =
wrote:
>=20
> Hi,
>=20
> a short update from our site.
>=20
> We resized CPU and RAM on the nfs server and the performance is good =
right now and the error messages are gone.
>=20
> Is there a guide what hardware requirements a fast nfs server has?
>=20
> Or an information, how many nfs prozesses are needed for x nfs =
clients?

The nfsd thread count depends on number of clients _and_ their workload.
There isn't a hard and fast rule.

The default thread count is probably too low for your workload. You can
edit /etc/sysconfig/nfs and find "RPCNFSDCOUNT". Increase it to, say,
64, and restart your NFS server.

With InfiniBand you also have the option of using NFS/RDMA. Mount with
"proto=3Drdma,port=3D20049" to try it.


> Best regards,
> Volker
>=20
>> Am 28.08.2018 um 09:45 schrieb Volker Lieder <v.lieder@uvensys.de>:
>>=20
>> Hi list,
>>=20
>> we have a setup with round about 15 centos 7.5 server.
>>=20
>> All are connected via infiniband 56Gbit and installed with new =
mellanox driver.
>> One server (4 Core, 8 threads, 16GB) is nfs server for a disk shelf =
with round about 500TB data.
>>=20
>> The server exports 4-6 mounts to each client.
>>=20
>> Since we added 3 further nodes to the setup, we recieve following =
messages:
>>=20
>> On nfs-server:
>> [Tue Aug 28 07:29:33 2018] rpc-srv/tcp: nfsd: sent only 224000 when =
sending 1048684 bytes - shutting down socket
>> [Tue Aug 28 07:30:13 2018] rpc-srv/tcp: nfsd: sent only 209004 when =
sending 1048684 bytes - shutting down socket
>> [Tue Aug 28 07:30:14 2018] rpc-srv/tcp: nfsd: sent only 204908 when =
sending 630392 bytes - shutting down socket
>> [Tue Aug 28 07:32:31 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 524396 bytes - shutting down socket
>> [Tue Aug 28 07:32:33 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 308 bytes - shutting down socket
>> [Tue Aug 28 07:32:35 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 172 bytes - shutting down socket
>> [Tue Aug 28 07:32:53 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 164 bytes - shutting down socket
>> [Tue Aug 28 07:38:52 2018] rpc-srv/tcp: nfsd: sent only 749452 when =
sending 1048684 bytes - shutting down socket
>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 244 bytes - shutting down socket
>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 1048684 bytes - shutting down socket
>>=20
>> on nfs-clients:
>> [229903.273435] nfs: server 172.16.55.221 not responding, still =
trying
>> [229903.523455] nfs: server 172.16.55.221 OK
>> [229939.080276] nfs: server 172.16.55.221 OK
>> [236527.473064] perf: interrupt took too long (6226 > 6217), lowering =
kernel.perf_event_max_sample_rate to 32000
>> [248874.777322] RPC: Could not send backchannel reply error: -105
>> [249484.823793] RPC: Could not send backchannel reply error: -105
>> [250382.497448] RPC: Could not send backchannel reply error: -105
>> [250671.054112] RPC: Could not send backchannel reply error: -105
>> [251284.622707] RPC: Could not send backchannel reply error: -105
>>=20
>> Also file requests or "df -h" ended sometimes in a stale nfs status =
whcih will be good after a minute.
>>=20
>> I googled all messages and tried different things without success.
>> We are now going on to upgrade cpu power on nfs server.=20
>>=20
>> Do you also have any hints or points i can look for?
>>=20
>> Best regards,
>> Volker
>=20

--
Chuck Lever
chucklever@gmail.com

Re: Question about nfs in infiniband environment

[Volker Lieder]

Hi Chuck,

> Am 28.08.2018 um 17:26 schrieb Chuck Lever <chucklever@gmail.com>:
> 
> Hi Volker-
> 
> 
>> On Aug 28, 2018, at 8:37 AM, Volker Lieder <v.lieder@uvensys.de> wrote:
>> 
>> Hi,
>> 
>> a short update from our site.
>> 
>> We resized CPU and RAM on the nfs server and the performance is good right now and the error messages are gone.
>> 
>> Is there a guide what hardware requirements a fast nfs server has?
>> 
>> Or an information, how many nfs prozesses are needed for x nfs clients?
> 
> The nfsd thread count depends on number of clients _and_ their workload.
> There isn't a hard and fast rule.
> 
> The default thread count is probably too low for your workload. You can
> edit /etc/sysconfig/nfs and find "RPCNFSDCOUNT". Increase it to, say,
> 64, and restart your NFS server.

I tried this, but then the load on the "small" server was to high to serve further requests, so that was the idea to grow this up.

> 
> With InfiniBand you also have the option of using NFS/RDMA. Mount with
> "proto=rdma,port=20049" to try it.

Yes, thats true, but in the mellanox driver set they disabled nfsordma in Version 3.4.
It should work with centos driver, but we didnt tested it right now in newer setups.

One more question, since other problems seem to be solved:

What about this message?

[Tue Aug 28 15:10:44 2018] NFSD: client 172.16.YY.XXX testing state ID with incorrect client ID


> 
> 
>> Best regards,
>> Volker
>> 
>>> Am 28.08.2018 um 09:45 schrieb Volker Lieder <v.lieder@uvensys.de>:
>>> 
>>> Hi list,
>>> 
>>> we have a setup with round about 15 centos 7.5 server.
>>> 
>>> All are connected via infiniband 56Gbit and installed with new mellanox driver.
>>> One server (4 Core, 8 threads, 16GB) is nfs server for a disk shelf with round about 500TB data.
>>> 
>>> The server exports 4-6 mounts to each client.
>>> 
>>> Since we added 3 further nodes to the setup, we recieve following messages:
>>> 
>>> On nfs-server:
>>> [Tue Aug 28 07:29:33 2018] rpc-srv/tcp: nfsd: sent only 224000 when sending 1048684 bytes - shutting down socket
>>> [Tue Aug 28 07:30:13 2018] rpc-srv/tcp: nfsd: sent only 209004 when sending 1048684 bytes - shutting down socket
>>> [Tue Aug 28 07:30:14 2018] rpc-srv/tcp: nfsd: sent only 204908 when sending 630392 bytes - shutting down socket
>>> [Tue Aug 28 07:32:31 2018] rpc-srv/tcp: nfsd: got error -11 when sending 524396 bytes - shutting down socket
>>> [Tue Aug 28 07:32:33 2018] rpc-srv/tcp: nfsd: got error -11 when sending 308 bytes - shutting down socket
>>> [Tue Aug 28 07:32:35 2018] rpc-srv/tcp: nfsd: got error -11 when sending 172 bytes - shutting down socket
>>> [Tue Aug 28 07:32:53 2018] rpc-srv/tcp: nfsd: got error -11 when sending 164 bytes - shutting down socket
>>> [Tue Aug 28 07:38:52 2018] rpc-srv/tcp: nfsd: sent only 749452 when sending 1048684 bytes - shutting down socket
>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 244 bytes - shutting down socket
>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 1048684 bytes - shutting down socket
>>> 
>>> on nfs-clients:
>>> [229903.273435] nfs: server 172.16.55.221 not responding, still trying
>>> [229903.523455] nfs: server 172.16.55.221 OK
>>> [229939.080276] nfs: server 172.16.55.221 OK
>>> [236527.473064] perf: interrupt took too long (6226 > 6217), lowering kernel.perf_event_max_sample_rate to 32000
>>> [248874.777322] RPC: Could not send backchannel reply error: -105
>>> [249484.823793] RPC: Could not send backchannel reply error: -105
>>> [250382.497448] RPC: Could not send backchannel reply error: -105
>>> [250671.054112] RPC: Could not send backchannel reply error: -105
>>> [251284.622707] RPC: Could not send backchannel reply error: -105
>>> 
>>> Also file requests or "df -h" ended sometimes in a stale nfs status whcih will be good after a minute.
>>> 
>>> I googled all messages and tried different things without success.
>>> We are now going on to upgrade cpu power on nfs server. 
>>> 
>>> Do you also have any hints or points i can look for?
>>> 
>>> Best regards,
>>> Volker
>> 
> 
> --
> Chuck Lever
> chucklever@gmail.com

Re: Question about nfs in infiniband environment

[Chuck Lever]

> On Aug 28, 2018, at 11:31 AM, Volker Lieder <v.lieder@uvensys.de> =
wrote:
>=20
> Hi Chuck,
>=20
>> Am 28.08.2018 um 17:26 schrieb Chuck Lever <chucklever@gmail.com>:
>>=20
>> Hi Volker-
>>=20
>>=20
>>> On Aug 28, 2018, at 8:37 AM, Volker Lieder <v.lieder@uvensys.de> =
wrote:
>>>=20
>>> Hi,
>>>=20
>>> a short update from our site.
>>>=20
>>> We resized CPU and RAM on the nfs server and the performance is good =
right now and the error messages are gone.
>>>=20
>>> Is there a guide what hardware requirements a fast nfs server has?
>>>=20
>>> Or an information, how many nfs prozesses are needed for x nfs =
clients?
>>=20
>> The nfsd thread count depends on number of clients _and_ their =
workload.
>> There isn't a hard and fast rule.
>>=20
>> The default thread count is probably too low for your workload. You =
can
>> edit /etc/sysconfig/nfs and find "RPCNFSDCOUNT". Increase it to, say,
>> 64, and restart your NFS server.
>=20
> I tried this, but then the load on the "small" server was to high to =
serve further requests, so that was the idea to grow this up.

That rather suggests the disks are slow. A deeper performance
analysis might help.


>> With InfiniBand you also have the option of using NFS/RDMA. Mount =
with
>> "proto=3Drdma,port=3D20049" to try it.
>=20
> Yes, thats true, but in the mellanox driver set they disabled nfsordma =
in Version 3.4.

Not quite sure what you mean by "mellanox driver". Do you
mean MOFED? My impression of the stock CentOS 7.5 code is
that it is close to upstream, and you shouldn't need to
replace it except in some very special circumstances (high
end database, eg).


> It should work with centos driver, but we didnt tested it right now in =
newer setups.
>=20
> One more question, since other problems seem to be solved:
>=20
> What about this message?
>=20
> [Tue Aug 28 15:10:44 2018] NFSD: client 172.16.YY.XXX testing state ID =
with incorrect client ID

Looks like an NFS bug. Someone else on the list should be able
to comment.


>>> Best regards,
>>> Volker
>>>=20
>>>> Am 28.08.2018 um 09:45 schrieb Volker Lieder <v.lieder@uvensys.de>:
>>>>=20
>>>> Hi list,
>>>>=20
>>>> we have a setup with round about 15 centos 7.5 server.
>>>>=20
>>>> All are connected via infiniband 56Gbit and installed with new =
mellanox driver.
>>>> One server (4 Core, 8 threads, 16GB) is nfs server for a disk shelf =
with round about 500TB data.
>>>>=20
>>>> The server exports 4-6 mounts to each client.
>>>>=20
>>>> Since we added 3 further nodes to the setup, we recieve following =
messages:
>>>>=20
>>>> On nfs-server:
>>>> [Tue Aug 28 07:29:33 2018] rpc-srv/tcp: nfsd: sent only 224000 when =
sending 1048684 bytes - shutting down socket
>>>> [Tue Aug 28 07:30:13 2018] rpc-srv/tcp: nfsd: sent only 209004 when =
sending 1048684 bytes - shutting down socket
>>>> [Tue Aug 28 07:30:14 2018] rpc-srv/tcp: nfsd: sent only 204908 when =
sending 630392 bytes - shutting down socket
>>>> [Tue Aug 28 07:32:31 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 524396 bytes - shutting down socket
>>>> [Tue Aug 28 07:32:33 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 308 bytes - shutting down socket
>>>> [Tue Aug 28 07:32:35 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 172 bytes - shutting down socket
>>>> [Tue Aug 28 07:32:53 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 164 bytes - shutting down socket
>>>> [Tue Aug 28 07:38:52 2018] rpc-srv/tcp: nfsd: sent only 749452 when =
sending 1048684 bytes - shutting down socket
>>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 244 bytes - shutting down socket
>>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when =
sending 1048684 bytes - shutting down socket
>>>>=20
>>>> on nfs-clients:
>>>> [229903.273435] nfs: server 172.16.55.221 not responding, still =
trying
>>>> [229903.523455] nfs: server 172.16.55.221 OK
>>>> [229939.080276] nfs: server 172.16.55.221 OK
>>>> [236527.473064] perf: interrupt took too long (6226 > 6217), =
lowering kernel.perf_event_max_sample_rate to 32000
>>>> [248874.777322] RPC: Could not send backchannel reply error: -105
>>>> [249484.823793] RPC: Could not send backchannel reply error: -105
>>>> [250382.497448] RPC: Could not send backchannel reply error: -105
>>>> [250671.054112] RPC: Could not send backchannel reply error: -105
>>>> [251284.622707] RPC: Could not send backchannel reply error: -105
>>>>=20
>>>> Also file requests or "df -h" ended sometimes in a stale nfs status =
whcih will be good after a minute.
>>>>=20
>>>> I googled all messages and tried different things without success.
>>>> We are now going on to upgrade cpu power on nfs server.=20
>>>>=20
>>>> Do you also have any hints or points i can look for?
>>>>=20
>>>> Best regards,
>>>> Volker
>>>=20
>>=20
>> --
>> Chuck Lever
>> chucklever@gmail.com

--
Chuck Lever

Re: Question about nfs in infiniband environment

[Jeff Becker]

Hi. Comment about MOFED below.

On 08/28/2018 08:40 AM, Chuck Lever wrote:
>
>> On Aug 28, 2018, at 11:31 AM, Volker Lieder <v.lieder@uvensys.de> wrote:
>>
>> Hi Chuck,
>>
>>> Am 28.08.2018 um 17:26 schrieb Chuck Lever <chucklever@gmail.com>:
>>>
>>> Hi Volker-
>>>
>>>
>>>> On Aug 28, 2018, at 8:37 AM, Volker Lieder <v.lieder@uvensys.de> wrote:
>>>>
>>>> Hi,
>>>>
>>>> a short update from our site.
>>>>
>>>> We resized CPU and RAM on the nfs server and the performance is good right now and the error messages are gone.
>>>>
>>>> Is there a guide what hardware requirements a fast nfs server has?
>>>>
>>>> Or an information, how many nfs prozesses are needed for x nfs clients?
>>> The nfsd thread count depends on number of clients _and_ their workload.
>>> There isn't a hard and fast rule.
>>>
>>> The default thread count is probably too low for your workload. You can
>>> edit /etc/sysconfig/nfs and find "RPCNFSDCOUNT". Increase it to, say,
>>> 64, and restart your NFS server.
>> I tried this, but then the load on the "small" server was to high to serve further requests, so that was the idea to grow this up.
> That rather suggests the disks are slow. A deeper performance
> analysis might help.
>
>
>>> With InfiniBand you also have the option of using NFS/RDMA. Mount with
>>> "proto=rdma,port=20049" to try it.
>> Yes, thats true, but in the mellanox driver set they disabled nfsordma in Version 3.4.
> Not quite sure what you mean by "mellanox driver". Do you
> mean MOFED? My impression of the stock CentOS 7.5 code is
> that it is close to upstream, and you shouldn't need to
> replace it except in some very special circumstances (high
> end database, eg).

Volker is right. Mellanox disables NFSRDMA in MOFED, because they don't 
backport it like I do for OFED, i.e, Mellanox forked off MOFED from OFED 
long time ago. As Chuck says, there probably isn't too much advantage of 
the MOFED kernel bits over the distro ones. It's probably good to use 
the subnet manager from MOFED if you can, as that is proprietary (closed 
source), and has several improvements in it. Not sure about now, but a 
few years ago, we had to switch to MOFED on our large Pleiades cluster 
since opensm (the open source version) couldn't handle the scale. Today, 
our primary reason for running MOFED is that we have a support contract 
with Mellanox.

HTH

-jeff
>
>> It should work with centos driver, but we didnt tested it right now in newer setups.
>>
>> One more question, since other problems seem to be solved:
>>
>> What about this message?
>>
>> [Tue Aug 28 15:10:44 2018] NFSD: client 172.16.YY.XXX testing state ID with incorrect client ID
> Looks like an NFS bug. Someone else on the list should be able
> to comment.
>
>
>>>> Best regards,
>>>> Volker
>>>>
>>>>> Am 28.08.2018 um 09:45 schrieb Volker Lieder <v.lieder@uvensys.de>:
>>>>>
>>>>> Hi list,
>>>>>
>>>>> we have a setup with round about 15 centos 7.5 server.
>>>>>
>>>>> All are connected via infiniband 56Gbit and installed with new mellanox driver.
>>>>> One server (4 Core, 8 threads, 16GB) is nfs server for a disk shelf with round about 500TB data.
>>>>>
>>>>> The server exports 4-6 mounts to each client.
>>>>>
>>>>> Since we added 3 further nodes to the setup, we recieve following messages:
>>>>>
>>>>> On nfs-server:
>>>>> [Tue Aug 28 07:29:33 2018] rpc-srv/tcp: nfsd: sent only 224000 when sending 1048684 bytes - shutting down socket
>>>>> [Tue Aug 28 07:30:13 2018] rpc-srv/tcp: nfsd: sent only 209004 when sending 1048684 bytes - shutting down socket
>>>>> [Tue Aug 28 07:30:14 2018] rpc-srv/tcp: nfsd: sent only 204908 when sending 630392 bytes - shutting down socket
>>>>> [Tue Aug 28 07:32:31 2018] rpc-srv/tcp: nfsd: got error -11 when sending 524396 bytes - shutting down socket
>>>>> [Tue Aug 28 07:32:33 2018] rpc-srv/tcp: nfsd: got error -11 when sending 308 bytes - shutting down socket
>>>>> [Tue Aug 28 07:32:35 2018] rpc-srv/tcp: nfsd: got error -11 when sending 172 bytes - shutting down socket
>>>>> [Tue Aug 28 07:32:53 2018] rpc-srv/tcp: nfsd: got error -11 when sending 164 bytes - shutting down socket
>>>>> [Tue Aug 28 07:38:52 2018] rpc-srv/tcp: nfsd: sent only 749452 when sending 1048684 bytes - shutting down socket
>>>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 244 bytes - shutting down socket
>>>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 1048684 bytes - shutting down socket
>>>>>
>>>>> on nfs-clients:
>>>>> [229903.273435] nfs: server 172.16.55.221 not responding, still trying
>>>>> [229903.523455] nfs: server 172.16.55.221 OK
>>>>> [229939.080276] nfs: server 172.16.55.221 OK
>>>>> [236527.473064] perf: interrupt took too long (6226 > 6217), lowering kernel.perf_event_max_sample_rate to 32000
>>>>> [248874.777322] RPC: Could not send backchannel reply error: -105
>>>>> [249484.823793] RPC: Could not send backchannel reply error: -105
>>>>> [250382.497448] RPC: Could not send backchannel reply error: -105
>>>>> [250671.054112] RPC: Could not send backchannel reply error: -105
>>>>> [251284.622707] RPC: Could not send backchannel reply error: -105
>>>>>
>>>>> Also file requests or "df -h" ended sometimes in a stale nfs status whcih will be good after a minute.
>>>>>
>>>>> I googled all messages and tried different things without success.
>>>>> We are now going on to upgrade cpu power on nfs server.
>>>>>
>>>>> Do you also have any hints or points i can look for?
>>>>>
>>>>> Best regards,
>>>>> Volker
>>> --
>>> Chuck Lever
>>> chucklever@gmail.com
> --
> Chuck Lever
>
>
>

Re: Question about nfs in infiniband environment

[Olga Kornievskaia]

On Tue, Aug 28, 2018 at 11:41 AM Chuck Lever <chuck.lever@oracle.com> wrote:
>
>
>
> > On Aug 28, 2018, at 11:31 AM, Volker Lieder <v.lieder@uvensys.de> wrote:
> >
> > Hi Chuck,
> >
> >> Am 28.08.2018 um 17:26 schrieb Chuck Lever <chucklever@gmail.com>:
> >>
> >> Hi Volker-
> >>
> >>
> >>> On Aug 28, 2018, at 8:37 AM, Volker Lieder <v.lieder@uvensys.de> wrote:
> >>>
> >>> Hi,
> >>>
> >>> a short update from our site.
> >>>
> >>> We resized CPU and RAM on the nfs server and the performance is good right now and the error messages are gone.
> >>>
> >>> Is there a guide what hardware requirements a fast nfs server has?
> >>>
> >>> Or an information, how many nfs prozesses are needed for x nfs clients?
> >>
> >> The nfsd thread count depends on number of clients _and_ their workload.
> >> There isn't a hard and fast rule.
> >>
> >> The default thread count is probably too low for your workload. You can
> >> edit /etc/sysconfig/nfs and find "RPCNFSDCOUNT". Increase it to, say,
> >> 64, and restart your NFS server.
> >
> > I tried this, but then the load on the "small" server was to high to serve further requests, so that was the idea to grow this up.
>
> That rather suggests the disks are slow. A deeper performance
> analysis might help.
>
>
> >> With InfiniBand you also have the option of using NFS/RDMA. Mount with
> >> "proto=rdma,port=20049" to try it.
> >
> > Yes, thats true, but in the mellanox driver set they disabled nfsordma in Version 3.4.
>
> Not quite sure what you mean by "mellanox driver". Do you
> mean MOFED? My impression of the stock CentOS 7.5 code is
> that it is close to upstream, and you shouldn't need to
> replace it except in some very special circumstances (high
> end database, eg).
>
>
> > It should work with centos driver, but we didnt tested it right now in newer setups.
> >
> > One more question, since other problems seem to be solved:
> >
> > What about this message?
> >
> > [Tue Aug 28 15:10:44 2018] NFSD: client 172.16.YY.XXX testing state ID with incorrect client ID
>
> Looks like an NFS bug. Someone else on the list should be able
> to comment.

I ran into this problem while testing RHEL7.5 NFSoRDMA (over
SoftRoCE). Here's a bugzilla
https://bugzilla.redhat.com/show_bug.cgi?id=1518006

I was having a hard time reproducing it consistently to debug it.
Because it was really a non-error error (and it wasn't upstream), it
went on a back burner.

>
>
> >>> Best regards,
> >>> Volker
> >>>
> >>>> Am 28.08.2018 um 09:45 schrieb Volker Lieder <v.lieder@uvensys.de>:
> >>>>
> >>>> Hi list,
> >>>>
> >>>> we have a setup with round about 15 centos 7.5 server.
> >>>>
> >>>> All are connected via infiniband 56Gbit and installed with new mellanox driver.
> >>>> One server (4 Core, 8 threads, 16GB) is nfs server for a disk shelf with round about 500TB data.
> >>>>
> >>>> The server exports 4-6 mounts to each client.
> >>>>
> >>>> Since we added 3 further nodes to the setup, we recieve following messages:
> >>>>
> >>>> On nfs-server:
> >>>> [Tue Aug 28 07:29:33 2018] rpc-srv/tcp: nfsd: sent only 224000 when sending 1048684 bytes - shutting down socket
> >>>> [Tue Aug 28 07:30:13 2018] rpc-srv/tcp: nfsd: sent only 209004 when sending 1048684 bytes - shutting down socket
> >>>> [Tue Aug 28 07:30:14 2018] rpc-srv/tcp: nfsd: sent only 204908 when sending 630392 bytes - shutting down socket
> >>>> [Tue Aug 28 07:32:31 2018] rpc-srv/tcp: nfsd: got error -11 when sending 524396 bytes - shutting down socket
> >>>> [Tue Aug 28 07:32:33 2018] rpc-srv/tcp: nfsd: got error -11 when sending 308 bytes - shutting down socket
> >>>> [Tue Aug 28 07:32:35 2018] rpc-srv/tcp: nfsd: got error -11 when sending 172 bytes - shutting down socket
> >>>> [Tue Aug 28 07:32:53 2018] rpc-srv/tcp: nfsd: got error -11 when sending 164 bytes - shutting down socket
> >>>> [Tue Aug 28 07:38:52 2018] rpc-srv/tcp: nfsd: sent only 749452 when sending 1048684 bytes - shutting down socket
> >>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 244 bytes - shutting down socket
> >>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 1048684 bytes - shutting down socket
> >>>>
> >>>> on nfs-clients:
> >>>> [229903.273435] nfs: server 172.16.55.221 not responding, still trying
> >>>> [229903.523455] nfs: server 172.16.55.221 OK
> >>>> [229939.080276] nfs: server 172.16.55.221 OK
> >>>> [236527.473064] perf: interrupt took too long (6226 > 6217), lowering kernel.perf_event_max_sample_rate to 32000
> >>>> [248874.777322] RPC: Could not send backchannel reply error: -105
> >>>> [249484.823793] RPC: Could not send backchannel reply error: -105
> >>>> [250382.497448] RPC: Could not send backchannel reply error: -105
> >>>> [250671.054112] RPC: Could not send backchannel reply error: -105
> >>>> [251284.622707] RPC: Could not send backchannel reply error: -105
> >>>>
> >>>> Also file requests or "df -h" ended sometimes in a stale nfs status whcih will be good after a minute.
> >>>>
> >>>> I googled all messages and tried different things without success.
> >>>> We are now going on to upgrade cpu power on nfs server.
> >>>>
> >>>> Do you also have any hints or points i can look for?
> >>>>
> >>>> Best regards,
> >>>> Volker
> >>>
> >>
> >> --
> >> Chuck Lever
> >> chucklever@gmail.com
>
> --
> Chuck Lever
>
>
>

Re: Question about nfs in infiniband environment

[Volker Lieder]

Hi Olga,

i dont have a redhat account.

Can you, if helpful, paste the result right here?

Regards
Volker

> Am 28.08.2018 um 21:10 schrieb Olga Kornievskaia <aglo@umich.edu>:
> 
> On Tue, Aug 28, 2018 at 11:41 AM Chuck Lever <chuck.lever@oracle.com> wrote:
>> 
>> 
>> 
>>> On Aug 28, 2018, at 11:31 AM, Volker Lieder <v.lieder@uvensys.de> wrote:
>>> 
>>> Hi Chuck,
>>> 
>>>> Am 28.08.2018 um 17:26 schrieb Chuck Lever <chucklever@gmail.com>:
>>>> 
>>>> Hi Volker-
>>>> 
>>>> 
>>>>> On Aug 28, 2018, at 8:37 AM, Volker Lieder <v.lieder@uvensys.de> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> a short update from our site.
>>>>> 
>>>>> We resized CPU and RAM on the nfs server and the performance is good right now and the error messages are gone.
>>>>> 
>>>>> Is there a guide what hardware requirements a fast nfs server has?
>>>>> 
>>>>> Or an information, how many nfs prozesses are needed for x nfs clients?
>>>> 
>>>> The nfsd thread count depends on number of clients _and_ their workload.
>>>> There isn't a hard and fast rule.
>>>> 
>>>> The default thread count is probably too low for your workload. You can
>>>> edit /etc/sysconfig/nfs and find "RPCNFSDCOUNT". Increase it to, say,
>>>> 64, and restart your NFS server.
>>> 
>>> I tried this, but then the load on the "small" server was to high to serve further requests, so that was the idea to grow this up.
>> 
>> That rather suggests the disks are slow. A deeper performance
>> analysis might help.
>> 
>> 
>>>> With InfiniBand you also have the option of using NFS/RDMA. Mount with
>>>> "proto=rdma,port=20049" to try it.
>>> 
>>> Yes, thats true, but in the mellanox driver set they disabled nfsordma in Version 3.4.
>> 
>> Not quite sure what you mean by "mellanox driver". Do you
>> mean MOFED? My impression of the stock CentOS 7.5 code is
>> that it is close to upstream, and you shouldn't need to
>> replace it except in some very special circumstances (high
>> end database, eg).
>> 
>> 
>>> It should work with centos driver, but we didnt tested it right now in newer setups.
>>> 
>>> One more question, since other problems seem to be solved:
>>> 
>>> What about this message?
>>> 
>>> [Tue Aug 28 15:10:44 2018] NFSD: client 172.16.YY.XXX testing state ID with incorrect client ID
>> 
>> Looks like an NFS bug. Someone else on the list should be able
>> to comment.
> 
> I ran into this problem while testing RHEL7.5 NFSoRDMA (over
> SoftRoCE). Here's a bugzilla
> https://bugzilla.redhat.com/show_bug.cgi?id=1518006
> 
> I was having a hard time reproducing it consistently to debug it.
> Because it was really a non-error error (and it wasn't upstream), it
> went on a back burner.
> 
>> 
>> 
>>>>> Best regards,
>>>>> Volker
>>>>> 
>>>>>> Am 28.08.2018 um 09:45 schrieb Volker Lieder <v.lieder@uvensys.de>:
>>>>>> 
>>>>>> Hi list,
>>>>>> 
>>>>>> we have a setup with round about 15 centos 7.5 server.
>>>>>> 
>>>>>> All are connected via infiniband 56Gbit and installed with new mellanox driver.
>>>>>> One server (4 Core, 8 threads, 16GB) is nfs server for a disk shelf with round about 500TB data.
>>>>>> 
>>>>>> The server exports 4-6 mounts to each client.
>>>>>> 
>>>>>> Since we added 3 further nodes to the setup, we recieve following messages:
>>>>>> 
>>>>>> On nfs-server:
>>>>>> [Tue Aug 28 07:29:33 2018] rpc-srv/tcp: nfsd: sent only 224000 when sending 1048684 bytes - shutting down socket
>>>>>> [Tue Aug 28 07:30:13 2018] rpc-srv/tcp: nfsd: sent only 209004 when sending 1048684 bytes - shutting down socket
>>>>>> [Tue Aug 28 07:30:14 2018] rpc-srv/tcp: nfsd: sent only 204908 when sending 630392 bytes - shutting down socket
>>>>>> [Tue Aug 28 07:32:31 2018] rpc-srv/tcp: nfsd: got error -11 when sending 524396 bytes - shutting down socket
>>>>>> [Tue Aug 28 07:32:33 2018] rpc-srv/tcp: nfsd: got error -11 when sending 308 bytes - shutting down socket
>>>>>> [Tue Aug 28 07:32:35 2018] rpc-srv/tcp: nfsd: got error -11 when sending 172 bytes - shutting down socket
>>>>>> [Tue Aug 28 07:32:53 2018] rpc-srv/tcp: nfsd: got error -11 when sending 164 bytes - shutting down socket
>>>>>> [Tue Aug 28 07:38:52 2018] rpc-srv/tcp: nfsd: sent only 749452 when sending 1048684 bytes - shutting down socket
>>>>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 244 bytes - shutting down socket
>>>>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 1048684 bytes - shutting down socket
>>>>>> 
>>>>>> on nfs-clients:
>>>>>> [229903.273435] nfs: server 172.16.55.221 not responding, still trying
>>>>>> [229903.523455] nfs: server 172.16.55.221 OK
>>>>>> [229939.080276] nfs: server 172.16.55.221 OK
>>>>>> [236527.473064] perf: interrupt took too long (6226 > 6217), lowering kernel.perf_event_max_sample_rate to 32000
>>>>>> [248874.777322] RPC: Could not send backchannel reply error: -105
>>>>>> [249484.823793] RPC: Could not send backchannel reply error: -105
>>>>>> [250382.497448] RPC: Could not send backchannel reply error: -105
>>>>>> [250671.054112] RPC: Could not send backchannel reply error: -105
>>>>>> [251284.622707] RPC: Could not send backchannel reply error: -105
>>>>>> 
>>>>>> Also file requests or "df -h" ended sometimes in a stale nfs status whcih will be good after a minute.
>>>>>> 
>>>>>> I googled all messages and tried different things without success.
>>>>>> We are now going on to upgrade cpu power on nfs server.
>>>>>> 
>>>>>> Do you also have any hints or points i can look for?
>>>>>> 
>>>>>> Best regards,
>>>>>> Volker
>>>>> 
>>>> 
>>>> --
>>>> Chuck Lever
>>>> chucklever@gmail.com
>> 
>> --
>> Chuck Lever
>> 
>> 
>> 

Re: Question about nfs in infiniband environment

[Olga Kornievskaia]

Hi Volker,

The issue was trigger from the following set of events. NFSoRDMA
connection over softRoCE was experiencing problems. There was a gap of
3mins between the last operation (OPEN) in the network trace and
(because RDMA connection was re-established) the BIND_CONN_TO_SESSION
which got the BAD_SESSION error. Client's lease has expired in 3mins
so that explains the BAD_SESSION error.

Then after the client notices that the session is bad and recovers
clientid and session and starts state recovery. When client sends the
recovery of OPEN the server sends NO_GRACE so client switches from
reboot recovery to no-grace recovery which includes testing the
stateid before sending the open. That's why we are sending the old
stateid (from the old client id) and gets that error and logs that
message in var log messages.

A network trace would be needed from your environment to tell if this
is a similar situation.
On Wed, Aug 29, 2018 at 5:03 AM Volker Lieder <v.lieder@uvensys.de> wrote:
>
> Hi Olga,
>
> i dont have a redhat account.
>
> Can you, if helpful, paste the result right here?
>
> Regards
> Volker
>
> > Am 28.08.2018 um 21:10 schrieb Olga Kornievskaia <aglo@umich.edu>:
> >
> > On Tue, Aug 28, 2018 at 11:41 AM Chuck Lever <chuck.lever@oracle.com> wrote:
> >>
> >>
> >>
> >>> On Aug 28, 2018, at 11:31 AM, Volker Lieder <v.lieder@uvensys.de> wrote:
> >>>
> >>> Hi Chuck,
> >>>
> >>>> Am 28.08.2018 um 17:26 schrieb Chuck Lever <chucklever@gmail.com>:
> >>>>
> >>>> Hi Volker-
> >>>>
> >>>>
> >>>>> On Aug 28, 2018, at 8:37 AM, Volker Lieder <v.lieder@uvensys.de> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> a short update from our site.
> >>>>>
> >>>>> We resized CPU and RAM on the nfs server and the performance is good right now and the error messages are gone.
> >>>>>
> >>>>> Is there a guide what hardware requirements a fast nfs server has?
> >>>>>
> >>>>> Or an information, how many nfs prozesses are needed for x nfs clients?
> >>>>
> >>>> The nfsd thread count depends on number of clients _and_ their workload.
> >>>> There isn't a hard and fast rule.
> >>>>
> >>>> The default thread count is probably too low for your workload. You can
> >>>> edit /etc/sysconfig/nfs and find "RPCNFSDCOUNT". Increase it to, say,
> >>>> 64, and restart your NFS server.
> >>>
> >>> I tried this, but then the load on the "small" server was to high to serve further requests, so that was the idea to grow this up.
> >>
> >> That rather suggests the disks are slow. A deeper performance
> >> analysis might help.
> >>
> >>
> >>>> With InfiniBand you also have the option of using NFS/RDMA. Mount with
> >>>> "proto=rdma,port=20049" to try it.
> >>>
> >>> Yes, thats true, but in the mellanox driver set they disabled nfsordma in Version 3.4.
> >>
> >> Not quite sure what you mean by "mellanox driver". Do you
> >> mean MOFED? My impression of the stock CentOS 7.5 code is
> >> that it is close to upstream, and you shouldn't need to
> >> replace it except in some very special circumstances (high
> >> end database, eg).
> >>
> >>
> >>> It should work with centos driver, but we didnt tested it right now in newer setups.
> >>>
> >>> One more question, since other problems seem to be solved:
> >>>
> >>> What about this message?
> >>>
> >>> [Tue Aug 28 15:10:44 2018] NFSD: client 172.16.YY.XXX testing state ID with incorrect client ID
> >>
> >> Looks like an NFS bug. Someone else on the list should be able
> >> to comment.
> >
> > I ran into this problem while testing RHEL7.5 NFSoRDMA (over
> > SoftRoCE). Here's a bugzilla
> > https://bugzilla.redhat.com/show_bug.cgi?id=1518006
> >
> > I was having a hard time reproducing it consistently to debug it.
> > Because it was really a non-error error (and it wasn't upstream), it
> > went on a back burner.
> >
> >>
> >>
> >>>>> Best regards,
> >>>>> Volker
> >>>>>
> >>>>>> Am 28.08.2018 um 09:45 schrieb Volker Lieder <v.lieder@uvensys.de>:
> >>>>>>
> >>>>>> Hi list,
> >>>>>>
> >>>>>> we have a setup with round about 15 centos 7.5 server.
> >>>>>>
> >>>>>> All are connected via infiniband 56Gbit and installed with new mellanox driver.
> >>>>>> One server (4 Core, 8 threads, 16GB) is nfs server for a disk shelf with round about 500TB data.
> >>>>>>
> >>>>>> The server exports 4-6 mounts to each client.
> >>>>>>
> >>>>>> Since we added 3 further nodes to the setup, we recieve following messages:
> >>>>>>
> >>>>>> On nfs-server:
> >>>>>> [Tue Aug 28 07:29:33 2018] rpc-srv/tcp: nfsd: sent only 224000 when sending 1048684 bytes - shutting down socket
> >>>>>> [Tue Aug 28 07:30:13 2018] rpc-srv/tcp: nfsd: sent only 209004 when sending 1048684 bytes - shutting down socket
> >>>>>> [Tue Aug 28 07:30:14 2018] rpc-srv/tcp: nfsd: sent only 204908 when sending 630392 bytes - shutting down socket
> >>>>>> [Tue Aug 28 07:32:31 2018] rpc-srv/tcp: nfsd: got error -11 when sending 524396 bytes - shutting down socket
> >>>>>> [Tue Aug 28 07:32:33 2018] rpc-srv/tcp: nfsd: got error -11 when sending 308 bytes - shutting down socket
> >>>>>> [Tue Aug 28 07:32:35 2018] rpc-srv/tcp: nfsd: got error -11 when sending 172 bytes - shutting down socket
> >>>>>> [Tue Aug 28 07:32:53 2018] rpc-srv/tcp: nfsd: got error -11 when sending 164 bytes - shutting down socket
> >>>>>> [Tue Aug 28 07:38:52 2018] rpc-srv/tcp: nfsd: sent only 749452 when sending 1048684 bytes - shutting down socket
> >>>>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 244 bytes - shutting down socket
> >>>>>> [Tue Aug 28 07:39:29 2018] rpc-srv/tcp: nfsd: got error -11 when sending 1048684 bytes - shutting down socket
> >>>>>>
> >>>>>> on nfs-clients:
> >>>>>> [229903.273435] nfs: server 172.16.55.221 not responding, still trying
> >>>>>> [229903.523455] nfs: server 172.16.55.221 OK
> >>>>>> [229939.080276] nfs: server 172.16.55.221 OK
> >>>>>> [236527.473064] perf: interrupt took too long (6226 > 6217), lowering kernel.perf_event_max_sample_rate to 32000
> >>>>>> [248874.777322] RPC: Could not send backchannel reply error: -105
> >>>>>> [249484.823793] RPC: Could not send backchannel reply error: -105
> >>>>>> [250382.497448] RPC: Could not send backchannel reply error: -105
> >>>>>> [250671.054112] RPC: Could not send backchannel reply error: -105
> >>>>>> [251284.622707] RPC: Could not send backchannel reply error: -105
> >>>>>>
> >>>>>> Also file requests or "df -h" ended sometimes in a stale nfs status whcih will be good after a minute.
> >>>>>>
> >>>>>> I googled all messages and tried different things without success.
> >>>>>> We are now going on to upgrade cpu power on nfs server.
> >>>>>>
> >>>>>> Do you also have any hints or points i can look for?
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Volker
> >>>>>
> >>>>
> >>>> --
> >>>> Chuck Lever
> >>>> chucklever@gmail.com
> >>
> >> --
> >> Chuck Lever
> >>
> >>
> >>
>

Re: Question about nfs in infiniband environment

[J. Bruce Fields]

On Wed, Aug 29, 2018 at 11:03:22AM +0200, Volker Lieder wrote:
> i dont have a redhat account.

You should be able to see that bug now:

	https://bugzilla.redhat.com/show_bug.cgi?id=1518006

--b.

Re: Question about nfs in infiniband environment

[Volker Lieder]

Am 05.09.2018 um 23:26 schrieb J. Bruce Fields <bfields@fieldses.org>:
> 
> On Wed, Aug 29, 2018 at 11:03:22AM +0200, Volker Lieder wrote:
>> i dont have a redhat account.
> 
> You should be able to see that bug now:
> 
> 	https://bugzilla.redhat.com/show_bug.cgi?id=1518006
> 
> --b.

Hi Bruce,

thank you.

Best regards,
Volker