Number of nfsd processes?

Number of nfsd processes?

[Simon Matthews]

I am running servers based on recent 2.6.x kernels.

I see "server timeout" messages from client machines running 2.4.x
kernels (typically RedHat kernels). I have tried increasing the number
of nfsd processes, but I seem to have got it up to absurd numbers (64
processes).

I read somewhere that each client only makes one connection to the
server, so one should not need more nfsd processes than clients, but
when I look at the processes duing some heavy nfs access, I see more
than one nfsd process taking significant  cpu time.

So, in a small network of say about 10 fast clients (with GigE
connections), how many  is a reasonable number of nfsd processes?
Should I see I/O errors if there are not enough nfsd processes?

Regards,
Simon

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Number of nfsd processes?

[Neil Brown]

On Saturday June 24, simon.d.matthews+nfs@gmail.com wrote:
> I am running servers based on recent 2.6.x kernels.
> 
> I see "server timeout" messages from client machines running 2.4.x
> kernels (typically RedHat kernels). I have tried increasing the number
> of nfsd processes, but I seem to have got it up to absurd numbers (64
> processes).
> 
> I read somewhere that each client only makes one connection to the
> server, so one should not need more nfsd processes than clients, but
> when I look at the processes duing some heavy nfs access, I see more
> than one nfsd process taking significant  cpu time.

There may be only one connection, but lots of requests may be sent
without waiting for the reply to the previous one, so the server may
be handling multiple requests at once for the one client.

> 
> So, in a small network of say about 10 fast clients (with GigE
> connections), how many  is a reasonable number of nfsd processes?

Lots? 
Look in /proc/net/rpc/nfsd
Look at the line starting 'th'.
 The first number is the number of threads
 The second is the number of times the a thread has started handling
  a request when all other threads were busy
 The remainder are the number of seconds when
    0-10% of threads were in use
   10-20% of threads were in user
   20-30%
    ...
   90-100%

 From this you should be able to get an idea of how many threads
 are getting used.
 Ideally, the top few numbers should be very small compared with the
 earlier numbers.  If they are, increase the number of threads.
 I usually run with 128, but there are probably 100-200 clients, but
 they aren't very busy clients.

> Should I see I/O errors if there are not enough nfsd processes?

Only if you are using 'soft' mounts.  Don't ever use soft mounts.
(Well, they might be OK in some read-only situations....)
You could see lots of 
   nfs: server foo not responding, still trying
   ...
   nfs: server foo OK
messages if you don't have enough threads.

NeilBrown


> 
> Regards,
> Simon
> 
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Number of nfsd processes?

[Ian Kent]

On Sat, 24 Jun 2006, Simon Matthews wrote:

> I am running servers based on recent 2.6.x kernels.
> 
> I see "server timeout" messages from client machines running 2.4.x
> kernels (typically RedHat kernels). I have tried increasing the number
> of nfsd processes, but I seem to have got it up to absurd numbers (64
> processes).

Absurd, don't think so.
I'd say for many "normal" NFS server environments this is likely a good 
starting point, certainly not the high end.

Last place I worked we started by setting the number of threads on NFS 
servers (Solaris) to 128 or 256 depending on expected work load, 128 for 
the lightly loaded servers. 

Ian

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Number of nfsd processes?

[Brian J. Murrell]

[-- Attachment #1.1: Type: text/plain, Size: 1530 bytes --]

Neil,

Your description was a bit ungrokkable.  Let me try to put some real
world values to it and try to make it out...

On Sun, 2006-06-25 at 19:42 +1000, Neil Brown wrote:
> Look at the line starting 'th'.

th 32 1439 108.420 17.016 6.044 3.724 2.464 0.296 0.256 0.280  0.672

>  The first number is the number of threads

Yup, 32 of them.

>  The second is the number of times the a thread has started handling
>   a request when all other threads were busy

1439 in my case.  I guess ideally you want this number quite low.  How
low is low enough?

>  The remainder are the number of seconds when
>     0-10% of threads were in use

108.420 seconds in my example.  By and far the largest value with the
next one being only 17.016 seconds.

>    10-20% of threads were in user

17.016 seconds

>    20-30%

6.044

>     ...
>    90-100%

0.672

Quite low indeed, yes?

> 
>  From this you should be able to get an idea of how many threads
>  are getting used.
>  Ideally, the top few numbers

What do you mean by the "top few numbers"

>  should be very small compared with the
>  earlier numbers.

What do you mean by the "earlier numbers"?

>   If they are, increase the number of threads.

So, should I increase threads?

>  I usually run with 128, but there are probably 100-200 clients, but
>  they aren't very busy clients.

Any WAGs (or better) on the "cost" of a thread?

b.

-- 
My other computer is your Microsoft Windows server.

Brian J. Murrell

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

[-- Attachment #2: Type: text/plain, Size: 299 bytes --]

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

[-- Attachment #3: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Number of nfsd processes?

[Simon Matthews]

Brian (and others):

On 6/25/06, Brian J. Murrell <brian@interlinx.bc.ca> wrote:
> Neil,
>
> Your description was a bit ungrokkable.  Let me try to put some real
> world values to it and try to make it out...

Here are my numbers:
more /proc/net/rpc/nfsd
rc 58145 148831950 678877510
fh 126136 0 0 0 0
io 3843561989 2872416345
th 64 750787 29187.339 10107.193 6025.710 3273.713 3225.384 1680.524
725.219 1305.846 780.037 3180.079
....

Do these numbers ever get re-set? For example, I have re-started the
nfs server process (to increas the number of nfsd processes), but I
suspec the numbers merely keep aggregating, so they don't show what is
happening now (with 64 processes).

Regards,
Simon

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Number of nfsd processes?

[Neil Brown]

On Sunday June 25, brian@interlinx.bc.ca wrote:
> Neil,
> 
> Your description was a bit ungrokkable.  Let me try to put some real
> world values to it and try to make it out...

You seem to do OK.

> 
> On Sun, 2006-06-25 at 19:42 +1000, Neil Brown wrote:
> > Look at the line starting 'th'.
> 
> th 32 1439 108.420 17.016 6.044 3.724 2.464 0.296 0.256 0.280  0.672
> 
> >  The first number is the number of threads
> 
> Yup, 32 of them.
> 
> >  The second is the number of times the a thread has started handling
> >   a request when all other threads were busy
> 
> 1439 in my case.  I guess ideally you want this number quite low.  How
> low is low enough?

There isn't much useful information you can get out of this number.
Either it is unchanging (in which case you have plenty of threads,
maybe more than you need) or it is changing (in which case you
sometimes are using all your threads, but that might be OK).

> 
> >  The remainder are the number of seconds when
> >     0-10% of threads were in use
> 
> 108.420 seconds in my example.  By and far the largest value with the
> next one being only 17.016 seconds.

This suggests that most of the time you are using < 10% of your
threads - i.e. 1, 2, or 3.  But sometimes you use more.

> >     ...
> >    90-100%
> 
> 0.672
> 
> Quite low indeed, yes?

Yes. quite low.

> 
> > 
> >  From this you should be able to get an idea of how many threads
> >  are getting used.
> >  Ideally, the top few numbers
> 
> What do you mean by the "top few numbers"
> 
   th 32 1439 108.420 17.016 6.044 3.724 2.464 0.296 0.256 0.280  0.672
                                                     ^^^^^^^^^^^^^^^^^^
these ones.

> >  should be very small compared with the
> >  earlier numbers.
> 
> What do you mean by the "earlier numbers"?

   th 32 1439 108.420 17.016 6.044 3.724 2.464 0.296 0.256 0.280  0.672
                      ^^^^^^^^^^^^^^^^^^
These ones (roughly).

> 
> >   If they are, increase the number of threads.
> 
> So, should I increase threads?
> 

No.


> >  I usually run with 128, but there are probably 100-200 clients, but
> >  they aren't very busy clients.
> 
> Any WAGs (or better) on the "cost" of a thread?
> 

I'm not really sure of the cost (and I have no idea what a 'WAG' is).
Several pages of RAM at least.  Possibly something more than that.

NeilBrown

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Number of nfsd processes?

[Neil Brown]

On Sunday June 25, simon.d.matthews+nfs@gmail.com wrote:
> Brian (and others):
> 
> On 6/25/06, Brian J. Murrell <brian@interlinx.bc.ca> wrote:
> > Neil,
> >
> > Your description was a bit ungrokkable.  Let me try to put some real
> > world values to it and try to make it out...
> 
> Here are my numbers:
> more /proc/net/rpc/nfsd
> rc 58145 148831950 678877510
> fh 126136 0 0 0 0
> io 3843561989 2872416345
> th 64 750787 29187.339 10107.193 6025.710 3273.713 3225.384 1680.524
> 725.219 1305.846 780.037 3180.079
> ....
> 
> Do these numbers ever get re-set? For example, I have re-started the
> nfs server process (to increas the number of nfsd processes), but I
> suspec the numbers merely keep aggregating, so they don't show what is
> happening now (with 64 processes).

No, they don't get reset.
It would be nice if we had a utility which reports the differences
between the current values and some previously stored values.  But we
don't.

As it stands, the fact that the last number (3180.079) is greater than
several earlier ones (1680.524, 725.219, 1305.846, 780.037) suggests
that it is often hitting the limit.

Now that you have increased to 64, take two samples one day apart (you
already have one sample here) and look at the differences.  They might
show something useful.

NeilBrown

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Number of nfsd processes?

[Carlos Carvalho]

Here are numbers from two servers:

th 8 513564 30829.484 1478.308 980.596 0.000 518.840 344.688 274.164
222.180 0.000 1613.580

th 8 28805553 21105.748 2502.784 2370.796 0.000 5226.212 2500.644
2579.308 3348.104 0.000 22526.404

According to Neils instructions I should increase the number of
threads in both of them because the last value is significant compared
to the previous ones, particularly in the second case. However what do
the zeros mean? They appear in both servers that run 2.6.16.22.


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Number of nfsd processes?

[J. Bruce Fields]

On Sat, Jul 08, 2006 at 08:12:30PM -0300, Carlos Carvalho wrote:
> Here are numbers from two servers:
> 
> th 8 513564 30829.484 1478.308 980.596 0.000 518.840 344.688 274.164
> 222.180 0.000 1613.580
> 
> th 8 28805553 21105.748 2502.784 2370.796 0.000 5226.212 2500.644
> 2579.308 3348.104 0.000 22526.404
> 
> According to Neils instructions I should increase the number of
> threads in both of them because the last value is significant compared
> to the previous ones, particularly in the second case. However what do
> the zeros mean? They appear in both servers that run 2.6.16.22.

They're totally unimportant; ignore them.

(The long version: it reports the amount of time that 10%, 20%, ...,
100% of the threads are busy.  You've only got 8 threads, so that's 8
values to stick in 10 slots, so a couple slots always end up empty.)

--b.


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs