statd: server localhost not responding, timed out; lockd: cannot monitor <client>

statd: server localhost not responding, timed out; lockd: cannot monitor <client>

[Dan Wallis]

Hello list :)

This is my first post here, so if it's in the wrong place, or if I've
not read the appropriate documentation, please feel free to send me a
link to the relevant bits. I've tried searching the archives, and the
internet at large, but haven't yet come across anything that's helped.

Our NFS-shared file-system is locking up.

At the time, there are a lot of processes in "disk sleep" state, and
the load averages on our machines sky-rocket. The machines are
responsive on SSH, but our the majority of our websites
(apache+mod_php) just hang, as does our email system (exim+dovecot).
Any websites which don't require write access to the file-system
continue to operate.

The load averages continue to rise until some kind of time-out is
reached, but for at least 10-15 minutes. I've seen load averages over
800, yet the machines are still responsive for actions which don't
require writing to the shared file-system.

I've been investigating a variety of options, which have all turned
out to be red-herrings: nagios, proftpd, bind, cron tasks.

I'm seeing these messages in the file server's system log:

Jul 30 09:37:17 fs0 kernel: [1810036.560046] statd: server localhost
not responding, timed out
Jul 30 09:37:17 fs0 kernel: [1810036.560053] nsm_mon_unmon: rpc
failed, status=-5
Jul 30 09:37:17 fs0 kernel: [1810036.560064] lockd: cannot monitor node2
Jul 30 09:38:22 fs0 kernel: [1810101.384027] statd: server localhost
not responding, timed out
Jul 30 09:38:22 fs0 kernel: [1810101.384033] nsm_mon_unmon: rpc
failed, status=-5
Jul 30 09:38:22 fs0 kernel: [1810101.384044] lockd: cannot monitor node0

Software involved:
VMWare, Debian lenny (64bit), ancient Red Hat (32 bit) (version 7 I
believe), Debian etch (32bit)
NFS, apache2+mod_php, exim, dovecot, bind, amanda, proftpd, nagios,
cacti, drbd, heartbeat, keepalived, LVS, cron, ssmtp, NIS, svn,
puppet, memcache, mysql, postgres
Joomla!, Magento, Typo3, Midgard, Symfony, custom php apps

So far, I've spent a lot of time trying to reproduce the problem, but
haven't been able to do so.


If you've any ideas, please let me know. :)



Cheers
Dan

Re: statd: server localhost not responding, timed out; lockd: cannot monitor <client>

[Bian Naimeng]

> Hello list :)
> 
> This is my first post here, so if it's in the wrong place, or if I've
> not read the appropriate documentation, please feel free to send me a
> link to the relevant bits. I've tried searching the archives, and the
> internet at large, but haven't yet come across anything that's helped.

   ... snip ...

> I'm seeing these messages in the file server's system log:
> 
> Jul 30 09:37:17 fs0 kernel: [1810036.560046] statd: server localhost
> not responding, timed out
> Jul 30 09:37:17 fs0 kernel: [1810036.560053] nsm_mon_unmon: rpc
> failed, status=-5
> Jul 30 09:37:17 fs0 kernel: [1810036.560064] lockd: cannot monitor node2
> Jul 30 09:38:22 fs0 kernel: [1810101.384027] statd: server localhost
> not responding, timed out
> Jul 30 09:38:22 fs0 kernel: [1810101.384033] nsm_mon_unmon: rpc
> failed, status=-5
> Jul 30 09:38:22 fs0 kernel: [1810101.384044] lockd: cannot monitor node0
> 

  It looks like that nfslock service(statd) not running, please try run the
  followed command,

  # service rpcbind start
  # service nfslock restart

-- 
Regards
Bian Naimeng