Failing to allocated file handles?

Failing to allocated file handles?

[Jamie Harris]

Hello all,

Sorry if this isn't the kind of thing you guys deal with, if this is the
case then feel free to tell me a better place to ask my question!  I
haven't managed to find anywhere else that's got lots of people who know
a sufficient amount about kernel internals.

We've got an SMP P3 server running Redhat's (sorry, I would roll my own
but it's not 'my' box) 2.4.18-27.7.xsmp kernel.  This box runs Apache
and ColdFusion (spit!).  The ColdFusion (spit!) server appears to decide
to open a number of files in a short burst, then promptly dies.  I've
worked out from monitoring /proc/sys/fs/file-nr that we're running out
of available file handles.  The number of allocated file handles (sorry
if my terminalogy is way off, I'm talking about the first number in
/proc/sys/fs/file-nr) hovers around 2500, the number of available file
handles (the second number) drops to around zero, then gets to zero and
the ColdFusion (spit!) server dies and reports being out of files.

It appears that the system is not allocating new file handles fast
enough to keep up with the rate they are being consumed.  Is this
possible?  I'm unable to replicated this on a single processor machine
so am assuming that it's related to different threads on different CPUs
not 'keeping up with one another'.  

What do the kernel devlopers think?  Any info creatly appreaciated, even
if its only "get and build the latest kernel", at least I'll have some
more info to tell our customer and the powers-that-be.  Oddly enough
we're only experiencing this on a single machine, although we have lots
of identical boxes running similar software.

Cheers all.

Jamie...

--
Jamie Harris, NISS Service Delivery Team, 
EduServ, Queen Anne House, 11 Charlotte Street, Bath, BA1 2NE
Tel: +44 (0)1225 474300 Fax: +44 (0)1225 474301
Direct Tel: +44 (0)1225 474388

**  This message was transmitted on 100% recycled electrons **


 

Re: Failing to allocated file handles?

[Alan Cox]

On Maw, 2003-05-06 at 09:40, Jamie Harris wrote:
> It appears that the system is not allocating new file handles fast
> enough to keep up with the rate they are being consumed.  Is this
> possible?  I'm unable to replicated this on a single processor machine
> so am assuming that it's related to different threads on different CPUs
> not 'keeping up with one another'.  

Seems wildly improbable. And if you've reached the max file limit you
set why do you expect the kernel to exceed it. Given the max files is
set based on the system memory I'd put my bet on it being a low memory
box that hits the problem. If so just write a bigger value to the
setting

RE: Failing to allocated file handles?

[Jamie Harris]

Alan, 

Thanks for the response...

> Seems wildly improbable. And if you've reached the max file 
> limit you set why do you expect the kernel to exceed it. 

That's the thing, its nowhere near file-max, yet the number
of available file handles hits zero.  File-max was at 235928, 
~2000 file handles were alloacted, the available handles hit
zero but the number allocated didn't increase, if you see what
I mean.

> Given the max files is set based on the system memory I'd put 
> my bet on it being a low memory box that hits the problem. If 
> so just write a bigger value to the setting

2GB RAM, lots available

[root@cuboid logs]#  free
             total       used       free     shared    buffers
cached
Mem:       2323304    2153068     170236          0     107176
1695984
-/+ buffers/cache:     349908    1973396
Swap:      2097112        832    2096280

Thanks for the response

Jamie...