Re: udevd is killing file write performance.

[Robin Holt <holt@sgi.com>]

On Wed, Feb 22, 2006 at 12:05:47PM -0800, Andrew Morton wrote:
> Robin Holt <holt@sgi.com> wrote:
> >
> > Let me reiterate, I know _VERY_ little about filesystems.  Can the
> >  dentry->d_lock be changed to a read/write lock?
> 
> Well, it could, but I suspect that won't help - the hold times in there
> will be very short so the problem is more likely acquisition frequency.
> 
> However it's a bit strange that this function is the bottleneck.  If their
> workload is doing large numbers of reads or writes from large numbers of
> processes against the same file then they should be hitting heavy
> contention on other locks, such as i_sem and/or tree_lock and/or lru_lock
> and others.
> 
> Can you tell us more about the kernel-visible behaviour of this app?

I looked at a little more of the output.  All I have to go on is a few
back traces generated by kdb of the entire system a few seconds apart.

In all of the traces, the first chunk of cpus are the only ones doing
writes.  I have not counted exactly, but I think it is around 32.  There
may be more or less, but that is the feeling (sometimes they are doing
reads as well).

The remainder of the cpus seem to be doing seeks to various parts of
the file and doing reads.  The seeks+read seem to overlap as I see
a nearly uniform read of 64k, but the seeks seem to be 4k aligned.
Also, at the time the snapshots were taken, they all are fairly
early in the file (between about 15% and 35% of the file) where
the writes are happening toward to early part of the second half
(50% - 70%).

I no longer have access to the test machine.  Since we told them
to turn off udevd, they no longer see a problem.  I will try to
convince them to allow some more testing.  I will also try to
recreate on our machine as well.

Thanks,
Robin