Re: ext3 IO latency measurements (was: Linux 2.6.29)

[Theodore Tso <tytso@mit.edu>]

On Fri, Mar 27, 2009 at 12:02:40AM +0100, Ingo Molnar wrote:
> This isnt me streaming gigs of data in and out of the system 
> dirtying 90% of all RAM. This is a trivial workload barely 
> scratching the RAM and CPU capabilities of the system.

Have you tried with maxcpus set to say, 2?  My guess is you won't see
the problems in that case.  So I'm not sure saying "barely scratching
the CPU capabilities of the system" is completely fair.  I can
probably get be able to get temporary access to a 16 CPU system, but
that's not the kind of system that I normally get to use for my kernel
devleopment.

> Have you tried to reproduce it? Have you tried CONFIG_LATENCYTOP? We 
> implemented that kernel feature specifically to make it easy for 
> developers to instrument their kernel and keep system latencies 
> down. This isnt some oddball workload or oddball system. These 
> latencies are reproducible on just about any Linux development 
> system i ever tried with ext3.

My normal development is not all that different from yours (make
-j<numcpus*2>) and I do edit and save files while the compile is
going.  I use emacs, but it calls fsync() when saving files, just like
vim does.  The big difference is that for me, numcpus is normally 2.
And my machine has 4 gigs of memory, not 12 gigs.  So I don't see
these problems.  I agree that what you have isn't an "oddball
workload"; as far as whether it is an "oddball system", it is
certainly a system I would lust after.  And I acknowledge the world is
a bit different from when Linus declared that 99% of the world was 1
or 2 CPU's.  I suspect the percentage of machines with 16 CPU's is
still somewhat small, though.

So I'll try to reproduce it on a 16 CPU system, when I have a chance
--- but it's something that I'm going to have to borrow and try to get
remote access to play with such a system.  Clearly your employer is
way more generous with equipment than mine is, at least for personal
development machines.  :-)

In the meantime, if you could run some of the tests and vary some of
the variables I requested, I'd appreciate it, and thank you for your
help.  Otherwise, I'll try to run them when I get remote access to
such a machine where I'm allowed to replace kernels and mount random
test filesystems.

						- Ted

P.S.  Another interesting test would be to plot the vim save latencies
versus the number of CPU's enabled when running the kernel build
workload.

P.P.S.  I assume there's no way you could give me remote ssh access to
your nice 16-way machine?  :-)