Help with pdflush latency / congestion on older kernel (2.6.18)

[Erik Slagter <erik@slagter.name>]

Hi all,

First of all I must apologise that this relates to an older kernel
version. This kernel is used on a satellite top-box with a closed source
module of it's vendor that handles the specific hardware, so we cannot
use a new kernel, sadly. The good news is that the problem has nothing
to do with the closed source module. I was hoping that maybe this is a
known problem and has been fixed in newer kernel version and so I can
try to backport it (or maybe that has already been done...)

This is the problem. Until recently a 2.6.9 kernel was used (yep, I
hesitate to admit...) The vendor now switched to 2.6.18 and since then
we have a sort of disk write congestion problem, at least on NFS but
maybe also on local (ata) disks. It appears that pdflush waits too long
before it starts writing the dirty pages, which results in the processes
that fetch and write the data, are being blocked at a certain point. In
the meantime, data keeps coming in (satellite tv being notorious for
it's lack of flow control ;-)) and the hardware and/or os level queues
overflow, because they're not fetched in time. This is especially true
when more than one recording takes place and/or a simultaneous playback
takes place.

I think changing the user space process (like using additional threads)
will not solve the problem completely and also would be rather a
workaround instead.

We've done lot's of testing and tweaking in the meantime, detailed
information is available at request!

Things we tried already:
 - various incantations of posix_fadvise on the file to be written, only
once or on a frequent base (posix_fadvise(...DONT_NEED...))
 - opening using O_SYNC
 - tweaking /proc/vm/: dirty_background_ratio, dirty_expire_centisecs, 
dirty_ratio, dirty_writeback_centisecs

It appears that dirty_writeback_centisecs cannot be set to any value
below 5 seconds. We believe that the possibility to wake up pdflush
earlier actually would help in this case.

Would using O_DIRECT be a possible solution? I cannot test that at the
moment though, because the out-of-the-box doesn't have it enabled in the
config, but I probably can get hold a kernel that has it enabled.

Can anyone supply some help, clues, hints, tips, etc., it's very much
appreciated! Thanks very much!