Performance drop 2.6.0-test7 -> 2.6.1-rc2

[Jan Kasprzak <kas@informatics.muni.cz>]

	Hello, kernel hackers!

	Yesterday I have upgraded the kernel on my FTP server to 2.6.1-rc2
(from 2.6.0-test7) and today I have found the server struggling under
load average of 40+ and the system was barely usable. The main load on the
server is ProFTPd serving mainly ISO 9660 images using sendfile().

	Under 2.6.1-rc2 the server put out about 30 Mbit/s of data and load
average was >40. According to top(1) under 2.6.1-rc2, the CPU was about
30% system and 70% iowait. Killing proftpds of course helped and the load
went back to normal. After rebooting back to 2.6.0-test7 the load was 1-3
and bandwidth >50 Mbit/s (at which point it is probably limited by
bandwidth of FTP clients). During past releases of Linux distributions
the same hardware was able to serve ~400 FTP clients and >220 Mbit/s of
bandwidth.

	The system is Athlon 850, 1.2GB RAM (CONFIG_HIGHMEM4G),
seven IDE disks, six of them joined to a 800GB LVM1 volume, which holds
an XFS filesystem with the anonymous FTP tree.
 
	I think I can see a one possible explanation to this behaviour:
I have written a simple script to parse /proc/diskstats, and according
to its output, it seems that a block layer in 2.6.1-rc2 issues a several
times larger read operations than 2.6.0-test7. The later has requests
between 64 and 128 KB (and never bigger than 128 KB), while former can
apparently issue as big as 1MByte requests. Anyway, the output of my
script (5 second average) looks like this under 2.6.1-rc2:

disk   rops    read   rsize rload  wops   write   wsize wload pend  util ioload
name  [1/s]  [KB/s]    [KB] [req] [1/s]  [KB/s]    [KB] [req]  [1]   [%]  [req]
hda    11.6      50     4.3   0.3  35.6     550    15.4   0.4    0  38.0    0.7
hde   104.6    6562    62.7  24.1   0.0       0     0.0   0.0    0  21.4   24.1
hdh    25.0     100     4.0   0.0   0.0       0     0.0   0.0    0   1.4    0.0
hdi    62.4    7798   125.0  12.0   0.0       0     0.0   0.0   38  95.2    9.9
hdj    20.0   15718   785.9 133.4   0.0       0     0.0   0.0  148 100.2  132.2
hdk    93.0   11807   127.0  67.2   0.0       0     0.0   0.0   23 100.2   42.6
hdl    58.2    9050   155.5  16.5   0.0       0     0.0   0.0   34  95.2   11.8
                                                                                
disk   rops    read   rsize rload  wops   write   wsize wload pend  util ioload
name  [1/s]  [KB/s]    [KB] [req] [1/s]  [KB/s]    [KB] [req]  [1]   [%]  [req]
hda    47.0    1303    27.7   1.9  13.6     170    12.5   2.5    0  71.5    4.3
hde    68.8    4362    63.4  17.4   0.0       0     0.0   0.0    0  16.2   17.4
hdh   283.0   16861    59.6  92.3   0.0       0     0.0   0.0    0  71.2   92.3
hdi    49.6    9586   193.3  17.1   0.0       0     0.0   0.0   26  89.0   18.6
hdj    27.0    9248   342.5 137.2   0.0       1     0.0   0.0  126 100.3  133.4
hdk   298.2   11202    37.6  14.4   0.0       0     0.0   0.0   34  93.3   17.8
hdl    87.8    9549   108.8  17.0   0.0       0     0.0   0.0   31  85.4   19.8

	and the following is from 2.6.0-test7:

rops    read   rsize rload  wops   write   wsize wload pend  util ioload
name  [1/s]  [KB/s]    [KB] [req] [1/s]  [KB/s]    [KB] [req]  [1]   [%]  [req]
hda     4.6      21     4.5   0.0  19.2     252    13.1   0.2    1  15.5    0.2
hde    42.6    2660    62.4   0.2   0.0       0     0.0   0.0    0  10.9    0.2
hdh    12.8     601    46.9   0.1   0.0       0     0.0   0.0    0   5.8    0.1
hdi    40.2    3992    99.3   0.3   0.0       0     0.0   0.0    0  25.0    0.3
hdj    30.8    2990    97.1   0.4   0.0       0     0.0   0.0    1  31.3    0.4
hdk    42.8    4843   113.2   0.3   0.0       0     0.0   0.0    0  24.3    0.3
hdl    81.8    9879   120.8   0.5   0.0       0     0.0   0.0    0  39.2    0.5
 
disk   rops    read   rsize rload  wops   write   wsize wload pend  util ioload
name  [1/s]  [KB/s]    [KB] [req] [1/s]  [KB/s]    [KB] [req]  [1]   [%]  [req]
hda     8.4      34     4.1   0.2  57.2     574    10.0   4.2    0  41.3    4.4
hde    26.0    1461    56.2   0.2   0.0       0     0.0   0.0    0   8.1    0.2
hdh    54.4    3272    60.1   0.3   0.0       0     0.0   0.0    0  14.0    0.3
hdi    35.4    3590   101.4   0.3   0.6       6     9.3   0.0    0  29.0    0.3
hdj    61.8    5895    95.4   1.0   0.2       2     8.0   0.0    2  57.4    1.1
hdk    44.2    5042   114.1   0.2   0.4       2     6.0   0.0    0  21.2    0.2
hdl    13.6    1136    83.5   0.1   0.0       0     0.0   0.0    0  11.3    0.1
 
note the "rsize" column.

	But of course the reason can be anything else (changes in XFS,
maybe?).

	I am willing to do more testing, but since this is a production
server, I cannot go through each kernel version between -test7 and 2.6.1-rc2.

	Can you explain (or better, suggest a fix for) this behaviour?
Thanks,

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
|  I actually have a lot of admiration and respect for the PATA knowledge  |
| embedded in drivers/ide. But I would never call it pretty:) -Jeff Garzik |