* Performance drop 2.6.0-test7 -> 2.6.1-rc2
@ 2004-01-07 9:25 Jan Kasprzak
2004-01-13 10:11 ` SOLVED: " Jan Kasprzak
0 siblings, 1 reply; 2+ messages in thread
From: Jan Kasprzak @ 2004-01-07 9:25 UTC (permalink / raw)
To: linux-kernel
Hello, kernel hackers!
Yesterday I have upgraded the kernel on my FTP server to 2.6.1-rc2
(from 2.6.0-test7) and today I have found the server struggling under
load average of 40+ and the system was barely usable. The main load on the
server is ProFTPd serving mainly ISO 9660 images using sendfile().
Under 2.6.1-rc2 the server put out about 30 Mbit/s of data and load
average was >40. According to top(1) under 2.6.1-rc2, the CPU was about
30% system and 70% iowait. Killing proftpds of course helped and the load
went back to normal. After rebooting back to 2.6.0-test7 the load was 1-3
and bandwidth >50 Mbit/s (at which point it is probably limited by
bandwidth of FTP clients). During past releases of Linux distributions
the same hardware was able to serve ~400 FTP clients and >220 Mbit/s of
bandwidth.
The system is Athlon 850, 1.2GB RAM (CONFIG_HIGHMEM4G),
seven IDE disks, six of them joined to a 800GB LVM1 volume, which holds
an XFS filesystem with the anonymous FTP tree.
I think I can see a one possible explanation to this behaviour:
I have written a simple script to parse /proc/diskstats, and according
to its output, it seems that a block layer in 2.6.1-rc2 issues a several
times larger read operations than 2.6.0-test7. The later has requests
between 64 and 128 KB (and never bigger than 128 KB), while former can
apparently issue as big as 1MByte requests. Anyway, the output of my
script (5 second average) looks like this under 2.6.1-rc2:
disk rops read rsize rload wops write wsize wload pend util ioload
name [1/s] [KB/s] [KB] [req] [1/s] [KB/s] [KB] [req] [1] [%] [req]
hda 11.6 50 4.3 0.3 35.6 550 15.4 0.4 0 38.0 0.7
hde 104.6 6562 62.7 24.1 0.0 0 0.0 0.0 0 21.4 24.1
hdh 25.0 100 4.0 0.0 0.0 0 0.0 0.0 0 1.4 0.0
hdi 62.4 7798 125.0 12.0 0.0 0 0.0 0.0 38 95.2 9.9
hdj 20.0 15718 785.9 133.4 0.0 0 0.0 0.0 148 100.2 132.2
hdk 93.0 11807 127.0 67.2 0.0 0 0.0 0.0 23 100.2 42.6
hdl 58.2 9050 155.5 16.5 0.0 0 0.0 0.0 34 95.2 11.8
disk rops read rsize rload wops write wsize wload pend util ioload
name [1/s] [KB/s] [KB] [req] [1/s] [KB/s] [KB] [req] [1] [%] [req]
hda 47.0 1303 27.7 1.9 13.6 170 12.5 2.5 0 71.5 4.3
hde 68.8 4362 63.4 17.4 0.0 0 0.0 0.0 0 16.2 17.4
hdh 283.0 16861 59.6 92.3 0.0 0 0.0 0.0 0 71.2 92.3
hdi 49.6 9586 193.3 17.1 0.0 0 0.0 0.0 26 89.0 18.6
hdj 27.0 9248 342.5 137.2 0.0 1 0.0 0.0 126 100.3 133.4
hdk 298.2 11202 37.6 14.4 0.0 0 0.0 0.0 34 93.3 17.8
hdl 87.8 9549 108.8 17.0 0.0 0 0.0 0.0 31 85.4 19.8
and the following is from 2.6.0-test7:
rops read rsize rload wops write wsize wload pend util ioload
name [1/s] [KB/s] [KB] [req] [1/s] [KB/s] [KB] [req] [1] [%] [req]
hda 4.6 21 4.5 0.0 19.2 252 13.1 0.2 1 15.5 0.2
hde 42.6 2660 62.4 0.2 0.0 0 0.0 0.0 0 10.9 0.2
hdh 12.8 601 46.9 0.1 0.0 0 0.0 0.0 0 5.8 0.1
hdi 40.2 3992 99.3 0.3 0.0 0 0.0 0.0 0 25.0 0.3
hdj 30.8 2990 97.1 0.4 0.0 0 0.0 0.0 1 31.3 0.4
hdk 42.8 4843 113.2 0.3 0.0 0 0.0 0.0 0 24.3 0.3
hdl 81.8 9879 120.8 0.5 0.0 0 0.0 0.0 0 39.2 0.5
disk rops read rsize rload wops write wsize wload pend util ioload
name [1/s] [KB/s] [KB] [req] [1/s] [KB/s] [KB] [req] [1] [%] [req]
hda 8.4 34 4.1 0.2 57.2 574 10.0 4.2 0 41.3 4.4
hde 26.0 1461 56.2 0.2 0.0 0 0.0 0.0 0 8.1 0.2
hdh 54.4 3272 60.1 0.3 0.0 0 0.0 0.0 0 14.0 0.3
hdi 35.4 3590 101.4 0.3 0.6 6 9.3 0.0 0 29.0 0.3
hdj 61.8 5895 95.4 1.0 0.2 2 8.0 0.0 2 57.4 1.1
hdk 44.2 5042 114.1 0.2 0.4 2 6.0 0.0 0 21.2 0.2
hdl 13.6 1136 83.5 0.1 0.0 0 0.0 0.0 0 11.3 0.1
note the "rsize" column.
But of course the reason can be anything else (changes in XFS,
maybe?).
I am willing to do more testing, but since this is a production
server, I cannot go through each kernel version between -test7 and 2.6.1-rc2.
Can you explain (or better, suggest a fix for) this behaviour?
Thanks,
-Yenya
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |
| I actually have a lot of admiration and respect for the PATA knowledge |
| embedded in drivers/ide. But I would never call it pretty:) -Jeff Garzik |
^ permalink raw reply [flat|nested] 2+ messages in thread
* SOLVED: Performance drop 2.6.0-test7 -> 2.6.1-rc2
2004-01-07 9:25 Performance drop 2.6.0-test7 -> 2.6.1-rc2 Jan Kasprzak
@ 2004-01-13 10:11 ` Jan Kasprzak
0 siblings, 0 replies; 2+ messages in thread
From: Jan Kasprzak @ 2004-01-13 10:11 UTC (permalink / raw)
To: linux-kernel
Jan Kasprzak wrote:
: Yesterday I have upgraded the kernel on my FTP server to 2.6.1-rc2
: (from 2.6.0-test7) and today I have found the server struggling under
: load average of 40+ and the system was barely usable. The main load on the
: server is ProFTPd serving mainly ISO 9660 images using sendfile().
For archives: Andrew Morton suggested to apply the following patch
to mm/filemap.c (which in fact reverts part of the patch-2.6.1), and this
solved the problem (bigger-than-needed read ahead when doing sendfile()
of large files). 2.6.1 with this patch works for me now. Thanks!
-Yenya
diff -puN mm/filemap.c~readahead-partial-backout mm/filemap.c
--- 25/mm/filemap.c~readahead-partial-backout 2004-01-09 22:19:32.000000000 -0800
+++ 25-akpm/mm/filemap.c 2004-01-09 22:19:32.000000000 -0800
@@ -587,22 +587,13 @@ void do_generic_mapping_read(struct addr
read_actor_t actor)
{
struct inode *inode = mapping->host;
- unsigned long index, offset, last;
+ unsigned long index, offset;
struct page *cached_page;
int error;
cached_page = NULL;
index = *ppos >> PAGE_CACHE_SHIFT;
offset = *ppos & ~PAGE_CACHE_MASK;
- last = (*ppos + desc->count) >> PAGE_CACHE_SHIFT;
-
- /*
- * Let the readahead logic know upfront about all
- * the pages we'll need to satisfy this request
- */
- for (; index < last; index++)
- page_cache_readahead(mapping, ra, filp, index);
- index = *ppos >> PAGE_CACHE_SHIFT;
for (;;) {
struct page *page;
@@ -621,6 +612,7 @@ void do_generic_mapping_read(struct addr
}
cond_resched();
+ page_cache_readahead(mapping, ra, filp, index);
nr = nr - offset;
find_page:
_
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |
| I actually have a lot of admiration and respect for the PATA knowledge |
| embedded in drivers/ide. But I would never call it pretty:) -Jeff Garzik |
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2004-01-13 10:12 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2004-01-07 9:25 Performance drop 2.6.0-test7 -> 2.6.1-rc2 Jan Kasprzak
2004-01-13 10:11 ` SOLVED: " Jan Kasprzak
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).