gdth / SCSI read performance issues (2.2.19 and 2.4.10)

gdth / SCSI read performance issues (2.2.19 and 2.4.10)

[Marinos Yannikos]

Hi,

our brand-new ICP GDT8523RZ controller with 6 disks peaks out at
45MB/s under 2.4.10, while with 2.2.19 it reaches 85MB/s (seq.
read performance). It should realistically be able to reach
at least 150-200MB/s in this configuration (RAID-5, 6 disks,
it's a 64-bit 66MHz PCI card). It's in a dual P3-1GHz box with
Tyan 2510NG board. Sequential write performance is OK with both
kernels, it's higher than the read performance on 2.4.10 (60MB/s).

Under 2.4.10, when I test the performance with a simple program
that just read()'s 16MB blocks from /dev/sda, both CPUs report
40-50% System time usage. 2.2.19 reports ~20% for one CPU (and
better performance).

Is there anything that can be done about this? It seems like a
serious performance problem either with the gdth driver or the
kernel and it renders this neat controller rather ineffective.

Regards,
-mjy
[please CC: answers]

Re: gdth / SCSI read performance issues (2.2.19 and 2.4.10)

[Andrew Morton]

Marinos Yannikos wrote:
> 
> Hi,
> 
> our brand-new ICP GDT8523RZ controller with 6 disks peaks out at
> 45MB/s under 2.4.10, while with 2.2.19 it reaches 85MB/s (seq.
> read performance). It should realistically be able to reach
> at least 150-200MB/s in this configuration

Well that's pretty bad, isn't it?

Some things which it would be interesting to try out:

- Disable CONFIG_HIGHMEM in your kernel config, see
  what that does.

- Try linux-2.4.13 -- 2.4.10 was a bit of a dog (this may make
  a difference if the driver uese the new PCI DMA API,
  but it doesn't explain why 2.2.x does so much better).

- Profile the kernel.  You may enable profiling by booting
  the kernel with the LILO option `profile=1'.

  Then, making sure that /boot/System.map reflects the
  running kernel, run this little script

#!/bin/sh
TIMEFILE=/tmp/$(basename $1).time
sudo readprofile -r
time "$@"
readprofile -v -m /boot/System.map | sort -n +2 | tail -40 | tee $TIMEFILE
echo created $TIMEFILE

  With something like:

	~/kern-prof.sh cp some_huge_file /dev/null

Re: gdth / SCSI read performance issues (2.2.19 and 2.4.10)

[Marinos J. Yannikos]

Andrew Morton wrote:

> Well that's pretty bad, isn't it?


We could have bought a much cheaper controller and slower disks ;-)


> - Disable CONFIG_HIGHMEM 


That seems to have no effect on performance. Btw., the 64GB support in
2.4.10 seemed to be buggy ("0-order allocation failed", then the DB
crashed), so we were using the 4GB setting.


> - Try linux-2.4.13


This helped - now performance is up to par with 2.2.19 (~ 85MB/s) - thanks!


> - Profile the kernel. [...]  With something like:
> 	~/kern-prof.sh cp some_huge_file /dev/null


I tried this, but with
  dd if=/dev/sda of=/dev/null bs=1024k count=3000

(the fs is too slow - so "cp" peaks out at 17MB/s!)

The result (last 4 lines):
c01388fc try_to_free_buffers                          55   0.1511
c0128b10 file_read_actor                            1179  14.0357
c01053b0 default_idle                               6784 130.4615
00000000 total                                      8695   0.0065

Does this suggest that the kernel isn't the bottleneck?


Regards,
  Marinos
-- 
Marinos Yannikos, CEO
Preisvergleich Internet Services AG
Franzensbrückenstraße 8/2/16, A-1020 Wien
Tel./Fax: (+431) 5811609-52/-55

Re: gdth / SCSI read performance issues (2.2.19 and 2.4.10)

[Andrew Morton]

"Marinos J. Yannikos" wrote:
> 
> Andrew Morton wrote:
> 
> > Well that's pretty bad, isn't it?
> 
> We could have bought a much cheaper controller and slower disks ;-)

It's Linux's way of saving you money :)

> > - Disable CONFIG_HIGHMEM
> 
> That seems to have no effect on performance. Btw., the 64GB support in
> 2.4.10 seemed to be buggy ("0-order allocation failed", then the DB
> crashed), so we were using the 4GB setting.

There have been many reports of this happening, but they
are tapering off now.

> > - Try linux-2.4.13
> 
> This helped - now performance is up to par with 2.2.19 (~ 85MB/s) - thanks!

It's surprising that 2.4.10->2.4.13 actually doubled throughput.
 
> > - Profile the kernel. [...]  With something like:
> >       ~/kern-prof.sh cp some_huge_file /dev/null
> 
> I tried this, but with
>   dd if=/dev/sda of=/dev/null bs=1024k count=3000
> 
> (the fs is too slow - so "cp" peaks out at 17MB/s!)

Really?  Are you saying that on a controller which can
do 85 megs/second, you can't read files through the filesystem
at greater than 17?  Which filesystem?

> The result (last 4 lines):
> c01388fc try_to_free_buffers                          55   0.1511
> c0128b10 file_read_actor                            1179  14.0357
> c01053b0 default_idle                               6784 130.4615
> 00000000 total                                      8695   0.0065

OK, that's normal and proper.  Almost all the kernel time
is spent copying data.
 
> Does this suggest that the kernel isn't the bottleneck?

Well...   We seem to have three issues here:

1: Why isn't the controller achieving the manufacturer's
   claimed throughput?

   Don't know.  Maybe it's the software copy.  Maybe it's the
   device driver.  Maybe they lied :).  It'd be interesting
   to test it on the same machine with the vendor's drivers
   and win2k.

2: 0-order allocation failures.

3: Poor `cp' throughput.  This one is strange.  Perhaps
   `cp' is using a small transfer-size and the kernel's
   readhead isn't working properly.  Could you experiment
   with this some more?  For example, what happens with

	dd if=large_file of=/dev/null bs=4096k

-

Re: gdth / SCSI read performance issues (2.2.19 and 2.4.10)

[Marinos J. Yannikos]

Andrew Morton wrote:

> [...] you can't read files through the filesystem
> at greater than 17?  Which filesystem?


ext2 (on which I did a "tune2fs -j", but then I found out that the
kernel apparently doesn't support ext3). But, see below.


>    [...] It'd be interesting to test it on the same machine with

>    the vendor's drivers and win2k.


Indeed - can't do so, unfortunately. It could also be an issue
with the RAID-5 configuration (block size and arrangement of the
disks in the array could be less than optimal as suggested by
the ICP BIOS), so I'm in touch with the very helpful ICP support
people as well.


> 	dd if=large_file of=/dev/null bs=4096k


OK, that's quite odd - now I get a reasonable ~55 MB/s whatever I
try (dd, cp), so that must have been my mistake (or due to the
kernel profiling option?).

By the way, I'm having lock-ups with 2.4.13 that I can reproduce
(vi <file on reiserfs partition> ... :q ... "Segmentation fault"
... lock-up - console input still works, but apparently processes
don't get scheduled).

Regards,
  Marinos

-- 
Marinos Yannikos, CEO
Preisvergleich Internet Services AG
Franzensbrückenstraße 8/2/16, A-1020 Wien
Tel./Fax: (+431) 5811609-52/-55