* Linux 2.4/2.5 SCSI considerably slower than FreeBSD
@ 2002-05-17 11:53 Matthias Andree
2002-05-19 12:38 ` Geert Uytterhoeven
0 siblings, 1 reply; 6+ messages in thread
From: Matthias Andree @ 2002-05-17 11:53 UTC (permalink / raw)
To: Linux-Kernel mailing list
Hello,
Marco Flohrer has posted an inquiry to de.comp.os.unix.linux.hardware
[German] <slrnae8q66.go4.marco.flohrer@diamond.csn.tu-chemnitz.de> that his
Seagate 36ES2 was slow with a DawiControl 2976UW (SYM53C875), only
around 25 MB/s. I have the same observation with a Fujitsu MAH3182MP
with an Adaptec 2940UW Pro which is not much faster. Either bus has an
active LVD/SE terminator.
Single-user mode,
time dd if=/dev/XXX of=/dev/null bs=65536 count=10240
(671,1 MB) linear read.
Table shows throughput in decimal MB/s (M = 1,000,000)
2.5 2.4 FBSD max.
UWSCSI Fuj MAH3182MP 7200/min 32,1 29,4 35,1 TQ 40
UDMA66 Max 4W060H4 5400/min 27,1 26,7 25,7 66
UDMA66 IBM DTLA307045 7200/min 37,2 37,5 37,2 TQ 2.5 66
UDMA66 WDC AC420400D 5400/min 15,5 15,5 15,5 TQ 2.5 66
--------------
table is in decimal MB/s.
2.4: Linux 2.4.19-pre2-ac3
2.5: Linux 2.5.15
FBSD: FreeBSD 4.6-RC (Tagged Queueing Broken)
The IDE drives are attached to a VIA 82C686 (KT133), the Fujitsu
(actually an U-160 drive) to the mentioned Adaptec.
FBSD gets about 20% better throughput. It's far from perfect, but 90% of
the maximum is probably almost as good as we can get.
Why is Linux SCSI so slow?
--
Matthias Andree
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux 2.4/2.5 SCSI considerably slower than FreeBSD
2002-05-17 11:53 Linux 2.4/2.5 SCSI considerably slower than FreeBSD Matthias Andree
@ 2002-05-19 12:38 ` Geert Uytterhoeven
0 siblings, 0 replies; 6+ messages in thread
From: Geert Uytterhoeven @ 2002-05-19 12:38 UTC (permalink / raw)
To: Matthias Andree; +Cc: Linux-Kernel mailing list
On Fri, 17 May 2002, Matthias Andree wrote:
> Marco Flohrer has posted an inquiry to de.comp.os.unix.linux.hardware
> [German] <slrnae8q66.go4.marco.flohrer@diamond.csn.tu-chemnitz.de> that his
> Seagate 36ES2 was slow with a DawiControl 2976UW (SYM53C875), only
> around 25 MB/s. I have the same observation with a Fujitsu MAH3182MP
> with an Adaptec 2940UW Pro which is not much faster. Either bus has an
> active LVD/SE terminator.
>
> Single-user mode,
> time dd if=/dev/XXX of=/dev/null bs=65536 count=10240
> (671,1 MB) linear read.
>
> Table shows throughput in decimal MB/s (M = 1,000,000)
>
> 2.5 2.4 FBSD max.
> UWSCSI Fuj MAH3182MP 7200/min 32,1 29,4 35,1 TQ 40
> UDMA66 Max 4W060H4 5400/min 27,1 26,7 25,7 66
> UDMA66 IBM DTLA307045 7200/min 37,2 37,5 37,2 TQ 2.5 66
> UDMA66 WDC AC420400D 5400/min 15,5 15,5 15,5 TQ 2.5 66
> --------------
> table is in decimal MB/s.
I used to get 17 MiB/s with a Quantum Viking II U2W connected to the wide chain
of a DawiControl 2976UW (SYM53C875) on my PPC box. This was using the old
sym53c8xx driver (don't remember whether I ever rerun the test with sym2).
I just reran the test (both dd and hdparm) using my current kernel
(2.4.17-pre2, using sym2) and I got only 12 MiB/s.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux 2.4/2.5 SCSI considerably slower than FreeBSD
2002-05-18 4:13 Douglas Gilbert
2002-05-19 20:50 ` Matthias Andree
@ 2002-05-23 3:41 ` Douglas Gilbert
1 sibling, 0 replies; 6+ messages in thread
From: Douglas Gilbert @ 2002-05-23 3:41 UTC (permalink / raw)
To: Matthias Andree, linux-kernel, linux-scsi
Douglas Gilbert wrote:
>
> Matthias Andree wrote:
> >
> > Marco Flohrer has posted an inquiry to de.comp.os.unix.linux.hardware
> > [German] <slrnae8q66.go4.marco.flohrer@diamond.csn.tu-chemnitz.de> that his
> > Seagate 36ES2 was slow with a DawiControl 2976UW (SYM53C875), only
> > around 25 MB/s. I have the same observation with a Fujitsu MAH3182MP
> > with an Adaptec 2940UW Pro which is not much faster. Either bus has an
> > active LVD/SE terminator.
> >
> > Single-user mode,
> > time dd if=/dev/XXX of=/dev/null bs=65536 count=10240
> > (671,1 MB) linear read.
> >
> > Table shows throughput in decimal MB/s (M = 1,000,000)
> >
> > 2.5 2.4 FBSD max.
> > UWSCSI Fuj MAH3182MP 7200/min 32,1 29,4 35,1 TQ 40
> > UDMA66 Max 4W060H4 5400/min 27,1 26,7 25,7 66
> > UDMA66 IBM DTLA307045 7200/min 37,2 37,5 37,2 TQ 2.5 66
> > UDMA66 WDC AC420400D 5400/min 15,5 15,5 15,5 TQ 2.5 66
> > --------------
> > table is in decimal MB/s.
> >
> > 2.4: Linux 2.4.19-pre2-ac3
> > 2.5: Linux 2.5.15
> > FBSD: FreeBSD 4.6-RC (Tagged Queueing Broken)
> >
> > The IDE drives are attached to a VIA 82C686 (KT133), the Fujitsu
> > (actually an U-160 drive) to the mentioned Adaptec.
> >
> > FBSD gets about 20% better throughput. It's far from perfect, but 90% of
> > the maximum is probably almost as good as we can get.
> >
> > Why is Linux SCSI so slow?
Matthias,
It is difficult to answer your main question but here are
some more figures on a pretty fast disk restricted by
a SCSI UW parallel bus (maximum bandwidth: 40 MB/sec).
Disk read speeds (MB/sec) for various block sizes
=================================================
block || dd | sg_dd | sgm_dd
size || lk2.4 | lk2.5 | lk2.4 | lk2.5 | lk2.4 | lk2.5
==================================================================
2 KB || 31.5 35.1 | 16.3 15.6 | 17.0 16.2
4 KB || 31.4 35.0 | 22.4 22.4 | 23.9 23.8
8 KB || 31.3 35.7 | 27.7 27.7 | 29.9 29.8
16 KB || 31.4 35.7 | 31.3 31.4 | 34.2 34.2
32 KB || 31.4 35.7 | 33.6 33.6 | 36.9 36.9
64 KB || 31.5 35.7 | 34.7 34.7 | 38.2 38.2
128 KB || 31.5 35.5 | 34.9 34.9 | 39.0 39.0
256 KB || 31.5 35.5 | 33.3 33.4 | 39.3 39.3
Reading a Fujitsu MAM 3184MP disk (SCSI u160 capable) on the
Ultra Wide channel (max bandwidth 40 MB/sec) of a Tekram DC-390U3W
dual controller. The HBA driver is sym53c8xx_2 (version sym-2.1.17a
for lk 2.4.19-pre7 and version sym-2.1.16a for lk 2.5.17).
The block size for the dd command was the figure given to the "bs"
argument and the effective figure given to "bpt" for sg_dd and
sgm_dd (i.e. bpt=block_size/512). [Dropping the block size for
dd to 512 and 256 bytes made virtually no difference either.]
sg_dd uses the sg interface to access the disk and copies data
via kernel buffers while sgm_dd memory maps those kernel
buffers to the user space. In all cases 1 GB of data was read
from the outer tracks (lba==0).
Conclusions:
- the block size given to dd has very little impact on its
performance
- the dd in lk 2.5 performs better than the one in lk 2.4
- otherwise performance is roughly similar
- obviously the 40 MB/sec bandwidth of the scsi UW bus is
the limiting factor with larger block sizes
Here are some numbers for the same disk on the U160 channel of the
same DC-390U3W controller in lk 2.5.17:
dd sg_dd sgm_dd
2 KB 56.2 20.5 22.8
8 KB 56.8 50.0 56.7
32 KB 56.7 57.0 57.2
The test system is a 1.2 GHz Athlon on a Asus A7M266 MoBo with
512 MB of DDR ram. The test disk had no fs mounted and was on
a bus by itself.
Doug Gilbert
22nd May 2002
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux 2.4/2.5 SCSI considerably slower than FreeBSD
2002-05-19 20:50 ` Matthias Andree
@ 2002-05-19 23:53 ` Tomas Szepe
0 siblings, 0 replies; 6+ messages in thread
From: Tomas Szepe @ 2002-05-19 23:53 UTC (permalink / raw)
To: linux-kernel
> > $ time sg_dd if=/dev/sg1 of=/dev/null bs=512 count=2m time=1
> > time to transfer data was 18.786448 secs, 57.16 MB/sec
> > 2097152+0 records in
> > 2097152+0 records out
> > real 0m18.799s user 0m0.030s sys 0m3.010s
>
> In a live system (actually, it's idle, but every 5 s, there is a short
> burst of disk activity -- reiserfs and ext3fs in use here, something is
> going on there), sg_dd is not really better than plain dd:
>
> > time sg_dd if=/dev/sg0 of=/dev/null count=1310720
> > Assume default 'bs' (block size) of 512 bytes
> > 1310720+0 records in
> > 1310720+0 records out
> >
> > real 0m24.348s
>
> gives: 27,56 MB/s. A little better than dd, but still much less than FreeBSD's.
hmmm.
# sync; sync; time dd if=/dev/sda of=/dev/null bs=64k count=10000
10000+0 records in
10000+0 records out
real 0m12.809s
user 0m0.020s
sys 0m2.560s
(first read, the data shouldn't have been cached)
-> 48.79 MB/s
I wouldn't suspect there's a problem, the numbers look reasonable.
$ uname -r
2.4.19-pre2
Disk:
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: SEAGATE Model: ST336706LW Rev: 0109
Type: Direct-Access ANSI SCSI revision: 03
(i.e., 36GiB 160MB/s Seagate Cheetah)
Controller - HP netserver onboard sym53c896.
T.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux 2.4/2.5 SCSI considerably slower than FreeBSD
2002-05-18 4:13 Douglas Gilbert
@ 2002-05-19 20:50 ` Matthias Andree
2002-05-19 23:53 ` Tomas Szepe
2002-05-23 3:41 ` Douglas Gilbert
1 sibling, 1 reply; 6+ messages in thread
From: Matthias Andree @ 2002-05-19 20:50 UTC (permalink / raw)
To: linux-kernel
On Sat, 18 May 2002, Douglas Gilbert wrote:
> $ time sg_dd if=/dev/sg1 of=/dev/null bs=512 count=2m time=1
> time to transfer data was 18.786448 secs, 57.16 MB/sec
> 2097152+0 records in
> 2097152+0 records out
> real 0m18.799s user 0m0.030s sys 0m3.010s
In a live system (actually, it's idle, but every 5 s, there is a short
burst of disk activity -- reiserfs and ext3fs in use here, something is
going on there), sg_dd is not really better than plain dd:
> time sg_dd if=/dev/sg0 of=/dev/null count=1310720
> Assume default 'bs' (block size) of 512 bytes
> 1310720+0 records in
> 1310720+0 records out
>
> real 0m24.348s
gives: 27,56 MB/s. A little better than dd, but still much less than FreeBSD's.
> >From memory, dd's performance in the lk 2.4 series was considerably
> lower than sg_dd. No doubt FreeBSD would also perform well but I
> doubt it could beat linux (2.5) by the type of margin your measurements
> indicate. [For sequential reads, tagged queueing will not have a
It does. I trust my last figures. I don't trust the above too much
because of the bogus 5 s interval write burst. Not sure where it comes
from, ext3fs or reiserfs, I'd pretty much like something like an inverse
strace: "who is writing to my disk?"
> significant impact.] It is also worth noting that the new aic7xxx
> and sym53c8xx_2 drivers are essentially the same on Linux and
> FreeBSD (i.e. same code base, same maintainers).
I know that, and I assume it's not the fault of the drivers themselves
(that's because the "far from theoretical optimum" holds on either
hardware).
Looks like either you're lucky or like Linux has some bad VM interaction
or Linux has issues with U160-drives on UWSCSI adapters (yours does
U160, ours don't) that FreeBSD doesn't have. (FreeBSD's IDE driver is
not as good as the Linux 2.5 IDE driver, so in the end, FreeBSD and
Linux are even again ;-)
--
Matthias Andree
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux 2.4/2.5 SCSI considerably slower than FreeBSD
@ 2002-05-18 4:13 Douglas Gilbert
2002-05-19 20:50 ` Matthias Andree
2002-05-23 3:41 ` Douglas Gilbert
0 siblings, 2 replies; 6+ messages in thread
From: Douglas Gilbert @ 2002-05-18 4:13 UTC (permalink / raw)
To: Matthias Andree; +Cc: linux-kernel, linux-scsi
Matthias Andree wrote:
>
> Marco Flohrer has posted an inquiry to de.comp.os.unix.linux.hardware
> [German] <slrnae8q66.go4.marco.flohrer@diamond.csn.tu-chemnitz.de> that his
> Seagate 36ES2 was slow with a DawiControl 2976UW (SYM53C875), only
> around 25 MB/s. I have the same observation with a Fujitsu MAH3182MP
> with an Adaptec 2940UW Pro which is not much faster. Either bus has an
> active LVD/SE terminator.
>
> Single-user mode,
> time dd if=/dev/XXX of=/dev/null bs=65536 count=10240
> (671,1 MB) linear read.
>
> Table shows throughput in decimal MB/s (M = 1,000,000)
>
> 2.5 2.4 FBSD max.
> UWSCSI Fuj MAH3182MP 7200/min 32,1 29,4 35,1 TQ 40
> UDMA66 Max 4W060H4 5400/min 27,1 26,7 25,7 66
> UDMA66 IBM DTLA307045 7200/min 37,2 37,5 37,2 TQ 2.5 66
> UDMA66 WDC AC420400D 5400/min 15,5 15,5 15,5 TQ 2.5 66
> --------------
> table is in decimal MB/s.
>
> 2.4: Linux 2.4.19-pre2-ac3
> 2.5: Linux 2.5.15
> FBSD: FreeBSD 4.6-RC (Tagged Queueing Broken)
>
> The IDE drives are attached to a VIA 82C686 (KT133), the Fujitsu
> (actually an U-160 drive) to the mentioned Adaptec.
>
> FBSD gets about 20% better throughput. It's far from perfect, but 90% of
> the maximum is probably almost as good as we can get.
>
> Why is Linux SCSI so slow?
With a Fujitsu MAM3184 (U160, 15Krpm 18GB) disk and a Tekram
DC-390U3W controller (sym53c8xx_2 driver) on lk 2.5.15 I get:
$ time dd if=/dev/sdb of=/dev/null bs=64k count=16k
16384+0 records in
16384+0 records out
real 0m18.948s user 0m0.010s sys 0m4.090s
That is 56.67 MB/sec (MB == 10^6).
$ time sg_dd if=/dev/sg1 of=/dev/null bs=512 count=2m time=1
time to transfer data was 18.786448 secs, 57.16 MB/sec
2097152+0 records in
2097152+0 records out
real 0m18.799s user 0m0.030s sys 0m3.010s
$ time sgm_dd if=/dev/sg1 of=/dev/null bs=512 count=2m time=1
time to transfer data was 18.777035 secs, 57.18 MB/sec
2097152+0 records in
2097152+0 records out
real 0m18.781s user 0m0.020s sys 0m0.100s
The MAM3184 disk was recently reviewed
( see http://www4.tomshardware.com/storage/02q2/020415/index.html )
and those speeds are very close to the maximum in their benchmarks
(and Fujitsu's published specifications) for outer track reads.
I am impressed by dd's performance in the lk 2.5 series.
When sg_dd and sgm_dd are used they bypass the block subsystem
and issue 64KB SCSI read commands (in this case). As can be seen
above, this improves the throughput by about 1 % compared to dd.
CPU utilization (on a Athlon 1.2 GHz box with 512 MB of DDR ram)
is a little more expensive with dd (4 seconds compared with 3
seconds). The "sgm_dd" command uses mmap() to do "zero copy" reads
which is why its CPU utilization is so low.
>From memory, dd's performance in the lk 2.4 series was considerably
lower than sg_dd. No doubt FreeBSD would also perform well but I
doubt it could beat linux (2.5) by the type of margin your measurements
indicate. [For sequential reads, tagged queueing will not have a
significant impact.] It is also worth noting that the new aic7xxx
and sym53c8xx_2 drivers are essentially the same on Linux and
FreeBSD (i.e. same code base, same maintainers).
Using scsi_debug (a ram disk) as a dummy scsi load yields:
# time dd if=/dev/sdc of=/dev/null bs=64k count=2k
2048+0 records in
2048+0 records out
real 0m1.082s user 0m0.000s sys 0m0.990s
That's 124 MB/sec and the CPU utilization is dominating. The
"sgm_dd" command yields 850 MB/sec for the same transfer.
Doug Gilbert
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2002-05-23 3:50 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2002-05-17 11:53 Linux 2.4/2.5 SCSI considerably slower than FreeBSD Matthias Andree
2002-05-19 12:38 ` Geert Uytterhoeven
2002-05-18 4:13 Douglas Gilbert
2002-05-19 20:50 ` Matthias Andree
2002-05-19 23:53 ` Tomas Szepe
2002-05-23 3:41 ` Douglas Gilbert
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).