* Serial Console and Slow SCSI Disk Access?
@ 2006-06-12 21:28 Jeff Gold
2006-06-12 21:39 ` Mark Lord
0 siblings, 1 reply; 9+ messages in thread
From: Jeff Gold @ 2006-06-12 21:28 UTC (permalink / raw)
To: linux-kernel
Does it make sense that enabling a serial console would reduce the
bandwidth of a SCSI disk by more than a factor of ten? This seems crazy
to me but there's no arguing with empirical facts. I have an IBM x345
system on which I installed an unmodified 2.6.16.20 kernel built with
gcc-4.0.2-8.fc4 (the kernel configuration file I used can be found at
http://augart.com/jgold/kconfig). With the control kernel command line
I get about 70 MB/sec with hdparm -t but when I add "console=ttyS0,9600
console=tty0" I get about 1.6 MB/sec instead.
Here are the complete kernel command lines from grub.conf (the second
one doesn't actually include a backslash in grub.conf -- that's just to
avoid email wrapping):
ro root=/dev/sda3 pci=biosirq loglevel=7
ro root=/dev/sda3 pci=biosirq console=ttyS0,9600n8 console=tty0 \
loglevel=7
I have not been able to connect this behavior to any known kernel bugs
using Google or mailing list archives. What might I be doing wrong?
Jeff
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Serial Console and Slow SCSI Disk Access?
2006-06-12 21:28 Serial Console and Slow SCSI Disk Access? Jeff Gold
@ 2006-06-12 21:39 ` Mark Lord
2006-06-12 22:04 ` Jeff Gold
0 siblings, 1 reply; 9+ messages in thread
From: Mark Lord @ 2006-06-12 21:39 UTC (permalink / raw)
To: Jeff Gold; +Cc: linux-kernel
Jeff Gold wrote:
> Does it make sense that enabling a serial console would reduce the
> bandwidth of a SCSI disk by more than a factor of ten? This seems crazy
> to me but there's no arguing with empirical facts. I have an IBM x345
> system on which I installed an unmodified 2.6.16.20 kernel built with
> gcc-4.0.2-8.fc4 (the kernel configuration file I used can be found at
> http://augart.com/jgold/kconfig). With the control kernel command line
> I get about 70 MB/sec with hdparm -t but when I add "console=ttyS0,9600
> console=tty0" I get about 1.6 MB/sec instead.
This can happen if there are kernel messages being printed on the serial console.
If all is quiet, I would expect things to be as fast as normal elsewhere.
-ml
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Serial Console and Slow SCSI Disk Access?
2006-06-12 21:39 ` Mark Lord
@ 2006-06-12 22:04 ` Jeff Gold
2006-06-19 6:50 ` Helge Hafting
0 siblings, 1 reply; 9+ messages in thread
From: Jeff Gold @ 2006-06-12 22:04 UTC (permalink / raw)
To: Mark Lord; +Cc: linux-kernel
Mark Lord wrote:
> This can happen if there are kernel messages being printed on the serial
> console.
> If all is quiet, I would expect things to be as fast as normal elsewhere.
Thank you for the suggestion. I don't see much in /var/log/messages
(syslogd is running). There are 3326 lines taking up about 256 kB
there, and when I run hdparm runs no further messages are generated.
I don't have anything attached to the serial port at the moment. Could
that cause problems? I'm going to attach something and see what
happens. Other advice is still welcome.
Jeff
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Serial Console and Slow SCSI Disk Access?
2006-06-12 22:04 ` Jeff Gold
@ 2006-06-19 6:50 ` Helge Hafting
2006-06-19 15:14 ` Jeff Gold
0 siblings, 1 reply; 9+ messages in thread
From: Helge Hafting @ 2006-06-19 6:50 UTC (permalink / raw)
To: Jeff Gold; +Cc: Mark Lord, linux-kernel
Jeff Gold wrote:
> Mark Lord wrote:
>> This can happen if there are kernel messages being printed on the
>> serial console.
>> If all is quiet, I would expect things to be as fast as normal
>> elsewhere.
>
> Thank you for the suggestion. I don't see much in /var/log/messages
> (syslogd is running). There are 3326 lines taking up about 256 kB
> there, and when I run hdparm runs no further messages are generated.
>
> I don't have anything attached to the serial port at the moment.
> Could that cause problems? I'm going to attach something and see what
> happens. Other advice is still welcome.
With nothing attached, any write to the serial device might go through
a lengthy timeout because of flow control. I'd consider that a bug
in this case though, and there is usually no console printout
per scsi disk access either.
I would not be surprised if your serial console causes a longer boot time,
as all boot messages have to be transferred over the slow serial link
or in the worst case timed out one message at a time.
But I can't see why it'd make scsi disks slower. The scsi host adapter
initialization
writes some messages of course, but there should be no more console accesses
during a hdparm test run.
Helge Hafting
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Serial Console and Slow SCSI Disk Access?
2006-06-19 6:50 ` Helge Hafting
@ 2006-06-19 15:14 ` Jeff Gold
2006-06-20 7:11 ` Helge Hafting
0 siblings, 1 reply; 9+ messages in thread
From: Jeff Gold @ 2006-06-19 15:14 UTC (permalink / raw)
To: Helge Hafting; +Cc: Mark Lord, linux-kernel
Helge Hafting wrote:
> With nothing attached, any write to the serial device might go
> through a lengthy timeout because of flow control. [...] But I can't
> see why it'd make scsi disks slower. The scsi host adapter
> initialization writes some messages of course, but there should be no
> more console accesses during a hdparm test run.
This makes sense to me. When I attach a serial cable and use that to
login (I've got agetty running), hdparm produces no console messages
that I can see using minicom. Still, the disk throughput is around 1.5
MB/sec for some reason. When I disable the serial console in grub.conf
and reboot I get over 70 MB/sec again.
A combination of out-of-tree patches (mainly network related but also
one to disable PM_TIMERS) seem to eliminate the issue even with the
serial console enabled, at least for the moment. That means I no longer
have a problem, but the whole thing is mysterious to me.
Jeff
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Serial Console and Slow SCSI Disk Access?
2006-06-19 15:14 ` Jeff Gold
@ 2006-06-20 7:11 ` Helge Hafting
2006-06-20 16:20 ` Jeff Gold
2006-06-22 0:32 ` David Lang
0 siblings, 2 replies; 9+ messages in thread
From: Helge Hafting @ 2006-06-20 7:11 UTC (permalink / raw)
To: Jeff Gold; +Cc: Mark Lord, linux-kernel
Jeff Gold wrote:
> Helge Hafting wrote:
>> With nothing attached, any write to the serial device might go
>> through a lengthy timeout because of flow control. [...] But I can't
>> see why it'd make scsi disks slower. The scsi host adapter
>> initialization writes some messages of course, but there should be no
>> more console accesses during a hdparm test run.
>
> This makes sense to me. When I attach a serial cable and use that to
> login (I've got agetty running), hdparm produces no console messages
> that I can see using minicom. Still, the disk throughput is around
> 1.5 MB/sec for some reason. When I disable the serial console in
> grub.conf and reboot I get over 70 MB/sec again.
>
> A combination of out-of-tree patches (mainly network related but also
> one to disable PM_TIMERS) seem to eliminate the issue even with the
> serial console enabled, at least for the moment. That means I no
> longer have a problem, but the whole thing is mysterious to me.
I can see one possibility, that I didn't think of yesterday.
Do the scsi host adapter share its interrupt with the serial line?
(Boot a kernel that has the problem, and when scsi is slow, do a
cat /proc/interrupts
If the scsi and the serial driver share an interrupt, then that is the
source
of the problem.
Linux can share interrupts without big performance problems - IF both of
the hardware drivers are written with interrupt sharing in mind. Many
linux drivers are, but some are not. So check if interrupt sharing is
happening,
and if so, contact the maintainers of your scsi driver and your serial
port driver. Ask them if such sharing is ok or not. If you don't
understand the /proc/interrupt listing, just send it to the
maintainers so they can have a look.
Shared interrupts are to some extent set up by the bios, and to some
extent by linux. So different kernel versions (or booting with a different
set of drivers loaded, or a different distribution) may make a difference.
You can often get a pci device to use a different interrupt by moving it
to another slot. That tends to solve interrupt sharing problems. I assume
your scsi adapter is pci.
Helge Hafting
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Serial Console and Slow SCSI Disk Access?
2006-06-20 7:11 ` Helge Hafting
@ 2006-06-20 16:20 ` Jeff Gold
2006-06-22 0:32 ` David Lang
1 sibling, 0 replies; 9+ messages in thread
From: Jeff Gold @ 2006-06-20 16:20 UTC (permalink / raw)
To: Helge Hafting; +Cc: Mark Lord, linux-kernel
Helge Hafting wrote:
> I can see one possibility, that I didn't think of yesterday.
> Do the scsi host adapter share its interrupt with the serial line?
That makes a lot of sense. I'll check it out and follow your
instructions as soon as I can. Thanks.
Jeff
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Serial Console and Slow SCSI Disk Access?
2006-06-20 7:11 ` Helge Hafting
2006-06-20 16:20 ` Jeff Gold
@ 2006-06-22 0:32 ` David Lang
2006-06-22 16:18 ` Andrey Melnikoff
1 sibling, 1 reply; 9+ messages in thread
From: David Lang @ 2006-06-22 0:32 UTC (permalink / raw)
To: Helge Hafting; +Cc: Jeff Gold, Mark Lord, linux-kernel
On Tue, 20 Jun 2006, Helge Hafting wrote:
> Jeff Gold wrote:
>> Helge Hafting wrote:
>>> With nothing attached, any write to the serial device might go
>>> through a lengthy timeout because of flow control. [...] But I can't
>>> see why it'd make scsi disks slower. The scsi host adapter initialization
>>> writes some messages of course, but there should be no
>>> more console accesses during a hdparm test run.
>>
>> This makes sense to me. When I attach a serial cable and use that to login
>> (I've got agetty running), hdparm produces no console messages that I can
>> see using minicom. Still, the disk throughput is around 1.5 MB/sec for
>> some reason. When I disable the serial console in grub.conf and reboot I
>> get over 70 MB/sec again.
>>
>> A combination of out-of-tree patches (mainly network related but also one
>> to disable PM_TIMERS) seem to eliminate the issue even with the serial
>> console enabled, at least for the moment. That means I no longer have a
>> problem, but the whole thing is mysterious to me.
> I can see one possibility, that I didn't think of yesterday.
> Do the scsi host adapter share its interrupt with the serial line?
> (Boot a kernel that has the problem, and when scsi is slow, do a
> cat /proc/interrupts
> If the scsi and the serial driver share an interrupt, then that is the source
> of the problem.
>
> Linux can share interrupts without big performance problems - IF both of
> the hardware drivers are written with interrupt sharing in mind. Many
> linux drivers are, but some are not. So check if interrupt sharing is
> happening,
> and if so, contact the maintainers of your scsi driver and your serial
> port driver. Ask them if such sharing is ok or not. If you don't
> understand the /proc/interrupt listing, just send it to the
> maintainers so they can have a look.
>
> Shared interrupts are to some extent set up by the bios, and to some
> extent by linux. So different kernel versions (or booting with a different
> set of drivers loaded, or a different distribution) may make a difference.
>
> You can often get a pci device to use a different interrupt by moving it
> to another slot. That tends to solve interrupt sharing problems. I assume
> your scsi adapter is pci.
>
I ran into a similar problem (extremely slow scsi performance), but hadn't
reported it as I made a lot of changes at the same time (moved from 2.6.7 to
2.6.12.3 as well as enabling the serial console). but with this report I wanted
to chime in with a weak 'me-too'. I haven't rebooted without the serial console
yet to confirm this, but I will do so shortly, and try 2.6.17 to see if it has a
similar problem or not.
David Lang
# cat /proc/interrupts
CPU0 CPU1
0: 458865186 29148 IO-APIC-edge timer
1: 216 7 IO-APIC-edge i8042
3: 78 1 IO-APIC-edge serial
4: 76 1 IO-APIC-edge serial
8: 8 1 IO-APIC-edge rtc
9: 0 0 IO-APIC-level acpi
12: 84 10 IO-APIC-edge i8042
14: 332 10 IO-APIC-edge ide0
24: 4426414 1 IO-APIC-level eth2
26: 426930991 1 IO-APIC-level eth0
27: 17795 365692411 IO-APIC-level eth1
29: 553120 87 IO-APIC-level ioc0
NMI: 0 0
LOC: 458903065 458904855
ERR: 0
MIS: 0
# lspci
0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03)
0000:00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02)
0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev
12)
0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev
12)
0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:01:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
0000:02:02.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 02)
0000:02:03.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5702X
Gigabit Ethernet (rev 02)
0000:02:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5702X
Gigabit Ethernet (rev 02)
0000:03:04.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet
Controller (rev 01)
0000:03:04.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet
Controller (rev 01)
0000:03:06.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet
Controller (rev 01)
0000:03:06.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet
Controller (rev 01)
0000:04:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X
Fusion-MPT Dual Ultra320 SCSI (rev 07)
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Serial Console and Slow SCSI Disk Access?
2006-06-22 0:32 ` David Lang
@ 2006-06-22 16:18 ` Andrey Melnikoff
0 siblings, 0 replies; 9+ messages in thread
From: Andrey Melnikoff @ 2006-06-22 16:18 UTC (permalink / raw)
To: linux-kernel
David Lang <dlang@digitalinsight.com> wrote:
> On Tue, 20 Jun 2006, Helge Hafting wrote:
> > Jeff Gold wrote:
> >> Helge Hafting wrote:
> >>> With nothing attached, any write to the serial device might go
> >>> through a lengthy timeout because of flow control. [...] But I can't
> >>> see why it'd make scsi disks slower. The scsi host adapter initialization
> >>> writes some messages of course, but there should be no
> >>> more console accesses during a hdparm test run.
> >>
> >> This makes sense to me. When I attach a serial cable and use that to login
> >> (I've got agetty running), hdparm produces no console messages that I can
> >> see using minicom. Still, the disk throughput is around 1.5 MB/sec for
> >> some reason. When I disable the serial console in grub.conf and reboot I
> >> get over 70 MB/sec again.
> >>
> >> A combination of out-of-tree patches (mainly network related but also one
> >> to disable PM_TIMERS) seem to eliminate the issue even with the serial
> >> console enabled, at least for the moment. That means I no longer have a
> >> problem, but the whole thing is mysterious to me.
> > I can see one possibility, that I didn't think of yesterday.
> > Do the scsi host adapter share its interrupt with the serial line?
> > (Boot a kernel that has the problem, and when scsi is slow, do a
> > cat /proc/interrupts
> > If the scsi and the serial driver share an interrupt, then that is the source
> > of the problem.
[skipp]
> I ran into a similar problem (extremely slow scsi performance), but hadn't
> reported it as I made a lot of changes at the same time (moved from 2.6.7 to
> 2.6.12.3 as well as enabling the serial console). but with this report I wanted
> to chime in with a weak 'me-too'. I haven't rebooted without the serial console
> yet to confirm this, but I will do so shortly, and try 2.6.17 to see if it has a
> similar problem or not.
2.6.17.1 here, MB Intel SCB20, aic7899 onboard. aic7xxx compilled in, serial
core compilled in, serial console enabled.
With/without serial sonsole - speed 66.0 Mb/s.
Maybe problem related to MB chipset?
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2006-06-22 16:31 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-06-12 21:28 Serial Console and Slow SCSI Disk Access? Jeff Gold
2006-06-12 21:39 ` Mark Lord
2006-06-12 22:04 ` Jeff Gold
2006-06-19 6:50 ` Helge Hafting
2006-06-19 15:14 ` Jeff Gold
2006-06-20 7:11 ` Helge Hafting
2006-06-20 16:20 ` Jeff Gold
2006-06-22 0:32 ` David Lang
2006-06-22 16:18 ` Andrey Melnikoff
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).