Re: Single P4, many IDE PCI cards == trouble??

Re: Single P4, many IDE PCI cards == trouble??

[Alex Finch]

  For what it's worth, here is my experience. Hope it helps you, and 
anyone else searching for solutions to the problems I have encountered...


  Our New Server
  ==============

(+ Lessons learnt along the way)
   ++++++++++++++++++++++++++++++

Motherboard: Asus P4S8X-X

CPU:         Intel Pentium 4 2400MHz

Memory:     2 x 512Mb PC2700
  +was supposed to be 3x512Mb but the motherboard can't handle 3xPC2700
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Network:     3Com  3c905C-TX/TX-M [Tornado]

             Do NOT use the onboard LAN - performance is diabolical!
            ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Additional controller cards:

	   2xPromise Ultra133 Tx2 PCI-IDE cards
	   1xPromise Ultra66      PCI-IDE card

   Do note use 3 identical Promise IDE cards, either system will not
   +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  boot or there will be i/o errors!
  ++++++++++++++++++++++++++++

Disks
=====
  (In addition to system disk)

  10 x Maxtor 6Y060L0 60Gb
           attached to onboard ide and Promise Ultra133 cards
  4  x Maxtor 4A250J0 240Gb disk
           attached to Promise Ultra66 cards


Software:

  RedHat 9
  kernel  2.4.20

  Boot parameters modified as follows:

Edit  /boot/grub/menu.lst
and append
  ide2=noautotune ide3=noautotune ide4=noutotune ide5=noautotune
ide6=noautotune ide7=noautotune
to the line
  kernel /boot/vmlinuz-2.4.20-8 ro root=LABEL=/

  this avoids errors of the type:
  ++++++++++++++++++++++++++++

  hdh: dma_intr: bad DMA status (dma_stat=75)

  when accessing the disks on the Promise cards
					
					
  Thereafter formatted disks/created raid arrays/started raid/made 
filesystem no problem.   Four 240Gb disks formatted as one big disk via 
raid0!
  (/dev/md8 Size=989GB)			
					
Tape drive: Spectra Logic 2000 (a.k.a. TreeFrog) attached via
              Adaptec AIC-7892A SCSI card
             Use mtx version 1.2.17rel to drive the robot, more recent
            versions cause problems (author informed).
					
			

  Alex Finch, Research Fellow, Physics Department, Lancaster University.

  NB ( I am not subscribed to the list so contact me directly for more
information)

Re: Single P4, many IDE PCI cards == trouble??

[Joel Jaeggli]

On Sun, 7 Sep 2003, Mike Fedyk wrote:

> On Sat, Aug 30, 2003 at 11:46:27AM +0800, Nick Urbanik wrote:
> > It's just that I am trying to set up servers that use cheap storage at the
> > college and teach students to build them.  It looks like we still have to pay
> > big bucks for SCSI and 3ware.  And that part of my teaching will be replaced
> > with something else.  Until I understand the problem and how to solve it.
> 
> 3ware is IDE...

nor is it terribly expensive in the realm of host-bus adapters.

> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
-------------------------------------------------------------------------- 
Joel Jaeggli  	       Unix Consulting 	       joelja@darkwing.uoregon.edu    
GPG Key Fingerprint:     5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2

Re: Single P4, many IDE PCI cards == trouble??

[Mike Fedyk]

On Sat, Aug 30, 2003 at 11:46:27AM +0800, Nick Urbanik wrote:
> It's just that I am trying to set up servers that use cheap storage at the
> college and teach students to build them.  It looks like we still have to pay
> big bucks for SCSI and 3ware.  And that part of my teaching will be replaced
> with something else.  Until I understand the problem and how to solve it.

3ware is IDE...

Re: Single P4, many IDE PCI cards == trouble??

[Steve Bromwich]

Here's a belated success story from me, Nick - hope it's still relevant.

On Fri, 29 Aug 2003, Nick Urbanik wrote:

> With a single 2.26GHz P4, an Asus P4B533-E motherboard, is it possible
> to reliably use two additional PCI IDE cards (using SI680), one hard
> disk per channel, and have the thing work reliably?

It should be. I have a couple of machines I've set up for clients that
should help:

Machine 1:
3 * Maxtor 4R060L0 (RAID1, 1 hot standby)
1 * OnStream ADR drive
1 * Via IDE onboard (OnStream)
1 * onboard Promise 20265
2 * Promise 20268 PCI cards

Only 3 drives, but there's plenty of controllers. The only problem we've
had has been with the Maxtor drives, which don't seem particularly
reliable (and tech support was atrocious). This machine runs a busy
mailserver and webmail interface, and has been up for 59 days currently.
Not sure what motherboard this is, I think it's an AOpen of some flavour
(!)

Machine 2 (this is from memory, as the machine's behind a firewall):
Tyan Tiger i7505 motherboard with dual Xeon (2.4GHz, I think)
6 * WD EJB 60 gig drives
1 * Onboard IDE (PIIX4? 2 drives)
1 * Onboard Promise RAID (Disabled, doesn't work too well in non-RAID
mode)
1 * Promise Ultra TX-2
1 * generic (possibly startech?) ATA100 IDE card (I think this is an
SI680, 2 drives).

This machine is a samba file server and PDC for a call centre running a
fleet of Win2K desktops. It's been in production for 2 or 3 weeks so far
without any problems, with files on RAID5 and system on RAID1. I think
peak bonnie bench speed was around 90 megabytes a second reading blocks.

> Could this be the cause of my lockups?  I have a total of 6 ATA133
> hard disks, one DVD player all connected to the two IDE channels on
> the motherboard, and one disk to each of the channels of the SI680
> cards.

The only rules of thumb I have are "don't have any device as slave on IDE"
- this includes stuff like DVDs, CDRs and the like, and "don't share IRQs"
- it works fine but performance can suffer. I usually disable USB, serial
port 2, any onboard sound, etc on servers to claw back some more IRQs.

> Some people have told me that this is just asking for trouble, and
> that I should buy a 3ware card instead.

3Ware cards are nice, but they're also quite expensive. If you can afford
them, I'd definitely go with 3Ware.

> I am also using software RAID1, RAID5 with LVM on top.

I don't use LVM, just RAID1 and RAID5.

> My machine locks solid at unpredictable intervals with no response
> from keyboard lights, no Alt-Sysrq-x response, etc, with a wide
> variety of 2.4.x kernels, including 2.4.22.

Try disabling all unnecessary hardware in the BIOS and recompile the
kernel with the drivers for the unnecessary hardware removed (or compiled
as modules) and see how that does?

Also, have you tried qualifying the rest of the hardware to make sure it's
not a problem there? Memtest, cpuburn, etc might be worth a go...

Cheers, Steve

Re: Single P4, many IDE PCI cards == trouble??

[Nick Urbanik]

Dear Folks,

Hans-Peter Jansen wrote:

> Hi Nick,
>
> On Friday 29 August 2003 16:00, Nick Urbanik wrote:
> >
> > Performance is a relatively minor issue compared with stability and
> > low cost.
> >
> > Is there _anyone_ who is using a number of ATA133 IDE disks (>=6),
> > each on its own IDE channel, on a number of PCI IDE cards, and
> > doing so successfully and reliably?  I begin to suspect not!  If
> > so, please tell us what motherboard, IDE cards you are using.  I
> > used to imagine that a terabyte of RAID storage on one P4 machine
> > with ordinary cheap IDE cards with software RAID would be feasible.
> >  I believe it is not (although I cannot afford to play musical
> > motherboards).
>
> It is. I'm running several pretty stable systems with IDE SW RAID 5
> on top of Promise TX2/100 (~30 Eur) controllers.

Is that the PDC20270 chipset?  What motherboard are you using (i.e., I guess
you are using two IDE channels on the motherboard, so what chipset are you
using there)?  What kernel?  Are you using ide-scsi at the same time?  A DVD
player?  If so, how have you connected them?  What is the biggest number of
hard disks you are using?  What size?

I would like to try this out myself.  Thank you---this is the first success
story I have heard!

--
Nick Urbanik   RHCE                               nicku(at)vtc.edu.hk
Dept. of Information & Communications Technology
Hong Kong Institute of Vocational Education (Tsing Yi)
Tel:   (852) 2436 8576, (852) 2436 8713          Fax: (852) 2436 8526
PGP: 53 B6 6D 73 52 EE 1F EE EC F8 21 98 45 1C 23 7B     ID: 7529555D
GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24   ID: BB9D2C24

Re: Single P4, many IDE PCI cards == trouble??

[joe briggs]

I am successful in using the Tyan 2460/2466 motherboard with a 3ware 8-channel 
IDE controller and 8 WD1200 drivers, and a single WD800 IDE system drive, and 
4 bt878 frame grabbers.  

However, on a Gigabyte GA7VXP via 400 with Athlon 2400 MP with onboard Promise 
20276 controller with 2 WD1200's and a WD800 for the system drive and 4 
framegrabbers, I get IDE DMA timeouts which are a symptom for future file 
system corruption. I am using the /dev/ataraid device, and have not tried 
this configuration using Promise's driver which creates a /dev/sda1 SCSI 
device out of it. I have tried 3ware's 7000 controller with the 
kernel-included (GPL) /dev/sda driver, and it results in lower probability of 
file system corruption.

On Sunday 31 August 2003 08:43 am, Hans-Peter Jansen wrote:
> Hi Nick,
>
> On Friday 29 August 2003 16:00, Nick Urbanik wrote:
> > Performance is a relatively minor issue compared with stability and
> > low cost.
> >
> > Is there _anyone_ who is using a number of ATA133 IDE disks (>=6),
> > each on its own IDE channel, on a number of PCI IDE cards, and
> > doing so successfully and reliably?  I begin to suspect not!  If
> > so, please tell us what motherboard, IDE cards you are using.  I
> > used to imagine that a terabyte of RAID storage on one P4 machine
> > with ordinary cheap IDE cards with software RAID would be feasible.
> >  I believe it is not (although I cannot afford to play musical
> > motherboards).
>
> It is. I'm running several pretty stable systems with IDE SW RAID 5
> on top of Promise TX2/100 (~30 Eur) controllers. There was a long
> standing limit of two cards from this type, which seems to be removed
> lately (as Alan stated). At least, the test system hasn't fallen on
> it's face, when adding a third controller, and attached devices acted
> as expected without freezing, throwing DMA errors, and the like.
>
> Of course, the usual "don't buy the latest and greatest hardware,
> if the manufacturer isn't fully commited to linux support" applies.
> Promise isn't!
>
> Pete
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Joe Briggs
Briggs Media Systems
105 Burnsen Ave.
Manchester NH 01304 USA
TEL 603-232-3115 FAX 603-625-5809 MOBILE 603-493-2386
www.briggsmedia.com

Re: Single P4, many IDE PCI cards == trouble??

[Hans-Peter Jansen]

Hi Nick,

On Friday 29 August 2003 16:00, Nick Urbanik wrote:
>
> Performance is a relatively minor issue compared with stability and
> low cost.
>
> Is there _anyone_ who is using a number of ATA133 IDE disks (>=6),
> each on its own IDE channel, on a number of PCI IDE cards, and
> doing so successfully and reliably?  I begin to suspect not!  If
> so, please tell us what motherboard, IDE cards you are using.  I
> used to imagine that a terabyte of RAID storage on one P4 machine
> with ordinary cheap IDE cards with software RAID would be feasible.
>  I believe it is not (although I cannot afford to play musical
> motherboards).

It is. I'm running several pretty stable systems with IDE SW RAID 5
on top of Promise TX2/100 (~30 Eur) controllers. There was a long
standing limit of two cards from this type, which seems to be removed
lately (as Alan stated). At least, the test system hasn't fallen on 
it's face, when adding a third controller, and attached devices acted
as expected without freezing, throwing DMA errors, and the like.

Of course, the usual "don't buy the latest and greatest hardware, 
if the manufacturer isn't fully commited to linux support" applies.
Promise isn't!

Pete

Re: Single P4, many IDE PCI cards == trouble??

[Nick Urbanik]

Dear Folks,
Alan Cox wrote:

> On Gwe, 2003-08-29 at 15:00, Nick Urbanik wrote:
> > Is there _anyone_ who is using a number of ATA133 IDE disks (>=6), each on
> > its own IDE channel, on a number of PCI IDE cards, and doing so
>
> The most I know of is 8, and that was one of the people who found the
> shared IRQ/IDE race cases that 2.4.21 or so fixed.

Do you know what chipsets they were using?  I would love to know!  Any success
stories, anyone?

> > > A freeze in an IRQ handler would cause that kind of thing, turning on
> > > the NMI watchdog might get you a trace in such a failure case - and
> > > that would help.
> >
> > If the NMI count is positive in /proc/interrupts, and I have nmi_watchdog=2
> > in /proc/cmdline, does that mean that the NMI watchdog is turned on?  If
>
> nmi watchdog trigger failure would indicate hardware problems in just
> about any situation I can imagine. The nmi is just that -not maskable-
> by software.

Yes, that's what I thought.  But I've replaced every piece of hardware in the
machine except the motherboard.  If the problem persists after the 3ware 7506-8
is finally installed at considerable expense, then I'll buy an Intel
motherboard.  Does the kernel work well with the chipset on the Intel
S875WP1-E?
<http://www.intel.com/design/servers/s875wp1-e/index.htm?iid=ipp_srvr_mthrbds+s875wpie_srvr>

I guess I would disable the Promise PDC20319 IDE and stick with the 3ware.

It's just that I am trying to set up servers that use cheap storage at the
college and teach students to build them.  It looks like we still have to pay
big bucks for SCSI and 3ware.  And that part of my teaching will be replaced
with something else.  Until I understand the problem and how to solve it.

--
Nick Urbanik   RHCE                               nicku(at)vtc.edu.hk
Dept. of Information & Communications Technology
Hong Kong Institute of Vocational Education (Tsing Yi)
Tel:   (852) 2436 8576, (852) 2436 8713          Fax: (852) 2436 8526
PGP: 53 B6 6D 73 52 EE 1F EE EC F8 21 98 45 1C 23 7B     ID: 7529555D
GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24   ID: BB9D2C24

Re: Single P4, many IDE PCI cards == trouble??

[Erik Andersen]

On Fri Aug 29, 2003 at 03:38:17PM +0100, Alan Cox wrote:
> On Gwe, 2003-08-29 at 15:00, Nick Urbanik wrote:
> > Is there _anyone_ who is using a number of ATA133 IDE disks (>=6), each on
> > its own IDE channel, on a number of PCI IDE cards, and doing so
> 
> The most I know of is 8, and that was one of the people who found the
> shared IRQ/IDE race cases that 2.4.21 or so fixed.

I have a ton of drives plugged into some promise IDE cards and my
motherboard's builtin ICH5 that I use for testing things.

I have not been seeing IRQ problems.  However, when I have both
the promise and the intel IDE drivers built into the kernel,
there _is_ some sortof a race condition present, such that
stat("/", &statbuf) returns the wrong value for statbuf.st_rdev
about 50% of the time when booting.  Instead of returning the
major/minor with the correct values (/dev/hda2 on the ICH5), it
instead returns some value from one of the drives on the promise
card such as /dev/hdh or some such.  I've tried tracking that
down, but havn't been able to squash it thus far.  Grrrr.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: Single P4, many IDE PCI cards == trouble??

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

On Fri, 29 Aug 2003 22:00:58 +0800, Nick Urbanik said:

> what motherboard, IDE cards you are using.  I used to imagine that a
> terabyte of RAID storage on one P4 machine with ordinary cheap IDE cards
> with software RAID would be feasible.  I believe it is not (although I
> cannot afford to play musical motherboards).
...
> I am giving up now, and have shelled out big dollars for a 3ware 7506-8,
> which I will install early next week once I've figured out how to back up
> and restore 203GB without shelling out even more money.

Exactly.  The *real* reason why a terabyte of RAID is expensive - if it's
important enough to RAID, it's important enough to back up (with the possible
exception of RAID striping for performance on a multi-hundred-gig scratch
space).

/Valdis (who once got to clean up 600M of mess after a busticated RAID
controller dropped its cache to the right blocks on the wrong disks :)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Single P4, many IDE PCI cards == trouble??

[Alan Cox]

On Gwe, 2003-08-29 at 15:00, Nick Urbanik wrote:
> Is there _anyone_ who is using a number of ATA133 IDE disks (>=6), each on
> its own IDE channel, on a number of PCI IDE cards, and doing so

The most I know of is 8, and that was one of the people who found the
shared IRQ/IDE race cases that 2.4.21 or so fixed.

> > A freeze in an IRQ handler would cause that kind of thing, turning on
> > the NMI watchdog might get you a trace in such a failure case - and
> > that would help.
> 
> If the NMI count is positive in /proc/interrupts, and I have nmi_watchdog=2
> in /proc/cmdline, does that mean that the NMI watchdog is turned on?  If

nmi watchdog trigger failure would indicate hardware problems in just
about any situation I can imagine. The nmi is just that -not maskable-
by software.

Re: Single P4, many IDE PCI cards == trouble??

[lk]

> Is there _anyone_ who is using a number of ATA133 IDE disks (>=6), each on
> its own IDE channel, on a number of PCI IDE cards, and doing so
> successfully and reliably?  I begin to suspect not!  If so, please tell us
> what motherboard, IDE cards you are using.  I used to imagine that a
> terabyte of RAID storage on one P4 machine with ordinary cheap IDE cards
> with software RAID would be feasible.  I believe it is not (although I
> cannot afford to play musical motherboards).

We've tried it with five or six different motherboards, with Intel, Via,
and Serverworks onboard chipsets, and Promise, Highpoint, and Silicon
Image add-in cards, and had no luck.  After extensive testing and
patching, we finally gave up and bought 3ware 8500 cards.  The only
non-3ware IDE we've found to be stable is the Intel onboard chipset.  

> > > My machine locks solid at unpredictable intervals with no response
> > > from keyboard lights, no Alt-Sysrq-x response, etc, with a wide
> > > variety of 2.4.x kernels, including 2.4.22.

We generally got lost DMA messages, but still, we were unable to obtain 
any stability.

> I am giving up now, and have shelled out big dollars for a 3ware 7506-8,
> which I will install early next week once I've figured out how to back up
> and restore 203GB without shelling out even more money.

Re: Single P4, many IDE PCI cards == trouble??

[Nick Urbanik]

Dear Folks,

Alan Cox wrote:

> On Gwe, 2003-08-29 at 01:49, Nick Urbanik wrote:
> > Dear Folks,
> >
> > With a single 2.26GHz P4, an Asus P4B533-E motherboard, is it possible
> > to reliably use two additional PCI IDE cards (using SI680), one hard
> > disk per channel, and have the thing work reliably?
>
> You should be able to, although with software raid your PCI bandwidth
> limits will limit the ultimate performance for mirroring/raid

Performance is a relatively minor issue compared with stability and low
cost.

Is there _anyone_ who is using a number of ATA133 IDE disks (>=6), each on
its own IDE channel, on a number of PCI IDE cards, and doing so
successfully and reliably?  I begin to suspect not!  If so, please tell us
what motherboard, IDE cards you are using.  I used to imagine that a
terabyte of RAID storage on one P4 machine with ordinary cheap IDE cards
with software RAID would be feasible.  I believe it is not (although I
cannot afford to play musical motherboards).

> > My machine locks solid at unpredictable intervals with no response
> > from keyboard lights, no Alt-Sysrq-x response, etc, with a wide
> > variety of 2.4.x kernels, including 2.4.22.
>
> A freeze in an IRQ handler would cause that kind of thing, turning on
> the NMI watchdog might get you a trace in such a failure case - and
> that would help.

If the NMI count is positive in /proc/interrupts, and I have nmi_watchdog=2
in /proc/cmdline, does that mean that the NMI watchdog is turned on?  If
so, it never yielded anything: the machine also never responded to keyboard
lights, Alt-Sysrq-s, etc.  I have a serial terminal (minicom on an old 486
with a capture log running) as my console, since I want to catch the
culprit.  I always checked console output up to the lock up.
$ sudo grep -c 'syslogd 1.4.1: restart.' messages*
messages:13
messages.1:9
messages.2:4
messages.3:6
messages.4:4
That's a lot of locking up!
A careful look through the output of
$ sudo grep -B10 'syslogd 1.4.1: restart.' messages*
shows no output from the watchdog.

I am giving up now, and have shelled out big dollars for a 3ware 7506-8,
which I will install early next week once I've figured out how to back up
and restore 203GB without shelling out even more money.

I am honoured by your reply!   Have fun with the MBA.

--
Nick Urbanik   RHCE                               nicku(at)vtc.edu.hk
Dept. of Information & Communications Technology
Hong Kong Institute of Vocational Education (Tsing Yi)
Tel:   (852) 2436 8576, (852) 2436 8713          Fax: (852) 2436 8526
PGP: 53 B6 6D 73 52 EE 1F EE EC F8 21 98 45 1C 23 7B     ID: 7529555D
GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24   ID: BB9D2C24

Re: Single P4, many IDE PCI cards == trouble??

[Alan Cox]

On Gwe, 2003-08-29 at 01:49, Nick Urbanik wrote:
> Dear Folks,
> 
> With a single 2.26GHz P4, an Asus P4B533-E motherboard, is it possible
> to reliably use two additional PCI IDE cards (using SI680), one hard
> disk per channel, and have the thing work reliably?

You should be able to, although with software raid your PCI bandwidth
limits will limit the ultimate performance for mirroring/raid

> My machine locks solid at unpredictable intervals with no response
> from keyboard lights, no Alt-Sysrq-x response, etc, with a wide
> variety of 2.4.x kernels, including 2.4.22.

A freeze in an IRQ handler would cause that kind of thing, turning on 
the NMI watchdog might get you a trace in such a failure case - and 
that would help.

Single P4, many IDE PCI cards == trouble??

[Nick Urbanik]

Dear Folks,

With a single 2.26GHz P4, an Asus P4B533-E motherboard, is it possible
to reliably use two additional PCI IDE cards (using SI680), one hard
disk per channel, and have the thing work reliably?

Could this be the cause of my lockups?  I have a total of 6 ATA133
hard disks, one DVD player all connected to the two IDE channels on
the motherboard, and one disk to each of the channels of the SI680
cards.

Some people have told me that this is just asking for trouble, and
that I should buy a 3ware card instead.

I am also using software RAID1, RAID5 with LVM on top.

My machine locks solid at unpredictable intervals with no response
from keyboard lights, no Alt-Sysrq-x response, etc, with a wide
variety of 2.4.x kernels, including 2.4.22.

--
Nick Urbanik   RHCE                               nicku(at)vtc.edu.hk
Dept. of Information & Communications Technology
Hong Kong Institute of Vocational Education (Tsing Yi)
Tel:   (852) 2436 8576, (852) 2436 8713          Fax: (852) 2436 8526
PGP: 53 B6 6D 73 52 EE 1F EE EC F8 21 98 45 1C 23 7B     ID: 7529555D
GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24   ID: BB9D2C24