All filesystems hang under long periods of heavy load (read and write) on a filesystem

All filesystems hang under long periods of heavy load (read and write) on a filesystem

[Shirley Shi]

Can anyone know why all filesystems hang under periods of heavy load on 
one of the filesystem? Once the filesystems hang, any command related to 
the filesystem, like 'ls', 'cat',etc., will stick forever until re-power 
cycling the machine.

I kept running the following script to read and write the data on a same 
filesystem(ext2 or XFS) since we need do some tests for the storage. Is 
half day, onn the beginning, the system was running well. But after 
running the script for a long time, such a half day, one day or two 
days,  all filesystems would get hung, including the root filesystem 
although I didn't do any heavy load on it. The file(M.1) I used for 
reading and writing is about 2.5GB.


@ total = 115
while (1)
  @ cc = 2
  while ($cc <= $total)
     dd bs=512k if=/data/M.1 of=/data/M.$cc
    echo "copying $cc  of   $total..."
    @ cc = $cc + 1
  end
  rm -f  /data/M.*
end


I tried RH8.0 with kernel 2.4.18 and kernel 2.4.21 with XFS and patch 
rmap15j. I have the same issue running with the two kernels. Basically I 
have two filesytems configured. One for the root configured with ext3, 
and another is for the data configured with ext2 or XFS. With either 
ext2 or XFS, I have the same problem.

I also tried on different Dual CPU machines as follows, but saw the same 
problem.

- A dual-CPU machine with an on-board 320 SCSI controller(running 
AIC79XX.o driver) connecting a disk drive as the root system and a 
MegaRAID 320-4X controller connecting to several disk drives. I created 
two H/W logical RAID devices and built the data filesystem with the S/W 
RAID0(/dev/md0).
- A HP dual-CPU machine with the HP Smart Array 5i connecting to a disk 
drive as the root system and a 160 SCSI controller connecting with 13 
disk drives and configured as a S/W RAID0(/dev/md0) the data filesystem.
- A dual-CPU machine with an on-board 320 SCSI controller(with AIC79XX.o 
driver) connecting a disk drive as the root system and a FC controller 
connecting to a RAID storage.

Any comment would be appreciated.

Thanks,

Shirley

Re: All filesystems hang under long periods of heavy load (read and write) on a filesystem

[bert hubert]

On Mon, Nov 03, 2003 at 03:46:22PM -0800, Shirley Shi wrote:
> Can anyone know why all filesystems hang under periods of heavy load on 
> one of the filesystem? Once the filesystems hang, any command related to 
> the filesystem, like 'ls', 'cat',etc., will stick forever until re-power 
> cycling the machine.

I suggest you figure out what your systems have in common, if this were
universal people would've noticed by now. If you have such a hang again, can
you show us the output of 'dmesg' and 'ps aux'? If at all possible, can you
run ShowTasks from the magic SysRQ menu?

Good luck!


-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: All filesystems hang under long periods of heavy load (read and write) on a filesystem

[Bernd Petrovitsch]

On Die, 2003-11-04 at 00:46, Shirley Shi wrote:
> filesystem(ext2 or XFS) since we need do some tests for the storage. Is 
> half day, onn the beginning, the system was running well. But after 
> running the script for a long time, such a half day, one day or two 
> days,  all filesystems would get hung, including the root filesystem 
> although I didn't do any heavy load on it. The file(M.1) I used for 
> reading and writing is about 2.5GB.

Wild guess: Perhaps a memory leak somewhere in the kernel and it shows
up after $BIG_NUMBER of operations?

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: All filesystems hang under long periods of heavy load (read and write) on a filesystem

[Randy.Dunlap]

On Mon, 03 Nov 2003 15:46:22 -0800 Shirley Shi <shirley@kasenna.com> wrote:

| Can anyone know why all filesystems hang under periods of heavy load on 
| one of the filesystem? Once the filesystems hang, any command related to 
| the filesystem, like 'ls', 'cat',etc., will stick forever until re-power 
| cycling the machine.
| 
| I kept running the following script to read and write the data on a same 
| filesystem(ext2 or XFS) since we need do some tests for the storage. Is 
| half day, onn the beginning, the system was running well. But after 
| running the script for a long time, such a half day, one day or two 
| days,  all filesystems would get hung, including the root filesystem 
| although I didn't do any heavy load on it. The file(M.1) I used for 
| reading and writing is about 2.5GB.
| 
| 
| @ total = 115
| while (1)
|   @ cc = 2
|   while ($cc <= $total)
|      dd bs=512k if=/data/M.1 of=/data/M.$cc
|     echo "copying $cc  of   $total..."
|     @ cc = $cc + 1
|   end
|   rm -f  /data/M.*
| end
| 
| 
| I tried RH8.0 with kernel 2.4.18 and kernel 2.4.21 with XFS and patch 
| rmap15j. I have the same issue running with the two kernels. Basically I 
| have two filesytems configured. One for the root configured with ext3, 
| and another is for the data configured with ext2 or XFS. With either 
| ext2 or XFS, I have the same problem.

Can you try a recent kernel, like 2.4.23-pre8 or -pre9?

--
~Randy

Re: All filesystems hang under long periods of heavy load (read and write) on a filesystem

[Ged Haywood]

Hi Marcelo,

On Mon, 29 Dec 2003, Marcelo Tosatti wrote:

> > there is certainly something nasty in the released 2.4.23 IDE code...
> 
> The attached patch from Daniel Lux should fix it. Its has just been
> applied to the 2.4 BK tree.
> 
> Please try it.

Thanks very much for the patch, I can see how that piece of code could
well be causing the problem.

Between my message and your reply, someone else came up with another
clue for me and I found that enabling DMA for the drive appears if not
to fix the problem at least to avoid it fairly well.  I think using
DMA for most of the transfers makes it less likely that the lockup
will be triggered.  At least this gets me out of the immediate bind.

It's a busy machine and it's a little painful to keep rebooting it.
I'm going to put an identical one together for testing.  When I've
done that I'll test the patch to destruction.

It will be a couple of weeks before I can get to the bench to do it,
but then it should only take a few hours to be sure it's a good fix.
I'll certainly get back to you with the results.

Thanks again for the pointer.

73,
Ged.

Re: All filesystems hang under long periods of heavy load (read and write) on a filesystem

[Marcelo Tosatti]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1103 bytes --]



On Tue, 23 Dec 2003, Ged Haywood wrote:

> Googled lots of similar postings, if anyone wants me to try anything
> out please let me know.  Otherwise this is just a heads up to say that
> there is certainly something nasty in the released 2.4.23 IDE code...

<snip>

> Dec 23 12:19:01 www2 kernel: hdb: status timeout: status=0xd0 { Busy }
> Dec 23 12:19:01 www2 kernel:  Dec 23 12:19:01 www2 kernel:
> hdb: no DRQ after issuing WRITE Dec 23 12:19:33 www2 kernel: ide0: reset timed-out,
> status=0x80 Dec 23 12:19:33 www2 kernel: hdb: status timeout:
> status=0xd0 { Busy } Dec 23 12:19:33 www2 kernel:  Dec 23 12:19:33 www2
> kernel: hdb: no DRQ after issuing WRITE Dec 23 12:20:03 www2 kernel:
> ide0: reset timed-out, status=0x80 Dec 23 12:20:03 www2 kernel:
> end_request: I/O error, dev 03:41 (hdb), sector 35289088 ... #

Hi Ged,

I believe this migh caused by a problem in ide_wait where the kernel
thinks a timeout has happened but it hasnt in practice (the drive is ready
to go).

The attached patch from Daniel Lux should fix it. Its has just been
applied to the 2.4 BK tree.

Please try it.

[-- Attachment #2: Type: TEXT/PLAIN, Size: 1120 bytes --]

===== drivers/ide/ide-iops.c 1.7 vs 1.9 =====
--- 1.7/drivers/ide/ide-iops.c	Sat Aug  9 11:48:39 2003
+++ 1.9/drivers/ide/ide-iops.c	Sun Dec 28 18:14:05 2003
@@ -664,12 +664,22 @@
 	if ((stat = hwif->INB(IDE_STATUS_REG)) & BUSY_STAT) {
 		local_irq_set(flags);
 		timeout += jiffies;
-		while ((stat = hwif->INB(IDE_STATUS_REG)) & BUSY_STAT) {
+		stat = hwif->INB(IDE_STATUS_REG);
+		while (stat & BUSY_STAT) {
 			if (time_after(jiffies, timeout)) {
-				local_irq_restore(flags);
-				*startstop = DRIVER(drive)->error(drive, "status timeout", stat);
-				return 1;
+				/*
+			       	 * do one more status read in case we were interrupted between last
+				 * stat = hwif->INB(IDE_STATUS_REG) and time_after(jiffies, timeout)
+				 * in wich case the timeout might have been shorter than specified.
+				 */
+				if ((stat = hwif->INB(IDE_STATUS_REG)) & BUSY_STAT) {
+					local_irq_restore(flags);
+					*startstop = DRIVER(drive)->error(drive, "status timeout", stat);
+					return 1;
+				}
 			}
+			else
+				stat = hwif->INB(IDE_STATUS_REG);
 		}
 		local_irq_restore(flags);
 	}

Re: All filesystems hang under long periods of heavy load (read and write) on a filesystem

[Ged Haywood]

Hello again all,

On Tue, 23 Dec 2003, Ged Haywood wrote:
[snip,snip]
> On Tue, 04 Nov 2003 Randy Dunlap wrote:
> > 
> > Can you try a recent kernel, like 2.4.23-pre8 or -pre9?
> 
> Copying large files from the smaller IDE drive to the larger trashed
> the filesystem on the partition being written to.

It seems that this boils down to using PIO mode, after enabling DMA I am
able to write 1.7G files with no trouble - of course it's early days yet,
the fault might not be fixed, it might just be less obvious...

Enabling DMA hadn't been top priority until I spoke to the 'hdparm'
maintainer at Debian.  The driver for the CMD Technology Inc CMD680 is
cunningly hidden under "Silicon Image chipset support" in the 2.4.23
config menus but was fairly easy to find once I started digging.  :)

Apparently the problem might still be present in 2.6.0 - see for example

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/must-fix/should-fix-7.txt

Anyway I'm thinking I'll put together an identical machine in the next
few weeks so I'll be able to reproduce the fault at will.  Would anyone
be interested in playing with it?

73,
Ged.

PS: Not subscribed, please CC me if you're interested.

PPS: Keeping the same subject line, but maybe this deserves a new one?

Re: All filesystems hang under long periods of heavy load (read and write) on a filesystem

[Ged Haywood]

Hi all,

On Tue, 04 Nov 2003 Randy Dunlap wrote:

> On Mon, 03 Nov 2003 15:46:22 -0800 Shirley Shi wrote:
> 
> | Can anyone know why all filesystems hang under periods of heavy load on
> | one of the filesystem? Once the filesystems hang, any command related to
> | the filesystem, like 'ls', 'cat',etc., will stick forever until re-power
> | cycling the machine.
> 
> Can you try a recent kernel, like 2.4.23-pre8 or -pre9?

Didn't see any followup to this.  Please CC me, I'm not subscribed.

Background:

Similar problems here, both kernels 2.2.16 and 2.4.23, the latter
compiled with the config abbreviated below, on a modest Webserver.

This is a Compaq DL380 with SCSI RAID and two IDE drives.  No problems
at all on the SCSI drives.  The IDE drives are 40GB Maxtor, 250GB WD,
one is on the DL380's own IDE controller, the other is on a PCI card.
One partition on the 40G drive, three (each about 80GB) on the 250.
The interrupts are not shared.  Both drives are jumpered as slaves,
there is a CD as master on ide0 but no master on ide2 (yeah, OK,
but that one's all right...:).  All filesystems ext2.  The IDE drives
are used primarily for some data logging (not system logs) and backup.

The 40GB Maxtor has given no trouble until today, the machine hadn't
even been rebooted for two years until the end of November when I put
in the 250G IDE drive.  It was still running kernel 2.2.16 at the time.
Neither kernel loads any modules, eveything needed is compiled in.

Fault:

Copying large files from the smaller IDE drive to the larger trashed
the filesystem on the partition being written to.  Before installation
in the Webserver the same operations had been carried out error-free
on a PC on the bench to stress test the drives.

When the fault appeared the machine was running 2.2.16, but the bench
test had been with 2.4.23 so I compiled 2.4.23 on the Webserver to see
if it would help.  Not a lot, read on...

When I compiled with CONFIG_IDEDISK_MULTI_MODE not set and tried to
copy a large (2GB) file from the smaller IDE drive to the larger, it
trashed the filesystem on the drive I was writing to.  This fault is
apparently fixed by compiling with CONFIG_IDEDISK_MULTI_MODE=y although
there was one other change which I think will have made no difference -
I specified IRQ11 explicitly in the LILO boot parameters.

Unfortuantely now when copying the same large file from the larger IDE
drive to the smaller the filesystem on the drive being READ is trashed
when another process tries to write to that same drive concurrently.
This also prevents any access to the drive until a reboot.

Googled lots of similar postings, if anyone wants me to try anything
out please let me know.  Otherwise this is just a heads up to say that
there is certainly something nasty in the released 2.4.23 IDE code...

73,
Ged.

=============================================================================
Extract from LILO config:

image=/boot/bzImage.2.4.23.004
	label=linux-2.4.23-d
	read-only
	root=/dev/ida/c0d0p1
	append=" ide2=0x2c00,0x2c0a,11 ide3=0x2c10,0x2c1a,11"

=============================================================================
Extracts from /var/log/messages
=============================================================================
Note that the LILO boot parameters are somehow truncated.  It shouldn't matter
on this machine since ide3 is not being used...
...
Dec 22 14:48:08 www2 kernel: Linux version 2.4.23 (ged@www2) (gcc version egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)) #4 Fri Dec 19 20:48:47 GMT 2003 
...
Dec 22 14:48:08 www2 kernel: Kernel command line: auto BOOT_IMAGE=linux-2.4.23-d ro root=4801 ide2=0x2c00,0x2c0a,11 ide3=0x2c10, 
Dec 22 14:48:08 www2 kernel: ide_setup: ide2=0x2c00,0x2c0a,11 
Dec 22 14:48:08 www2 kernel:  
Dec 22 14:48:08 www2 kernel: ide_setup: ide3=0x2c10, 
Dec 22 14:48:08 www2 kernel:  
Dec 22 14:48:08 www2 kernel: Initializing CPU#0 
Dec 22 14:48:08 www2 kernel: Detected 731.081 MHz processor. 
...
Dec 22 14:48:09 www2 kernel: PCI: PCI BIOS revision 2.10 entry at 0xf0084, last bus=3 
Dec 22 14:48:09 www2 kernel: PCI: Using configuration type 1 
Dec 22 14:48:09 www2 kernel: PCI: Probing PCI hardware 
Dec 22 14:48:09 www2 kernel: PCI: Probing PCI hardware (bus 00) 
Dec 22 14:48:09 www2 kernel: PCI: Device 00:00 not found by BIOS 
Dec 22 14:48:09 www2 kernel: PCI: Device 00:01 not found by BIOS 
Dec 22 14:48:09 www2 kernel: PCI: Device 00:78 not found by BIOS 
...
Dec 22 14:48:11 www2 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4 
Dec 22 14:48:11 www2 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx 
Dec 22 14:48:11 www2 kernel: hda: CD-224E, ATAPI CD/DVD-ROM drive 
Dec 22 14:48:11 www2 kernel: hdb: Maxtor 94098U8, ATA DISK drive 
Dec 22 14:48:11 www2 kernel: hdf: WDC WD2500JB-00EVA0, ATA DISK drive 
Dec 22 14:48:11 www2 kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 
Dec 22 14:48:11 www2 kernel: ide2 at 0x2c00-0x2c07,0x2c0a on irq 11 
Dec 22 14:48:11 www2 kernel: hdb: attached ide-disk driver. 
Dec 22 14:48:11 www2 kernel: hdb: host protected area => 1 
Dec 22 14:48:11 www2 kernel: hdb: 80041248 sectors (40981 MB) w/2048KiB Cache, CHS=79406/16/63 
Dec 22 14:48:11 www2 kernel: hdf: attached ide-disk driver. 
Dec 22 14:48:11 www2 kernel: hdf: host protected area => 1 
Dec 22 14:48:11 www2 kernel: hdf: 488397168 sectors (250059 MB) w/8192KiB Cache, CHS=30401/255/63 
Dec 22 14:48:11 www2 kernel: hda: attached ide-cdrom driver. 
Dec 22 14:48:11 www2 kernel: hda: ATAPI 24X CD-ROM drive, 512kB Cache 
Dec 22 14:48:11 www2 kernel: Uniform CD-ROM driver Revision: 3.12 
Dec 22 14:48:11 www2 kernel:  hdb: [PTBL] [4982/255/63] hdb1 
Dec 22 14:48:11 www2 kernel:  hdf: hdf1 hdf2 hdf3 hdf4 
...
Dec 23 12:19:01 www2 kernel: hdb: status timeout: status=0xd0 { Busy } 
Dec 23 12:19:01 www2 kernel:  
Dec 23 12:19:01 www2 kernel: hdb: no DRQ after issuing WRITE 
Dec 23 12:19:33 www2 kernel: ide0: reset timed-out, status=0x80 
Dec 23 12:19:33 www2 kernel: hdb: status timeout: status=0xd0 { Busy } 
Dec 23 12:19:33 www2 kernel:  
Dec 23 12:19:33 www2 kernel: hdb: no DRQ after issuing WRITE 
Dec 23 12:20:03 www2 kernel: ide0: reset timed-out, status=0x80 
Dec 23 12:20:03 www2 kernel: end_request: I/O error, dev 03:41 (hdb), sector 35289088 
...

At this stage everything is hunky-dory. :)
=============================================================================
cat /proc/interrupts:

(This is after /dev/hdb failed.)

           CPU0       
  0:    9643877          XT-PIC  timer
  1:          4          XT-PIC  keyboard
  2:          0          XT-PIC  cascade
  5:     525270          XT-PIC  ida0
 10:    1224738          XT-PIC  eth0
 11:    1506346          XT-PIC  ide2
 12:          0          XT-PIC  PS/2 Mouse
 14:     955863          XT-PIC  ide0
NMI:          0 
ERR:       1412
=============================================================================
Extracts from kernel 2.4.23 .config:
=============================================================================
CONFIG_X86=y
# CONFIG_EXPERIMENTAL is not set
CONFIG_MPENTIUMIII=y
# CONFIG_SMP is not set
# CONFIG_X86_UP_APIC is not set
# CONFIG_X86_UP_IOAPIC is not set
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_ISA=y
CONFIG_PCI_NAMES=y
# CONFIG_HOTPLUG_PCI is not set
# CONFIG_HOTPLUG_PCI_COMPAQ is not set
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not set
# CONFIG_ACPI is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID5=m
CONFIG_MD_MULTIPATH=m
CONFIG_BLK_DEV_LVM=m
#
# ATA/IDE/MFM/RLL support
#
CONFIG_IDE=y

#
# IDE, ATA and ATAPI Block devices
#
CONFIG_BLK_DEV_IDE=y
# CONFIG_BLK_DEV_HD_IDE is not set
# CONFIG_BLK_DEV_HD is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
# CONFIG_IDEDISK_STROKE is not set
# CONFIG_BLK_DEV_IDECS is not set
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set
CONFIG_BLK_DEV_CMD640=y
# CONFIG_BLK_DEV_CMD640_ENHANCED is not set
# CONFIG_BLK_DEV_ISAPNP is not set
CONFIG_BLK_DEV_IDEPCI=y
# CONFIG_BLK_DEV_GENERIC is not set
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
CONFIG_BLK_DEV_IDEDMA=y

CONFIG_BLK_DEV_PIIX=y
CONFIG_BLK_DEV_RZ1000=y
CONFIG_IDEDMA_AUTO=y
CONFIG_BLK_DEV_IDE_MODES=y

#
# Kernel hacking
#
# CONFIG_DEBUG_KERNEL is not set
CONFIG_LOG_BUF_SHIFT=0
=============================================================================
hdparm -vi /dev/hdb
(This drive is not working; MultSect is normally 16).

/dev/hdb:
 multcount    =  0 (off)
 I/O support  =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  0 (off)
 keepsettings =  0 (off)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)
 geometry     = 4982/255/63, sectors = 80041248, start = 0

 Model=Maxtor 94098U8, FwRev=FA500S60, SerialNo=G80CWGYC
 Config={ Fixed }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=57
 BuffType=3(DualPortCache), BuffSize=2048kB, MaxMultSect=16, MultSect=off
 DblWordIO=no, OldPIO=2, DMA=yes, OldDMA=0
 CurCHS=16383/16/63, CurSects=-66060037, LBA=yes, LBAsects=80041248
 tDMA={min:120,rec:120}, DMA modes: mword0 mword1 *mword2 
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4 
 UDMA modes: mode0 mode1 mode2 mode3 mode4 
=============================================================================
hdparm /dev/hdf
(This drive is working.)

/dev/hdf:
 multcount    = 16 (on)
 I/O support  =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  0 (off)
 keepsettings =  0 (off)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)
 geometry     = 30401/255/63, sectors = 488397168, start = 0

 Model=WDC WD2500JB-00EVA0, FwRev=15.05R15, SerialNo=WD-WMAEH1325940
 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
 RawCHS=16383/16/63, TrkSize=57600, SectSize=600, ECCbytes=74
 BuffType=3(DualPortCache), BuffSize=8192kB, MaxMultSect=16, MultSect=16
 DblWordIO=no, OldPIO=2, DMA=yes, OldDMA=0
 CurCHS=16383/16/63, CurSects=-66060037, LBA=yes, LBAsects=268435455
 tDMA={min:120,rec:120}, DMA modes: mword0 mword1 *mword2 
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4 
 UDMA modes: mode0 mode1 mode2 mode3 mode4 mode5 
=============================================================================

This is odd - maybe it's why I can't talk to the drive on ide0 any more.
The RAID bus controller on 00:05 is use as ide2/3 and working fine as I write.

lspci -vvv
...

00:05.0 RAID bus controller: CMD Technology Inc: Unknown device 0680 (rev 02)
	Subsystem: CMD Technology Inc: Unknown device 3680
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 set, cache line size 01
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at 2c00 [size=8]
	Region 1: I/O ports at 2c08 [size=4]
	Region 2: I/O ports at 2c10 [size=8]
	Region 3: I/O ports at 2c18 [size=4]
	Region 4: I/O ports at 2c20 [size=16]
	Region 5: Memory at c4dfee00 (32-bit, non-prefetchable) [size=256]
	Expansion ROM at <unassigned> [disabled] [size=512K]
	Capabilities: [60] Power Management version 2
		Flags: PMEClk- AuxPwr- DSI+ D1+ D2+ PME-
		Status: D0 PME-Enable- DSel=0 DScale=2 PME-


00:0f.1 IDE interface: Relience Computer: Unknown device 0211 (prog-if 8a [Master SecP PriP])
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 set
	Region 4: I/O ports at 2c30 [size=16]

=============================================================================