All of lore.kernel.org
 help / color / mirror / Atom feed
* md data-check causes soft lockup
@ 2009-09-21 18:08 Lee Howard
  2009-09-21 18:54 ` Majed B.
                   ` (2 more replies)
  0 siblings, 3 replies; 14+ messages in thread
From: Lee Howard @ 2009-09-21 18:08 UTC (permalink / raw)
  To: neilb, dan.j.williams, mingo, linux-raid

[-- Attachment #1: Type: text/plain, Size: 1994 bytes --]

Linux software RAID maintainers and developers,

Two months ago I wrote to the Linux kernel mailing list regarding a 
condition expressed as "BUG: soft lockup - CPU#3 stuck for 61s!".  I 
initially battled this recurring problem in both Fedora 10 and Fedora 
11.  Rafael J. Wysocki suggested that I update the kernel (to 2.6.31-rc4 
or later) and see if the problem resurfaced.  I then used kernel 
2.6.31-0.94.rc4.fc12.x86_64 and found that the problem still continued, 
but noticeably only when the md data-check process was run.

You can read the last post to the LKML thread (with links to the entire 
thread) here:

http://lkml.org/lkml/2009/8/6/387

The md data-check is being run in Fedora 11 by 
/etc/cron.weekly/raid-check which is a little shell script that looks 
like this:

------------------------------------------------------
#!/bin/bash

for dev in `grep "^md.*: active" /proc/mdstat | cut -f 1 -d ' '`; do
        [ -f /sys/block/$dev/md/sync_action ] && \
                echo "check" > /sys/block/$dev/md/sync_action
done
------------------------------------------------------

I have disabled this weekly data-check, and since doing so have not 
encountered any soft lockup (or any other problem, for that matter).

For reference, you can see in this thread here that I am not the only 
one to have this problem:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/212684

It may important to note that I am running a 4-disk RAID1 array on 500GB 
Western-Digial SATA drives.  I've attached the output of 'lspci -vv' as 
"lspci.out".  I've attached the output of 'hdparm -I /dev/sda' as 
"hdparm.out".  /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd are all 
identical drive types.  I've attached the output of 'for i in 0 1 2; do 
mdadm --detail /dev/md$i; done' as "mdadm.out".

Please let me know if this is a configuration problem, a kernel bug, or 
something else.  Please let me know how to fix this problem so that I 
can safely re-enable md data-check.

Thanks,

Lee.

[-- Attachment #2: lspci.out --]
[-- Type: text/plain, Size: 28864 bytes --]

00:00.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a2)
	Subsystem: nVidia Corporation Device cb84
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Capabilities: [44] HyperTransport: Slave or Primary Interface
		Command: BaseUnitID=0 UnitCnt=15 MastHost- DefDir- DUL-
		Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
		Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Link Control 1: CFlE- CST- CFE- <LkFail+ Init- EOC+ TXO+ <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
		Link Config 1: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut- LWI=8bit DwFcInEn- LWO=8bit DwFcOutEn-
		Revision ID: 1.03
		Link Frequency 0: 1.0GHz
		Link Error 0: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability 0: 200MHz+ 300MHz+ 400MHz+ 500MHz+ 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA- UIDRD-
		Link Frequency 1: 200MHz
		Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability 1: 200MHz- 300MHz- 400MHz- 500MHz- 600MHz- 800MHz- 1.0GHz- 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Error Handling: PFlE+ OFlE+ PFE- OFE- EOCFE- RFE- CRCFE- SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
		Prefetchable memory behind bridge Upper: 00-00
		Bus Number: 00
	Capabilities: [dc] HyperTransport: MSI Mapping Enable+ Fixed-
		Mapping Address Base: 00000000fee00000

00:01.0 ISA bridge: nVidia Corporation MCP55 LPC Bridge (rev a3)
	Subsystem: nVidia Corporation Device cb84
	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Region 0: I/O ports at 2f00 [size=128]

00:01.1 SMBus: nVidia Corporation MCP55 SMBus (rev a3)
	Subsystem: nVidia Corporation Device cb84
	Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 10
	Region 0: I/O ports at dc00 [size=64]
	Region 4: I/O ports at 2d00 [size=64]
	Region 5: I/O ports at 2e00 [size=64]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: nForce2_smbus
	Kernel modules: i2c-nforce2

00:02.0 USB Controller: nVidia Corporation MCP55 USB Controller (rev a1) (prog-if 10 [OHCI])
	Subsystem: nVidia Corporation Device cb84
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (750ns min, 250ns max)
	Interrupt: pin A routed to IRQ 22
	Region 0: Memory at feafb000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ohci_hcd

00:02.1 USB Controller: nVidia Corporation MCP55 USB Controller (rev a2) (prog-if 20 [EHCI])
	Subsystem: nVidia Corporation Device cb84
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (750ns min, 250ns max)
	Interrupt: pin B routed to IRQ 23
	Region 0: Memory at feafac00 (32-bit, non-prefetchable) [size=256]
	Capabilities: [44] Debug port: BAR=1 offset=0098
	Capabilities: [80] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ehci_hcd

00:04.0 IDE interface: nVidia Corporation MCP55 IDE (rev a1) (prog-if 8a [Master SecP PriP])
	Subsystem: nVidia Corporation Device cb84
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (750ns min, 250ns max)
	Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
	Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
	Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
	Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
	Region 4: I/O ports at ffa0 [size=16]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: pata_amd
	Kernel modules: ata_generic, pata_acpi, pata_amd

00:05.0 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3) (prog-if 85 [Master SecO PriO])
	Subsystem: nVidia Corporation Device cb84
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (750ns min, 250ns max)
	Interrupt: pin A routed to IRQ 21
	Region 0: I/O ports at d480 [size=8]
	Region 1: I/O ports at d400 [size=4]
	Region 2: I/O ports at d080 [size=8]
	Region 3: I/O ports at d000 [size=4]
	Region 4: I/O ports at cc00 [size=16]
	Region 5: Memory at feaf9000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
	Kernel driver in use: sata_nv
	Kernel modules: ata_generic, pata_acpi, sata_nv

00:05.1 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3) (prog-if 85 [Master SecO PriO])
	Subsystem: nVidia Corporation Device cb84
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (750ns min, 250ns max)
	Interrupt: pin B routed to IRQ 20
	Region 0: I/O ports at c880 [size=8]
	Region 1: I/O ports at c800 [size=4]
	Region 2: I/O ports at c480 [size=8]
	Region 3: I/O ports at c400 [size=4]
	Region 4: I/O ports at c080 [size=16]
	Region 5: Memory at feaf8000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
	Kernel driver in use: sata_nv
	Kernel modules: ata_generic, pata_acpi, sata_nv

00:05.2 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3) (prog-if 85 [Master SecO PriO])
	Subsystem: nVidia Corporation Device cb84
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (750ns min, 250ns max)
	Interrupt: pin C routed to IRQ 23
	Region 0: I/O ports at c000 [size=8]
	Region 1: I/O ports at bc00 [size=4]
	Region 2: I/O ports at b880 [size=8]
	Region 3: I/O ports at b800 [size=4]
	Region 4: I/O ports at b480 [size=16]
	Region 5: Memory at feaf7000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
	Kernel driver in use: sata_nv
	Kernel modules: ata_generic, pata_acpi, sata_nv

00:06.0 PCI bridge: nVidia Corporation MCP55 PCI bridge (rev a2) (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: feb00000-febfffff
	Prefetchable memory behind bridge: f8000000-fbffffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr+ DiscTmrStat- DiscTmrSERREn-
	Capabilities: [b8] Subsystem: nVidia Corporation Device cb84
	Capabilities: [8c] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000

00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
	Subsystem: nVidia Corporation Device cb84
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (250ns min, 5000ns max)
	Interrupt: pin A routed to IRQ 30
	Region 0: Memory at feaf6000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at b400 [size=8]
	Region 2: Memory at feafa800 (32-bit, non-prefetchable) [size=256]
	Region 3: Memory at feafa400 (32-bit, non-prefetchable) [size=16]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
		Vector table: BAR=2 offset=00000000
		PBA: BAR=3 offset=00000000
	Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
		Address: 00000000fee0400c  Data: 4199
		Masking: 000000fe  Pending: 00000000
	Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
	Kernel driver in use: forcedeth
	Kernel modules: forcedeth

00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
	Subsystem: nVidia Corporation Device cb84
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (250ns min, 5000ns max)
	Interrupt: pin A routed to IRQ 31
	Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at b080 [size=8]
	Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256]
	Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
	Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
		Vector table: BAR=2 offset=00000000
		PBA: BAR=3 offset=00000000
	Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
		Address: 00000000fee0200c  Data: 41a1
		Masking: 000000fe  Pending: 00000000
	Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
	Kernel driver in use: forcedeth
	Kernel modules: forcedeth

00:0a.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: nVidia Corporation Device 0000
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
		Address: 00000000fee0f00c  Data: 4149
	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #5, Speed 2.5GT/s, Width x8, ASPM L0s L1, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
			Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd On, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [100] Virtual Channel <?>
	Kernel driver in use: pcieport-driver
	Kernel modules: shpchp

00:0b.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: nVidia Corporation Device 0000
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
		Address: 00000000fee0f00c  Data: 4151
	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
			Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd On, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [100] Virtual Channel <?>
	Kernel driver in use: pcieport-driver
	Kernel modules: shpchp

00:0c.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: nVidia Corporation Device 0000
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
		Address: 00000000fee0f00c  Data: 4159
	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
			Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd On, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [100] Virtual Channel <?>
	Kernel driver in use: pcieport-driver
	Kernel modules: shpchp

00:0d.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: nVidia Corporation Device 0000
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
		Address: 00000000fee0f00c  Data: 4161
	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
			Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd On, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [100] Virtual Channel <?>
	Kernel driver in use: pcieport-driver
	Kernel modules: shpchp

00:0e.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: nVidia Corporation Device 0000
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
		Address: 00000000fee0f00c  Data: 4169
	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
			Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd On, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [100] Virtual Channel <?>
	Kernel driver in use: pcieport-driver
	Kernel modules: shpchp

00:0f.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: nVidia Corporation Device 0000
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
		Address: 00000000fee0f00c  Data: 4171
	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
			Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd On, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [100] Virtual Channel <?>
	Kernel driver in use: pcieport-driver
	Kernel modules: shpchp

00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] HyperTransport Configuration
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [80] HyperTransport: Host or Secondary Interface
		Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+ Slave- <EOCErr- DUL-
		Link Control: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
		Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Revision ID: 3.00
		Link Frequency: 1.0GHz
		Link Error: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz+ 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+ UIDRD- ExtRS- UCnfE-

00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Address Map
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] DRAM Controller
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel modules: amd64_edac_mod

00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Miscellaneous Control
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [f0] Secure device <?>

00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Link Control
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

01:0a.0 VGA compatible controller: XGI Technology Inc. (eXtreme Graphics Innovation) Volari Z7/Z9/Z9s (prog-if 00 [VGA controller])
	Subsystem: XGI Technology Inc. (eXtreme Graphics Innovation) Volari Z7/Z9/Z9s
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	BIST result: 00
	Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
	Region 1: Memory at febc0000 (32-bit, non-prefetchable) [size=256K]
	Region 2: I/O ports at ec00 [size=128]
	Expansion ROM at <unassigned> [disabled]
	Capabilities: [40] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-


[-- Attachment #3: hdparm.out --]
[-- Type: text/plain, Size: 2853 bytes --]


/dev/sda:

ATA device, with non-removable media
	Model Number:       WDC WD5000AAKS-00A7B2                   
	Serial Number:      WD-WCASZ0321108
	Firmware Revision:  01.03B01
	Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5
Standards:
	Supported: 8 7 6 5 
	Likely used: 8
Configuration:
	Logical		max	current
	cylinders	16383	16383
	heads		16	16
	sectors/track	63	63
	--
	CHS current addressable sectors:   16514064
	LBA    user addressable sectors:  268435455
	LBA48  user addressable sectors:  976773168
	Logical/Physical Sector size:           512 bytes
	device size with M = 1024*1024:      476940 MBytes
	device size with M = 1000*1000:      500107 MBytes (500 GB)
	cache/buffer size  = 16384 KBytes
Capabilities:
	LBA, IORDY(can be disabled)
	Queue depth: 32
	Standby timer values: spec'd by Standard, with device specific minimum
	R/W multiple sector transfer: Max = 16	Current = 16
	Recommended acoustic management value: 128, current value: 254
	DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
	     Cycle time: min=120ns recommended=120ns
	PIO: pio0 pio1 pio2 pio3 pio4 
	     Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
	Enabled	Supported:
	   *	SMART feature set
	    	Security Mode feature set
	   *	Power Management feature set
	   *	Write cache
	   *	Look-ahead
	   *	Host Protected Area feature set
	   *	WRITE_BUFFER command
	   *	READ_BUFFER command
	   *	NOP cmd
	   *	DOWNLOAD_MICROCODE
	    	Power-Up In Standby feature set
	   *	SET_FEATURES required to spinup after power up
	    	SET_MAX security extension
	    	Automatic Acoustic Management feature set
	   *	48-bit Address feature set
	   *	Device Configuration Overlay feature set
	   *	Mandatory FLUSH_CACHE
	   *	FLUSH_CACHE_EXT
	   *	SMART error logging
	   *	SMART self-test
	   *	General Purpose Logging feature set
	   *	64-bit World wide name
	   *	Segmented DOWNLOAD_MICROCODE
	   *	Gen1 signaling speed (1.5Gb/s)
	   *	Gen2 signaling speed (3.0Gb/s)
	   *	Native Command Queueing (NCQ)
	   *	Host-initiated interface power management
	   *	Phy event counters
	    	DMA Setup Auto-Activate optimization
	   *	Software settings preservation
	   *	SMART Command Transport (SCT) feature set
	   *	SCT Long Sector Access (AC1)
	   *	SCT LBA Segment Access (AC2)
	   *	SCT Error Recovery Control (AC3)
	   *	SCT Features Control (AC4)
	   *	SCT Data Tables (AC5)
	    	unknown 206[12] (vendor specific)
	    	unknown 206[13] (vendor specific)
Security: 
	Master password revision code = 65534
		supported
	not	enabled
	not	locked
		frozen
	not	expired: security count
		supported: enhanced erase
	84min for SECURITY ERASE UNIT. 84min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50014ee101cedc24
	NAA		: 5
	IEEE OUI	: 0014ee
	Unique ID	: 101cedc24
Checksum: correct

[-- Attachment #4: mdadm.out --]
[-- Type: text/plain, Size: 2482 bytes --]

/dev/md0:
        Version : 0.90
  Creation Time : Mon Jul 27 22:38:26 2009
     Raid Level : raid1
     Array Size : 102336 (99.95 MiB 104.79 MB)
  Used Dev Size : 102336 (99.95 MiB 104.79 MB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Sep 21 11:02:14 2009
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

           UUID : 9037def2:035b0d10:bfe78010:bc810f04
         Events : 0.44

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1
/dev/md1:
        Version : 0.90
  Creation Time : Mon Jul 27 22:38:28 2009
     Raid Level : raid1
     Array Size : 4095936 (3.91 GiB 4.19 GB)
  Used Dev Size : 4095936 (3.91 GiB 4.19 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Mon Sep 21 11:02:14 2009
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

           UUID : 642c2833:bf033a7d:bfe78010:bc810f04
         Events : 0.44

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2
       3       8       50        3      active sync   /dev/sdd2
/dev/md2:
        Version : 0.90
  Creation Time : Mon Jul 27 22:38:30 2009
     Raid Level : raid1
     Array Size : 484185536 (461.76 GiB 495.81 GB)
  Used Dev Size : 484185536 (461.76 GiB 495.81 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Mon Sep 21 11:02:24 2009
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

           UUID : ce106d52:0c0373f7:bfe78010:bc810f04
         Events : 0.66

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       2       8       35        2      active sync   /dev/sdc3
       3       8       51        3      active sync   /dev/sdd3

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-21 18:08 md data-check causes soft lockup Lee Howard
@ 2009-09-21 18:54 ` Majed B.
  2009-09-22 14:43   ` Lee Howard
  2009-09-21 19:13 ` kwick
  2009-09-25  6:54 ` Neil Brown
  2 siblings, 1 reply; 14+ messages in thread
From: Majed B. @ 2009-09-21 18:54 UTC (permalink / raw)
  To: Lee Howard; +Cc: linux-raid

Hello,

Check out the thread with the subject: BUG: soft lockup in [md4_raid5:21137]

I think the guys have already fixed this in 2.6.32. The patch is
mentioned in one of the replies, if you wish to try it.

On Mon, Sep 21, 2009 at 9:08 PM, Lee Howard <faxguy@howardsilvan.com> wrote:
> Linux software RAID maintainers and developers,
>
> Two months ago I wrote to the Linux kernel mailing list regarding a
> condition expressed as "BUG: soft lockup - CPU#3 stuck for 61s!".  I
> initially battled this recurring problem in both Fedora 10 and Fedora 11.
>  Rafael J. Wysocki suggested that I update the kernel (to 2.6.31-rc4 or
> later) and see if the problem resurfaced.  I then used kernel
> 2.6.31-0.94.rc4.fc12.x86_64 and found that the problem still continued, but
> noticeably only when the md data-check process was run.
>
> You can read the last post to the LKML thread (with links to the entire
> thread) here:
>
> http://lkml.org/lkml/2009/8/6/387
>
> The md data-check is being run in Fedora 11 by /etc/cron.weekly/raid-check
> which is a little shell script that looks like this:
>
> ------------------------------------------------------
> #!/bin/bash
>
> for dev in `grep "^md.*: active" /proc/mdstat | cut -f 1 -d ' '`; do
>       [ -f /sys/block/$dev/md/sync_action ] && \
>               echo "check" > /sys/block/$dev/md/sync_action
> done
> ------------------------------------------------------
>
> I have disabled this weekly data-check, and since doing so have not
> encountered any soft lockup (or any other problem, for that matter).
>
> For reference, you can see in this thread here that I am not the only one to
> have this problem:
>
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/212684
>
> It may important to note that I am running a 4-disk RAID1 array on 500GB
> Western-Digial SATA drives.  I've attached the output of 'lspci -vv' as
> "lspci.out".  I've attached the output of 'hdparm -I /dev/sda' as
> "hdparm.out".  /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd are all identical
> drive types.  I've attached the output of 'for i in 0 1 2; do mdadm --detail
> /dev/md$i; done' as "mdadm.out".
>
> Please let me know if this is a configuration problem, a kernel bug, or
> something else.  Please let me know how to fix this problem so that I can
> safely re-enable md data-check.
>
> Thanks,
>
> Lee.
>
> 00:00.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a2)
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0
>        Capabilities: [44] HyperTransport: Slave or Primary Interface
>                Command: BaseUnitID=0 UnitCnt=15 MastHost- DefDir- DUL-
>                Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO-
> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>                Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut-
> LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
>                Link Control 1: CFlE- CST- CFE- <LkFail+ Init- EOC+ TXO+
> <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
>                Link Config 1: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut- LWI=8bit
> DwFcInEn- LWO=8bit DwFcOutEn-
>                Revision ID: 1.03
>                Link Frequency 0: 1.0GHz
>                Link Error 0: <Prot- <Ovfl- <EOC- CTLTm-
>                Link Frequency Capability 0: 200MHz+ 300MHz+ 400MHz+ 500MHz+
> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>                Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA-
> UIDRD-
>                Link Frequency 1: 200MHz
>                Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
>                Link Frequency Capability 1: 200MHz- 300MHz- 400MHz- 500MHz-
> 600MHz- 800MHz- 1.0GHz- 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>                Error Handling: PFlE+ OFlE+ PFE- OFE- EOCFE- RFE- CRCFE-
> SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
>                Prefetchable memory behind bridge Upper: 00-00
>                Bus Number: 00
>        Capabilities: [dc] HyperTransport: MSI Mapping Enable+ Fixed-
>                Mapping Address Base: 00000000fee00000
>
> 00:01.0 ISA bridge: nVidia Corporation MCP55 LPC Bridge (rev a3)
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0
>        Region 0: I/O ports at 2f00 [size=128]
>
> 00:01.1 SMBus: nVidia Corporation MCP55 SMBus (rev a3)
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Interrupt: pin A routed to IRQ 10
>        Region 0: I/O ports at dc00 [size=64]
>        Region 4: I/O ports at 2d00 [size=64]
>        Region 5: I/O ports at 2e00 [size=64]
>        Capabilities: [44] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Kernel driver in use: nForce2_smbus
>        Kernel modules: i2c-nforce2
>
> 00:02.0 USB Controller: nVidia Corporation MCP55 USB Controller (rev a1)
> (prog-if 10 [OHCI])
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0 (750ns min, 250ns max)
>        Interrupt: pin A routed to IRQ 22
>        Region 0: Memory at feafb000 (32-bit, non-prefetchable) [size=4K]
>        Capabilities: [44] Power Management version 2
>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Kernel driver in use: ohci_hcd
>
> 00:02.1 USB Controller: nVidia Corporation MCP55 USB Controller (rev a2)
> (prog-if 20 [EHCI])
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0 (750ns min, 250ns max)
>        Interrupt: pin B routed to IRQ 23
>        Region 0: Memory at feafac00 (32-bit, non-prefetchable) [size=256]
>        Capabilities: [44] Debug port: BAR=1 offset=0098
>        Capabilities: [80] Power Management version 2
>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Kernel driver in use: ehci_hcd
>
> 00:04.0 IDE interface: nVidia Corporation MCP55 IDE (rev a1) (prog-if 8a
> [Master SecP PriP])
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0 (750ns min, 250ns max)
>        Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable)
> [size=8]
>        Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
> [size=1]
>        Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable)
> [size=8]
>        Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
> [size=1]
>        Region 4: I/O ports at ffa0 [size=16]
>        Capabilities: [44] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Kernel driver in use: pata_amd
>        Kernel modules: ata_generic, pata_acpi, pata_amd
>
> 00:05.0 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
> (prog-if 85 [Master SecO PriO])
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0 (750ns min, 250ns max)
>        Interrupt: pin A routed to IRQ 21
>        Region 0: I/O ports at d480 [size=8]
>        Region 1: I/O ports at d400 [size=4]
>        Region 2: I/O ports at d080 [size=8]
>        Region 3: I/O ports at d000 [size=4]
>        Region 4: I/O ports at cc00 [size=16]
>        Region 5: Memory at feaf9000 (32-bit, non-prefetchable) [size=4K]
>        Capabilities: [44] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>                Address: 0000000000000000  Data: 0000
>        Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>        Kernel driver in use: sata_nv
>        Kernel modules: ata_generic, pata_acpi, sata_nv
>
> 00:05.1 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
> (prog-if 85 [Master SecO PriO])
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0 (750ns min, 250ns max)
>        Interrupt: pin B routed to IRQ 20
>        Region 0: I/O ports at c880 [size=8]
>        Region 1: I/O ports at c800 [size=4]
>        Region 2: I/O ports at c480 [size=8]
>        Region 3: I/O ports at c400 [size=4]
>        Region 4: I/O ports at c080 [size=16]
>        Region 5: Memory at feaf8000 (32-bit, non-prefetchable) [size=4K]
>        Capabilities: [44] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>                Address: 0000000000000000  Data: 0000
>        Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>        Kernel driver in use: sata_nv
>        Kernel modules: ata_generic, pata_acpi, sata_nv
>
> 00:05.2 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
> (prog-if 85 [Master SecO PriO])
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0 (750ns min, 250ns max)
>        Interrupt: pin C routed to IRQ 23
>        Region 0: I/O ports at c000 [size=8]
>        Region 1: I/O ports at bc00 [size=4]
>        Region 2: I/O ports at b880 [size=8]
>        Region 3: I/O ports at b800 [size=4]
>        Region 4: I/O ports at b480 [size=16]
>        Region 5: Memory at feaf7000 (32-bit, non-prefetchable) [size=4K]
>        Capabilities: [44] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>                Address: 0000000000000000  Data: 0000
>        Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>        Kernel driver in use: sata_nv
>        Kernel modules: ata_generic, pata_acpi, sata_nv
>
> 00:06.0 PCI bridge: nVidia Corporation MCP55 PCI bridge (rev a2) (prog-if 01
> [Subtractive decode])
>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0
>        Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>        I/O behind bridge: 0000e000-0000efff
>        Memory behind bridge: feb00000-febfffff
>        Prefetchable memory behind bridge: f8000000-fbffffff
>        Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>        BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
>                PriDiscTmr- SecDiscTmr+ DiscTmrStat- DiscTmrSERREn-
>        Capabilities: [b8] Subsystem: nVidia Corporation Device cb84
>        Capabilities: [8c] HyperTransport: MSI Mapping Enable- Fixed-
>                Mapping Address Base: 00000000fee00000
>
> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx+
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0 (250ns min, 5000ns max)
>        Interrupt: pin A routed to IRQ 30
>        Region 0: Memory at feaf6000 (32-bit, non-prefetchable) [size=4K]
>        Region 1: I/O ports at b400 [size=8]
>        Region 2: Memory at feafa800 (32-bit, non-prefetchable) [size=256]
>        Region 3: Memory at feafa400 (32-bit, non-prefetchable) [size=16]
>        Capabilities: [44] Power Management version 2
>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>        Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>                Vector table: BAR=2 offset=00000000
>                PBA: BAR=3 offset=00000000
>        Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>                Address: 00000000fee0400c  Data: 4199
>                Masking: 000000fe  Pending: 00000000
>        Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>        Kernel driver in use: forcedeth
>        Kernel modules: forcedeth
>
> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>        Subsystem: nVidia Corporation Device cb84
>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx+
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0 (250ns min, 5000ns max)
>        Interrupt: pin A routed to IRQ 31
>        Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
>        Region 1: I/O ports at b080 [size=8]
>        Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256]
>        Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16]
>        Capabilities: [44] Power Management version 2
>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>        Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>                Vector table: BAR=2 offset=00000000
>                PBA: BAR=3 offset=00000000
>        Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>                Address: 00000000fee0200c  Data: 41a1
>                Masking: 000000fe  Pending: 00000000
>        Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>        Kernel driver in use: forcedeth
>        Kernel modules: forcedeth
>
> 00:0a.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
> (prog-if 00 [Normal decode])
>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx+
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0, Cache Line Size: 64 bytes
>        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>        Capabilities: [48] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>                Address: 00000000fee0f00c  Data: 4149
>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>                Mapping Address Base: 00000000fee00000
>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                        ExtTag- RBE+ FLReset-
>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>                LnkCap: Port #5, Speed 2.5GT/s, Width x8, ASPM L0s L1,
> Latency L0 <512ns, L1 <4us
>                        ClockPM- Surprise- LLActRep+ BwNot-
>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
> CommClk-
>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
> HPIrq- LinkChg-
>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                        Changed: MRL- PresDet- LinkState-
>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
> CRSVisible-
>                RootCap: CRSVisible-
>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>        Capabilities: [100] Virtual Channel <?>
>        Kernel driver in use: pcieport-driver
>        Kernel modules: shpchp
>
> 00:0b.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
> (prog-if 00 [Normal decode])
>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx+
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0, Cache Line Size: 64 bytes
>        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>        Capabilities: [48] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>                Address: 00000000fee0f00c  Data: 4151
>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>                Mapping Address Base: 00000000fee00000
>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                        ExtTag- RBE+ FLReset-
>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>                LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> Latency L0 <512ns, L1 <4us
>                        ClockPM- Surprise- LLActRep+ BwNot-
>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
> CommClk-
>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
> HPIrq- LinkChg-
>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                        Changed: MRL- PresDet- LinkState-
>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
> CRSVisible-
>                RootCap: CRSVisible-
>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>        Capabilities: [100] Virtual Channel <?>
>        Kernel driver in use: pcieport-driver
>        Kernel modules: shpchp
>
> 00:0c.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
> (prog-if 00 [Normal decode])
>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx+
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0, Cache Line Size: 64 bytes
>        Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>        Capabilities: [48] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>                Address: 00000000fee0f00c  Data: 4159
>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>                Mapping Address Base: 00000000fee00000
>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                        ExtTag- RBE+ FLReset-
>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>                LnkCap: Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> Latency L0 <512ns, L1 <4us
>                        ClockPM- Surprise- LLActRep+ BwNot-
>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
> CommClk-
>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
> HPIrq- LinkChg-
>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                        Changed: MRL- PresDet- LinkState-
>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
> CRSVisible-
>                RootCap: CRSVisible-
>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>        Capabilities: [100] Virtual Channel <?>
>        Kernel driver in use: pcieport-driver
>        Kernel modules: shpchp
>
> 00:0d.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
> (prog-if 00 [Normal decode])
>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx+
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0, Cache Line Size: 64 bytes
>        Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>        Capabilities: [48] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>                Address: 00000000fee0f00c  Data: 4161
>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>                Mapping Address Base: 00000000fee00000
>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                        ExtTag- RBE+ FLReset-
>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>                LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> Latency L0 <512ns, L1 <4us
>                        ClockPM- Surprise- LLActRep+ BwNot-
>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
> CommClk-
>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
> HPIrq- LinkChg-
>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                        Changed: MRL- PresDet- LinkState-
>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
> CRSVisible-
>                RootCap: CRSVisible-
>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>        Capabilities: [100] Virtual Channel <?>
>        Kernel driver in use: pcieport-driver
>        Kernel modules: shpchp
>
> 00:0e.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
> (prog-if 00 [Normal decode])
>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx+
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0, Cache Line Size: 64 bytes
>        Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>        Capabilities: [48] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>                Address: 00000000fee0f00c  Data: 4169
>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>                Mapping Address Base: 00000000fee00000
>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                        ExtTag- RBE+ FLReset-
>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>                LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1,
> Latency L0 <512ns, L1 <4us
>                        ClockPM- Surprise- LLActRep+ BwNot-
>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
> CommClk-
>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
> HPIrq- LinkChg-
>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                        Changed: MRL- PresDet- LinkState-
>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
> CRSVisible-
>                RootCap: CRSVisible-
>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>        Capabilities: [100] Virtual Channel <?>
>        Kernel driver in use: pcieport-driver
>        Kernel modules: shpchp
>
> 00:0f.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
> (prog-if 00 [Normal decode])
>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx+
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Latency: 0, Cache Line Size: 64 bytes
>        Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- <SERR- <PERR-
>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>        Capabilities: [48] Power Management version 2
>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>                Address: 00000000fee0f00c  Data: 4171
>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>                Mapping Address Base: 00000000fee00000
>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
> L1 <1us
>                        ExtTag- RBE+ FLReset-
>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
>                LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1,
> Latency L0 <512ns, L1 <4us
>                        ClockPM- Surprise- LLActRep+ BwNot-
>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
> CommClk-
>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
> Surpise-
>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
> HPIrq- LinkChg-
>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
> Interlock-
>                        Changed: MRL- PresDet- LinkState-
>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
> CRSVisible-
>                RootCap: CRSVisible-
>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>        Capabilities: [100] Virtual Channel <?>
>        Kernel driver in use: pcieport-driver
>        Kernel modules: shpchp
>
> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
> Athlon64, Sempron] HyperTransport Configuration
>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Capabilities: [80] HyperTransport: Host or Secondary Interface
>                Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+
> Slave- <EOCErr- DUL-
>                Link Control: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO-
> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>                Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit
> DwFcInEn- LWO=16bit DwFcOutEn-
>                Revision ID: 3.00
>                Link Frequency: 1.0GHz
>                Link Error: <Prot- <Ovfl- <EOC- CTLTm-
>                Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz-
> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz+ 1.4GHz- 1.6GHz- Vend-
>                Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+
> UIDRD- ExtRS- UCnfE-
>
> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
> Athlon64, Sempron] Address Map
>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
> Athlon64, Sempron] DRAM Controller
>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Kernel modules: amd64_edac_mod
>
> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
> Athlon64, Sempron] Miscellaneous Control
>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Capabilities: [f0] Secure device <?>
>
> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
> Athlon64, Sempron] Link Control
>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
> 01:0a.0 VGA compatible controller: XGI Technology Inc. (eXtreme Graphics
> Innovation) Volari Z7/Z9/Z9s (prog-if 00 [VGA controller])
>        Subsystem: XGI Technology Inc. (eXtreme Graphics Innovation) Volari
> Z7/Z9/Z9s
>        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>        BIST result: 00
>        Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
>        Region 1: Memory at febc0000 (32-bit, non-prefetchable) [size=256K]
>        Region 2: I/O ports at ec00 [size=128]
>        Expansion ROM at <unassigned> [disabled]
>        Capabilities: [40] Power Management version 2
>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>
>
>
> /dev/sda:
>
> ATA device, with non-removable media
>        Model Number:       WDC WD5000AAKS-00A7B2
>        Serial Number:      WD-WCASZ0321108
>        Firmware Revision:  01.03B01
>        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev
> 2.5
> Standards:
>        Supported: 8 7 6 5
>        Likely used: 8
> Configuration:
>        Logical         max     current
>        cylinders       16383   16383
>        heads           16      16
>        sectors/track   63      63
>        --
>        CHS current addressable sectors:   16514064
>        LBA    user addressable sectors:  268435455
>        LBA48  user addressable sectors:  976773168
>        Logical/Physical Sector size:           512 bytes
>        device size with M = 1024*1024:      476940 MBytes
>        device size with M = 1000*1000:      500107 MBytes (500 GB)
>        cache/buffer size  = 16384 KBytes
> Capabilities:
>        LBA, IORDY(can be disabled)
>        Queue depth: 32
>        Standby timer values: spec'd by Standard, with device specific
> minimum
>        R/W multiple sector transfer: Max = 16  Current = 16
>        Recommended acoustic management value: 128, current value: 254
>        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
>             Cycle time: min=120ns recommended=120ns
>        PIO: pio0 pio1 pio2 pio3 pio4
>             Cycle time: no flow control=120ns  IORDY flow control=120ns
> Commands/features:
>        Enabled Supported:
>           *    SMART feature set
>                Security Mode feature set
>           *    Power Management feature set
>           *    Write cache
>           *    Look-ahead
>           *    Host Protected Area feature set
>           *    WRITE_BUFFER command
>           *    READ_BUFFER command
>           *    NOP cmd
>           *    DOWNLOAD_MICROCODE
>                Power-Up In Standby feature set
>           *    SET_FEATURES required to spinup after power up
>                SET_MAX security extension
>                Automatic Acoustic Management feature set
>           *    48-bit Address feature set
>           *    Device Configuration Overlay feature set
>           *    Mandatory FLUSH_CACHE
>           *    FLUSH_CACHE_EXT
>           *    SMART error logging
>           *    SMART self-test
>           *    General Purpose Logging feature set
>           *    64-bit World wide name
>           *    Segmented DOWNLOAD_MICROCODE
>           *    Gen1 signaling speed (1.5Gb/s)
>           *    Gen2 signaling speed (3.0Gb/s)
>           *    Native Command Queueing (NCQ)
>           *    Host-initiated interface power management
>           *    Phy event counters
>                DMA Setup Auto-Activate optimization
>           *    Software settings preservation
>           *    SMART Command Transport (SCT) feature set
>           *    SCT Long Sector Access (AC1)
>           *    SCT LBA Segment Access (AC2)
>           *    SCT Error Recovery Control (AC3)
>           *    SCT Features Control (AC4)
>           *    SCT Data Tables (AC5)
>                unknown 206[12] (vendor specific)
>                unknown 206[13] (vendor specific)
> Security:
>        Master password revision code = 65534
>                supported
>        not     enabled
>        not     locked
>                frozen
>        not     expired: security count
>                supported: enhanced erase
>        84min for SECURITY ERASE UNIT. 84min for ENHANCED SECURITY ERASE
> UNIT.
> Logical Unit WWN Device Identifier: 50014ee101cedc24
>        NAA             : 5
>        IEEE OUI        : 0014ee
>        Unique ID       : 101cedc24
> Checksum: correct
>
> /dev/md0:
>        Version : 0.90
>  Creation Time : Mon Jul 27 22:38:26 2009
>     Raid Level : raid1
>     Array Size : 102336 (99.95 MiB 104.79 MB)
>  Used Dev Size : 102336 (99.95 MiB 104.79 MB)
>   Raid Devices : 4
>  Total Devices : 4
> Preferred Minor : 0
>    Persistence : Superblock is persistent
>
>    Update Time : Mon Sep 21 11:02:14 2009
>          State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 0
>  Spare Devices : 0
>
>           UUID : 9037def2:035b0d10:bfe78010:bc810f04
>         Events : 0.44
>
>    Number   Major   Minor   RaidDevice State
>       0       8        1        0      active sync   /dev/sda1
>       1       8       17        1      active sync   /dev/sdb1
>       2       8       33        2      active sync   /dev/sdc1
>       3       8       49        3      active sync   /dev/sdd1
> /dev/md1:
>        Version : 0.90
>  Creation Time : Mon Jul 27 22:38:28 2009
>     Raid Level : raid1
>     Array Size : 4095936 (3.91 GiB 4.19 GB)
>  Used Dev Size : 4095936 (3.91 GiB 4.19 GB)
>   Raid Devices : 4
>  Total Devices : 4
> Preferred Minor : 1
>    Persistence : Superblock is persistent
>
>    Update Time : Mon Sep 21 11:02:14 2009
>          State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 0
>  Spare Devices : 0
>
>           UUID : 642c2833:bf033a7d:bfe78010:bc810f04
>         Events : 0.44
>
>    Number   Major   Minor   RaidDevice State
>       0       8        2        0      active sync   /dev/sda2
>       1       8       18        1      active sync   /dev/sdb2
>       2       8       34        2      active sync   /dev/sdc2
>       3       8       50        3      active sync   /dev/sdd2
> /dev/md2:
>        Version : 0.90
>  Creation Time : Mon Jul 27 22:38:30 2009
>     Raid Level : raid1
>     Array Size : 484185536 (461.76 GiB 495.81 GB)
>  Used Dev Size : 484185536 (461.76 GiB 495.81 GB)
>   Raid Devices : 4
>  Total Devices : 4
> Preferred Minor : 2
>    Persistence : Superblock is persistent
>
>    Update Time : Mon Sep 21 11:02:24 2009
>          State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 0
>  Spare Devices : 0
>
>           UUID : ce106d52:0c0373f7:bfe78010:bc810f04
>         Events : 0.66
>
>    Number   Major   Minor   RaidDevice State
>       0       8        3        0      active sync   /dev/sda3
>       1       8       19        1      active sync   /dev/sdb3
>       2       8       35        2      active sync   /dev/sdc3
>       3       8       51        3      active sync   /dev/sdd3
>
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-21 18:08 md data-check causes soft lockup Lee Howard
  2009-09-21 18:54 ` Majed B.
@ 2009-09-21 19:13 ` kwick
  2009-09-25  6:54 ` Neil Brown
  2 siblings, 0 replies; 14+ messages in thread
From: kwick @ 2009-09-21 19:13 UTC (permalink / raw)
  To: linux-raid

Lee Howard wrote:
> Linux software RAID maintainers and developers,
>
> Two months ago I wrote to the Linux kernel mailing list regarding a 
> condition expressed as "BUG: soft lockup - CPU#3 stuck for 61s!".
> [...CUT...]

I also have seen this problem, on our new server with Ubuntu karmic 
alpha1 with ubuntu kernel 2.6.31-5-server (this is not a stable 2.6.31, 
dunno to which release candidate it corresponds exactly) reproducible if 
I set high stripe cache size of 1024 or higher, while performing "check" 
as you report.

The problem starts with stripe_cache_size around 512 for our mdadm 
raid-6 with 26 drives, already gives a noticeable starvation of disk 
accesses. The higher you set it, the more reproducible is the problem, 
and the longer the starvation time. At 1024 seems already 100% 
reproducible and the starvation time seems endless already, I tried up 
to 32768. 256 seems safe.

All disk activity on the RAID is completely stuck while performing 
check. I mean endless starvation on disk access for high 
stripe_cache_sizes. Luckily one can stop the check, lower the 
stripe_cache_size, then start the check again.

This is the dmesg error for me while on soft lockup (appearing 
periodically on various CPUs.)

    [21652.013121] BUG: soft lockup - CPU#5 stuck for 61s! [md1_raid5:1506]
    [21652.013130] Modules linked in: xfs exportfs snd_hda_codec_realtek 
snd_hda_intel snd_hda_codec snd_pcm snd_timer snd ppdev parport_pc 
soundcore parport snd_page_alloc shpchp i5400_edac psmouse edac_core 
serio_raw i5k_amb tpm_infineon tpm tpm_bios ohci1394 ieee1394 e1000e 
3w_9xxx raid10 raid456 raid6_pqasync_xor async_memcpy async_tx xor raid1 
raid0 multipath linear fbcon tileblit font bitblit softcursor i915 drm 
i2c_algo_bit video output intel_agp
    [21652.013130] CPU 5:
    [21652.013130] Modules linked in: xfs exportfs snd_hda_codec_realtek 
snd_hda_intel snd_hda_codec snd_pcm snd_timer snd ppdev parport_pc 
soundcore parport snd_page_alloc shpchp i5400_edac psmouse edac_core 
serio_raw i5k_amb tpm_infineon tpm tpm_bios ohci1394 ieee1394 e1000e 
3w_9xxx raid10 raid456 raid6_pqasync_xor async_memcpy async_tx xor raid1 
raid0 multipath linear fbcon tileblit font bitblit softcursor i915 drm 
i2c_algo_bit video output intel_agp
    [21652.013130] Pid: 1506, comm: md1_raid5 Tainted: G        W  
2.6.31-5-server #24-Ubuntu empty
    [21652.013130] RIP: 0010:[<ffffffffa00f32a1>]  [<ffffffffa00f32a1>] 
raid6_sse24_gen_syndrome+0x161/0x28e [raid6_pq]
    [21652.013130] RSP: 0018:ffff8808159b7a88  EFLAGS: 00000217
    [21652.013130] RAX: ffff88080e52e000 RBX: ffff8808159b7ab0 RCX: 
ffff88080e52e6e0
    [21652.013130] RDX: 0000000000000005 RSI: ffff88080e52e6c0 RDI: 
00000000000006c0
    [21652.013130] RBP: ffffffff81012b6e R08: 00000000000006f0 R09: 
00000000000006e0
    [21652.013130] R10: 00000000000006d0 R11: 00000000000006c0 R12: 
00000000000005f0
    [21652.013130] R13: 00000000000005e0 R14: 00000000000005d0 R15: 
00000000000005c0
    [21652.013130] FS:  0000000000000000(0000) GS:ffff8800280c7000(0000) 
knlGS:0000000000000000
    [21652.013130] CS:  0010 DS: 0018 ES: 0018 CR0: 0000000080050033
    [21652.013130] CR2: 0000000000b20018 CR3: 0000000001001000 CR4: 
00000000000006a0
    [21652.013130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
    [21652.013130] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
    [21652.013130] Call Trace:
    [21652.013130]  [<ffffffffa00f218c>] ? raid6_datap_recov+0x4c/0xc0 
[raid6_pq]
    [21652.013130]  [<ffffffffa010e238>] ? compute_block_2+0x228/0x240 
[raid456]
    [21652.013130]  [<ffffffffa010e39b>] ? 
handle_stripe_fill6+0x14b/0x1e0 [raid456]
    [21652.013130]  [<ffffffffa01103e6>] ? handle_stripe6+0x4c6/0xac0 
[raid456]
    [21652.013130]  [<ffffffffa01113b5>] ? handle_stripe+0x25/0x30 [raid456]
    [21652.013130]  [<ffffffffa01117ba>] ? raid5d+0x20a/0x320 [raid456]
    [21652.013130]  [<ffffffff813cea84>] ? md_thread+0x44/0x120
    [21652.013130]  [<ffffffff81072810>] ? autoremove_wake_function+0x0/0x40
    [21652.013130]  [<ffffffff813cea40>] ? md_thread+0x0/0x120
    [21652.013130]  [<ffffffff81072426>] ? kthread+0x96/0xa0
    [21652.013130]  [<ffffffff8101308a>] ? child_rip+0xa/0x20
    [21652.013130]  [<ffffffff81072390>] ? kthread+0x0/0xa0
    [21652.013130]  [<ffffffff81013080>] ? child_rip+0x0/0x20


Here is a stack trace of vi while it is stuck starved forever, while it 
tries to write the file:
    root@myserver:/proc# cat 7533/stack
        [<ffffffffa030ebd5>] xfs_iget+0xb5/0x1a0 [xfs]
        [<ffffffffa0327e71>] xfs_trans_iget+0xd1/0x100 [xfs]
        [<ffffffffa03111f6>] xfs_ialloc+0x96/0x6c0 [xfs]
        [<ffffffffa0328a4d>] xfs_dir_ialloc+0x7d/0x2d0 [xfs]
        [<ffffffffa032a6d3>] xfs_create+0x3e3/0x5f0 [xfs]
        [<ffffffffa0335c3b>] xfs_vn_mknod+0xab/0x1c0 [xfs]
        [<ffffffffa0335d6b>] xfs_vn_create+0xb/0x10 [xfs]
        [<ffffffff81121fff>] vfs_create+0xaf/0xd0
        [<ffffffff811220dc>] __open_namei_create+0xbc/0x100
        [<ffffffff81125656>] do_filp_open+0x9e6/0xac0
        [<ffffffff81115624>] do_sys_open+0x64/0x160
        [<ffffffff8111574b>] sys_open+0x1b/0x20
        [<ffffffff81011fc2>] system_call_fastpath+0x16/0x1b
        [<ffffffffffffffff>] 0xffffffffffffffff


I have just checked and the same problem happens on our older server 
with Ubuntu Hardy 2.6.24-22-openvz, but only at stripe_cache_size 32768. 
Consider it's a smaller RAID-6 there, 12 devices. At lower 
stripe_cache_sizes I only see a very long wait time for whatever 
application tries to access the disk, such as vi trying to write the file.
Both servers have really plenty of RAM. The older server is a dual-xeon 
5345, the new one is a dual-xeon-5420.

This is the dmesg for the older server:

    [2327104.227028] BUG: soft lockup - CPU#4 stuck for 11s! 
[md1_raid5:3850]
    [2327104.227108] CPU 4:
    [2327104.227110] Modules linked in: nfsd auth_rpcgss exportfs 
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state 
nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables 
kvm_intel kvm nfs lockd nfs_acl sunrpc bridge tun bonding i5k_amb 
coretemp w83627hf hwmon_vid adt7470 i2c_i801 i2c_core ipv6 snd_hda_intel 
snd_pcm sr_mod snd_timer cdrom snd_page_alloc snd_hwdep iTCO_wdt 
iTCO_vendor_support ata_generic serio_raw pata_acpi snd i5000_edac 
button container shpchp pci_hotplug edac_core soundcore evdev parport_pc 
parport psmouse pcspkr ext3 jbd mbcache sg sd_mod mptsas mptscsih 
ata_piix mptbase ehci_hcd scsi_transport_sas 3w_9xxx libata ohci1394 
floppy uhci_hcd scsi_mod ieee1394 usbcore e1000 raid10 raid456 async_xor 
async_memcpy async_txxor raid1 raid0 multipath linear md_mod dm_mirror 
dm_snapshot dm_mod thermal processor fan
    [2327104.227267] Pid: 3850, comm: md1_raid5 Not tainted 
2.6.24-22-openvz #1 ovz005
    [2327104.227269] RIP: 0010:[<ffffffff88095237>]  
[<ffffffff88095237>] :raid456:raid6_sse24_gen_syndrome+0x107/0x200
    [2327104.227286] RSP: 0018:ffff810415c65c10  EFLAGS: 00000217
    [2327104.227290] RAX: ffff810316e9e000 RBX: ffff810415c65c78 RCX: 
ffff810316e9eae0
    [2327104.227293] RDX: ffff810316e9eac0 RSI: 0000000000000ac0 RDI: 
0000000000000ac0
    [2327104.227297] RBP: ffffffff80293bca R08: 0000000000000008 R09: 
ffff810415c65c70
    [2327104.227301] R10: ffff810316e1f000 R11: ffff8103819dc000 R12: 
ffff810415c65bb8
    [2327104.227306] R13: ffff81041ab46210 R14: ffff810415c65ba0 R15: 
ffff81041ab461e0
    [2327104.227310] FS:  0000000000000000(0000) 
GS:ffff81041ab12300(0000) knlGS:0000000000000000
    [2327104.227315] CS:  0010 DS: 0018 ES: 0018 CR0: 0000000080050033
    [2327104.227317] CR2: 00007ffc53bc5000 CR3: 000000041cd25000 CR4: 
00000000000026e0
    [2327104.227321] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
    [2327104.227325] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
    [2327104.227329]
    [2327104.227330] Call Trace:
    [2327104.227340]  [<ffffffff88091db4>] 
:raid456:compute_parity6+0x1e4/0x380
    [2327104.227361]  [<ffffffff8809335d>] 
:raid456:handle_stripe+0xfbd/0x1090
    [2327104.227387]  [<ffffffff8804ce50>] :md_mod:md_thread+0x0/0x100
    [2327104.227394]  [<ffffffff88093774>] :raid456:raid5d+0x344/0x460
    [2327104.227412]  [<ffffffff8804ce50>] :md_mod:md_thread+0x0/0x100
    [2327104.227422]  [<ffffffff8804ce80>] :md_mod:md_thread+0x30/0x100
    [2327104.227432]  [<ffffffff80256fc0>] autoremove_wake_function+0x0/0x30
    [2327104.227443]  [<ffffffff8804ce50>] :md_mod:md_thread+0x0/0x100
    [2327104.227448]  [<ffffffff80256bfb>] kthread+0x4b/0x80
    [2327104.227456]  [<ffffffff8020d278>] child_rip+0xa/0x12
    [2327104.227469]  [<ffffffff80256bb0>] kthread+0x0/0x80
    [2327104.227474]  [<ffffffff8020d26e>] child_rip+0x0/0x12
    [2327104.227479]


The bug can be workarounded by setting a resync_speed_max which is lower 
than what the hardware is capable of. Or by lowering the 
stripe_cache_size as I said.

Thank you MD people for looking into this problem.


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-21 18:54 ` Majed B.
@ 2009-09-22 14:43   ` Lee Howard
  2009-09-22 14:48     ` Majed B.
  0 siblings, 1 reply; 14+ messages in thread
From: Lee Howard @ 2009-09-22 14:43 UTC (permalink / raw)
  To: Majed B.; +Cc: linux-raid

Thanks Majed,

I've looked at the patch and it modifies md/raid5.c.  However, I'm using 
RAID1.  Am I ignorant for doubting that the patch to raid5.c has any 
affect on RAID1?

Thanks,

Lee.


Majed B. wrote:
> Hello,
>
> Check out the thread with the subject: BUG: soft lockup in [md4_raid5:21137]
>
> I think the guys have already fixed this in 2.6.32. The patch is
> mentioned in one of the replies, if you wish to try it.
>
> On Mon, Sep 21, 2009 at 9:08 PM, Lee Howard <faxguy@howardsilvan.com> wrote:
>   
>> Linux software RAID maintainers and developers,
>>
>> Two months ago I wrote to the Linux kernel mailing list regarding a
>> condition expressed as "BUG: soft lockup - CPU#3 stuck for 61s!".  I
>> initially battled this recurring problem in both Fedora 10 and Fedora 11.
>>  Rafael J. Wysocki suggested that I update the kernel (to 2.6.31-rc4 or
>> later) and see if the problem resurfaced.  I then used kernel
>> 2.6.31-0.94.rc4.fc12.x86_64 and found that the problem still continued, but
>> noticeably only when the md data-check process was run.
>>
>> You can read the last post to the LKML thread (with links to the entire
>> thread) here:
>>
>> http://lkml.org/lkml/2009/8/6/387
>>
>> The md data-check is being run in Fedora 11 by /etc/cron.weekly/raid-check
>> which is a little shell script that looks like this:
>>
>> ------------------------------------------------------
>> #!/bin/bash
>>
>> for dev in `grep "^md.*: active" /proc/mdstat | cut -f 1 -d ' '`; do
>>       [ -f /sys/block/$dev/md/sync_action ] && \
>>               echo "check" > /sys/block/$dev/md/sync_action
>> done
>> ------------------------------------------------------
>>
>> I have disabled this weekly data-check, and since doing so have not
>> encountered any soft lockup (or any other problem, for that matter).
>>
>> For reference, you can see in this thread here that I am not the only one to
>> have this problem:
>>
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/212684
>>
>> It may important to note that I am running a 4-disk RAID1 array on 500GB
>> Western-Digial SATA drives.  I've attached the output of 'lspci -vv' as
>> "lspci.out".  I've attached the output of 'hdparm -I /dev/sda' as
>> "hdparm.out".  /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd are all identical
>> drive types.  I've attached the output of 'for i in 0 1 2; do mdadm --detail
>> /dev/md$i; done' as "mdadm.out".
>>
>> Please let me know if this is a configuration problem, a kernel bug, or
>> something else.  Please let me know how to fix this problem so that I can
>> safely re-enable md data-check.
>>
>> Thanks,
>>
>> Lee.
>>
>> 00:00.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a2)
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR+ FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0
>>        Capabilities: [44] HyperTransport: Slave or Primary Interface
>>                Command: BaseUnitID=0 UnitCnt=15 MastHost- DefDir- DUL-
>>                Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO-
>> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>>                Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut-
>> LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
>>                Link Control 1: CFlE- CST- CFE- <LkFail+ Init- EOC+ TXO+
>> <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
>>                Link Config 1: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut- LWI=8bit
>> DwFcInEn- LWO=8bit DwFcOutEn-
>>                Revision ID: 1.03
>>                Link Frequency 0: 1.0GHz
>>                Link Error 0: <Prot- <Ovfl- <EOC- CTLTm-
>>                Link Frequency Capability 0: 200MHz+ 300MHz+ 400MHz+ 500MHz+
>> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>>                Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA-
>> UIDRD-
>>                Link Frequency 1: 200MHz
>>                Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
>>                Link Frequency Capability 1: 200MHz- 300MHz- 400MHz- 500MHz-
>> 600MHz- 800MHz- 1.0GHz- 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>>                Error Handling: PFlE+ OFlE+ PFE- OFE- EOCFE- RFE- CRCFE-
>> SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
>>                Prefetchable memory behind bridge Upper: 00-00
>>                Bus Number: 00
>>        Capabilities: [dc] HyperTransport: MSI Mapping Enable+ Fixed-
>>                Mapping Address Base: 00000000fee00000
>>
>> 00:01.0 ISA bridge: nVidia Corporation MCP55 LPC Bridge (rev a3)
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0
>>        Region 0: I/O ports at 2f00 [size=128]
>>
>> 00:01.1 SMBus: nVidia Corporation MCP55 SMBus (rev a3)
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Interrupt: pin A routed to IRQ 10
>>        Region 0: I/O ports at dc00 [size=64]
>>        Region 4: I/O ports at 2d00 [size=64]
>>        Region 5: I/O ports at 2e00 [size=64]
>>        Capabilities: [44] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Kernel driver in use: nForce2_smbus
>>        Kernel modules: i2c-nforce2
>>
>> 00:02.0 USB Controller: nVidia Corporation MCP55 USB Controller (rev a1)
>> (prog-if 10 [OHCI])
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0 (750ns min, 250ns max)
>>        Interrupt: pin A routed to IRQ 22
>>        Region 0: Memory at feafb000 (32-bit, non-prefetchable) [size=4K]
>>        Capabilities: [44] Power Management version 2
>>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Kernel driver in use: ohci_hcd
>>
>> 00:02.1 USB Controller: nVidia Corporation MCP55 USB Controller (rev a2)
>> (prog-if 20 [EHCI])
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0 (750ns min, 250ns max)
>>        Interrupt: pin B routed to IRQ 23
>>        Region 0: Memory at feafac00 (32-bit, non-prefetchable) [size=256]
>>        Capabilities: [44] Debug port: BAR=1 offset=0098
>>        Capabilities: [80] Power Management version 2
>>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Kernel driver in use: ehci_hcd
>>
>> 00:04.0 IDE interface: nVidia Corporation MCP55 IDE (rev a1) (prog-if 8a
>> [Master SecP PriP])
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0 (750ns min, 250ns max)
>>        Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable)
>> [size=8]
>>        Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
>> [size=1]
>>        Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable)
>> [size=8]
>>        Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
>> [size=1]
>>        Region 4: I/O ports at ffa0 [size=16]
>>        Capabilities: [44] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Kernel driver in use: pata_amd
>>        Kernel modules: ata_generic, pata_acpi, pata_amd
>>
>> 00:05.0 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
>> (prog-if 85 [Master SecO PriO])
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0 (750ns min, 250ns max)
>>        Interrupt: pin A routed to IRQ 21
>>        Region 0: I/O ports at d480 [size=8]
>>        Region 1: I/O ports at d400 [size=4]
>>        Region 2: I/O ports at d080 [size=8]
>>        Region 3: I/O ports at d000 [size=4]
>>        Region 4: I/O ports at cc00 [size=16]
>>        Region 5: Memory at feaf9000 (32-bit, non-prefetchable) [size=4K]
>>        Capabilities: [44] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>                Address: 0000000000000000  Data: 0000
>>        Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>        Kernel driver in use: sata_nv
>>        Kernel modules: ata_generic, pata_acpi, sata_nv
>>
>> 00:05.1 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
>> (prog-if 85 [Master SecO PriO])
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0 (750ns min, 250ns max)
>>        Interrupt: pin B routed to IRQ 20
>>        Region 0: I/O ports at c880 [size=8]
>>        Region 1: I/O ports at c800 [size=4]
>>        Region 2: I/O ports at c480 [size=8]
>>        Region 3: I/O ports at c400 [size=4]
>>        Region 4: I/O ports at c080 [size=16]
>>        Region 5: Memory at feaf8000 (32-bit, non-prefetchable) [size=4K]
>>        Capabilities: [44] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>                Address: 0000000000000000  Data: 0000
>>        Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>        Kernel driver in use: sata_nv
>>        Kernel modules: ata_generic, pata_acpi, sata_nv
>>
>> 00:05.2 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
>> (prog-if 85 [Master SecO PriO])
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0 (750ns min, 250ns max)
>>        Interrupt: pin C routed to IRQ 23
>>        Region 0: I/O ports at c000 [size=8]
>>        Region 1: I/O ports at bc00 [size=4]
>>        Region 2: I/O ports at b880 [size=8]
>>        Region 3: I/O ports at b800 [size=4]
>>        Region 4: I/O ports at b480 [size=16]
>>        Region 5: Memory at feaf7000 (32-bit, non-prefetchable) [size=4K]
>>        Capabilities: [44] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>                Address: 0000000000000000  Data: 0000
>>        Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>        Kernel driver in use: sata_nv
>>        Kernel modules: ata_generic, pata_acpi, sata_nv
>>
>> 00:06.0 PCI bridge: nVidia Corporation MCP55 PCI bridge (rev a2) (prog-if 01
>> [Subtractive decode])
>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0
>>        Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>>        I/O behind bridge: 0000e000-0000efff
>>        Memory behind bridge: feb00000-febfffff
>>        Prefetchable memory behind bridge: f8000000-fbffffff
>>        Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort-
>> <TAbort- <MAbort- <SERR- <PERR-
>>        BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
>>                PriDiscTmr- SecDiscTmr+ DiscTmrStat- DiscTmrSERREn-
>>        Capabilities: [b8] Subsystem: nVidia Corporation Device cb84
>>        Capabilities: [8c] HyperTransport: MSI Mapping Enable- Fixed-
>>                Mapping Address Base: 00000000fee00000
>>
>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx+
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0 (250ns min, 5000ns max)
>>        Interrupt: pin A routed to IRQ 30
>>        Region 0: Memory at feaf6000 (32-bit, non-prefetchable) [size=4K]
>>        Region 1: I/O ports at b400 [size=8]
>>        Region 2: Memory at feafa800 (32-bit, non-prefetchable) [size=256]
>>        Region 3: Memory at feafa400 (32-bit, non-prefetchable) [size=16]
>>        Capabilities: [44] Power Management version 2
>>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>>        Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>>                Vector table: BAR=2 offset=00000000
>>                PBA: BAR=3 offset=00000000
>>        Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>>                Address: 00000000fee0400c  Data: 4199
>>                Masking: 000000fe  Pending: 00000000
>>        Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>>        Kernel driver in use: forcedeth
>>        Kernel modules: forcedeth
>>
>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>>        Subsystem: nVidia Corporation Device cb84
>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx+
>>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0 (250ns min, 5000ns max)
>>        Interrupt: pin A routed to IRQ 31
>>        Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
>>        Region 1: I/O ports at b080 [size=8]
>>        Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256]
>>        Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16]
>>        Capabilities: [44] Power Management version 2
>>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>>        Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>>                Vector table: BAR=2 offset=00000000
>>                PBA: BAR=3 offset=00000000
>>        Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>>                Address: 00000000fee0200c  Data: 41a1
>>                Masking: 000000fe  Pending: 00000000
>>        Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>>        Kernel driver in use: forcedeth
>>        Kernel modules: forcedeth
>>
>> 00:0a.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>> (prog-if 00 [Normal decode])
>>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx+
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0, Cache Line Size: 64 bytes
>>        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- <SERR- <PERR-
>>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>        Capabilities: [48] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>                Address: 00000000fee0f00c  Data: 4149
>>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>                Mapping Address Base: 00000000fee00000
>>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
>> L1 <1us
>>                        ExtTag- RBE+ FLReset-
>>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>> TransPend-
>>                LnkCap: Port #5, Speed 2.5GT/s, Width x8, ASPM L0s L1,
>> Latency L0 <512ns, L1 <4us
>>                        ClockPM- Surprise- LLActRep+ BwNot-
>>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>> CommClk-
>>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
>> DLActive- BWMgmt- ABWMgmt-
>>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>> Surpise-
>>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>> HPIrq- LinkChg-
>>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>> Interlock-
>>                        Changed: MRL- PresDet- LinkState-
>>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>> CRSVisible-
>>                RootCap: CRSVisible-
>>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>        Capabilities: [100] Virtual Channel <?>
>>        Kernel driver in use: pcieport-driver
>>        Kernel modules: shpchp
>>
>> 00:0b.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>> (prog-if 00 [Normal decode])
>>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx+
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0, Cache Line Size: 64 bytes
>>        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
>>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- <SERR- <PERR-
>>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>        Capabilities: [48] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>                Address: 00000000fee0f00c  Data: 4151
>>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>                Mapping Address Base: 00000000fee00000
>>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
>> L1 <1us
>>                        ExtTag- RBE+ FLReset-
>>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>> TransPend-
>>                LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>> Latency L0 <512ns, L1 <4us
>>                        ClockPM- Surprise- LLActRep+ BwNot-
>>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>> CommClk-
>>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>> DLActive- BWMgmt- ABWMgmt-
>>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>> Surpise-
>>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>> HPIrq- LinkChg-
>>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>> Interlock-
>>                        Changed: MRL- PresDet- LinkState-
>>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>> CRSVisible-
>>                RootCap: CRSVisible-
>>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>        Capabilities: [100] Virtual Channel <?>
>>        Kernel driver in use: pcieport-driver
>>        Kernel modules: shpchp
>>
>> 00:0c.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>> (prog-if 00 [Normal decode])
>>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx+
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0, Cache Line Size: 64 bytes
>>        Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
>>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- <SERR- <PERR-
>>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>        Capabilities: [48] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>                Address: 00000000fee0f00c  Data: 4159
>>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>                Mapping Address Base: 00000000fee00000
>>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
>> L1 <1us
>>                        ExtTag- RBE+ FLReset-
>>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>> TransPend-
>>                LnkCap: Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>> Latency L0 <512ns, L1 <4us
>>                        ClockPM- Surprise- LLActRep+ BwNot-
>>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>> CommClk-
>>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>> DLActive- BWMgmt- ABWMgmt-
>>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>> Surpise-
>>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>> HPIrq- LinkChg-
>>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>> Interlock-
>>                        Changed: MRL- PresDet- LinkState-
>>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>> CRSVisible-
>>                RootCap: CRSVisible-
>>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>        Capabilities: [100] Virtual Channel <?>
>>        Kernel driver in use: pcieport-driver
>>        Kernel modules: shpchp
>>
>> 00:0d.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>> (prog-if 00 [Normal decode])
>>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx+
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0, Cache Line Size: 64 bytes
>>        Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
>>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- <SERR- <PERR-
>>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>        Capabilities: [48] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>                Address: 00000000fee0f00c  Data: 4161
>>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>                Mapping Address Base: 00000000fee00000
>>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
>> L1 <1us
>>                        ExtTag- RBE+ FLReset-
>>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>> TransPend-
>>                LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>> Latency L0 <512ns, L1 <4us
>>                        ClockPM- Surprise- LLActRep+ BwNot-
>>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>> CommClk-
>>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>> DLActive- BWMgmt- ABWMgmt-
>>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>> Surpise-
>>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>> HPIrq- LinkChg-
>>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>> Interlock-
>>                        Changed: MRL- PresDet- LinkState-
>>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>> CRSVisible-
>>                RootCap: CRSVisible-
>>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>        Capabilities: [100] Virtual Channel <?>
>>        Kernel driver in use: pcieport-driver
>>        Kernel modules: shpchp
>>
>> 00:0e.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>> (prog-if 00 [Normal decode])
>>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx+
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0, Cache Line Size: 64 bytes
>>        Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
>>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- <SERR- <PERR-
>>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>        Capabilities: [48] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>                Address: 00000000fee0f00c  Data: 4169
>>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>                Mapping Address Base: 00000000fee00000
>>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
>> L1 <1us
>>                        ExtTag- RBE+ FLReset-
>>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>> TransPend-
>>                LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>> Latency L0 <512ns, L1 <4us
>>                        ClockPM- Surprise- LLActRep+ BwNot-
>>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>> CommClk-
>>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
>> DLActive- BWMgmt- ABWMgmt-
>>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>> Surpise-
>>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>> HPIrq- LinkChg-
>>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>> Interlock-
>>                        Changed: MRL- PresDet- LinkState-
>>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>> CRSVisible-
>>                RootCap: CRSVisible-
>>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>        Capabilities: [100] Virtual Channel <?>
>>        Kernel driver in use: pcieport-driver
>>        Kernel modules: shpchp
>>
>> 00:0f.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>> (prog-if 00 [Normal decode])
>>        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx+
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 0, Cache Line Size: 64 bytes
>>        Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
>>        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- <SERR- <PERR-
>>        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>        Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>        Capabilities: [48] Power Management version 2
>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>        Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>                Address: 00000000fee0f00c  Data: 4171
>>        Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>                Mapping Address Base: 00000000fee00000
>>        Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns,
>> L1 <1us
>>                        ExtTag- RBE+ FLReset-
>>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>                        MaxPayload 128 bytes, MaxReadReq 512 bytes
>>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>> TransPend-
>>                LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1,
>> Latency L0 <512ns, L1 <4us
>>                        ClockPM- Surprise- LLActRep+ BwNot-
>>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>> CommClk-
>>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>                LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+
>> DLActive- BWMgmt- ABWMgmt-
>>                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>> Surpise-
>>                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>> HPIrq- LinkChg-
>>                        Control: AttnInd Off, PwrInd On, Power- Interlock-
>>                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>> Interlock-
>>                        Changed: MRL- PresDet- LinkState-
>>                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>> CRSVisible-
>>                RootCap: CRSVisible-
>>                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>        Capabilities: [100] Virtual Channel <?>
>>        Kernel driver in use: pcieport-driver
>>        Kernel modules: shpchp
>>
>> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] HyperTransport Configuration
>>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Capabilities: [80] HyperTransport: Host or Secondary Interface
>>                Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+
>> Slave- <EOCErr- DUL-
>>                Link Control: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO-
>> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>>                Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit
>> DwFcInEn- LWO=16bit DwFcOutEn-
>>                Revision ID: 3.00
>>                Link Frequency: 1.0GHz
>>                Link Error: <Prot- <Ovfl- <EOC- CTLTm-
>>                Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz-
>> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz+ 1.4GHz- 1.6GHz- Vend-
>>                Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+
>> UIDRD- ExtRS- UCnfE-
>>
>> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] Address Map
>>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>
>> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] DRAM Controller
>>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Kernel modules: amd64_edac_mod
>>
>> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] Miscellaneous Control
>>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Capabilities: [f0] Secure device <?>
>>
>> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] Link Control
>>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>
>> 01:0a.0 VGA compatible controller: XGI Technology Inc. (eXtreme Graphics
>> Innovation) Volari Z7/Z9/Z9s (prog-if 00 [VGA controller])
>>        Subsystem: XGI Technology Inc. (eXtreme Graphics Innovation) Volari
>> Z7/Z9/Z9s
>>        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        BIST result: 00
>>        Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
>>        Region 1: Memory at febc0000 (32-bit, non-prefetchable) [size=256K]
>>        Region 2: I/O ports at ec00 [size=128]
>>        Expansion ROM at <unassigned> [disabled]
>>        Capabilities: [40] Power Management version 2
>>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>
>>
>>
>> /dev/sda:
>>
>> ATA device, with non-removable media
>>        Model Number:       WDC WD5000AAKS-00A7B2
>>        Serial Number:      WD-WCASZ0321108
>>        Firmware Revision:  01.03B01
>>        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev
>> 2.5
>> Standards:
>>        Supported: 8 7 6 5
>>        Likely used: 8
>> Configuration:
>>        Logical         max     current
>>        cylinders       16383   16383
>>        heads           16      16
>>        sectors/track   63      63
>>        --
>>        CHS current addressable sectors:   16514064
>>        LBA    user addressable sectors:  268435455
>>        LBA48  user addressable sectors:  976773168
>>        Logical/Physical Sector size:           512 bytes
>>        device size with M = 1024*1024:      476940 MBytes
>>        device size with M = 1000*1000:      500107 MBytes (500 GB)
>>        cache/buffer size  = 16384 KBytes
>> Capabilities:
>>        LBA, IORDY(can be disabled)
>>        Queue depth: 32
>>        Standby timer values: spec'd by Standard, with device specific
>> minimum
>>        R/W multiple sector transfer: Max = 16  Current = 16
>>        Recommended acoustic management value: 128, current value: 254
>>        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
>>             Cycle time: min=120ns recommended=120ns
>>        PIO: pio0 pio1 pio2 pio3 pio4
>>             Cycle time: no flow control=120ns  IORDY flow control=120ns
>> Commands/features:
>>        Enabled Supported:
>>           *    SMART feature set
>>                Security Mode feature set
>>           *    Power Management feature set
>>           *    Write cache
>>           *    Look-ahead
>>           *    Host Protected Area feature set
>>           *    WRITE_BUFFER command
>>           *    READ_BUFFER command
>>           *    NOP cmd
>>           *    DOWNLOAD_MICROCODE
>>                Power-Up In Standby feature set
>>           *    SET_FEATURES required to spinup after power up
>>                SET_MAX security extension
>>                Automatic Acoustic Management feature set
>>           *    48-bit Address feature set
>>           *    Device Configuration Overlay feature set
>>           *    Mandatory FLUSH_CACHE
>>           *    FLUSH_CACHE_EXT
>>           *    SMART error logging
>>           *    SMART self-test
>>           *    General Purpose Logging feature set
>>           *    64-bit World wide name
>>           *    Segmented DOWNLOAD_MICROCODE
>>           *    Gen1 signaling speed (1.5Gb/s)
>>           *    Gen2 signaling speed (3.0Gb/s)
>>           *    Native Command Queueing (NCQ)
>>           *    Host-initiated interface power management
>>           *    Phy event counters
>>                DMA Setup Auto-Activate optimization
>>           *    Software settings preservation
>>           *    SMART Command Transport (SCT) feature set
>>           *    SCT Long Sector Access (AC1)
>>           *    SCT LBA Segment Access (AC2)
>>           *    SCT Error Recovery Control (AC3)
>>           *    SCT Features Control (AC4)
>>           *    SCT Data Tables (AC5)
>>                unknown 206[12] (vendor specific)
>>                unknown 206[13] (vendor specific)
>> Security:
>>        Master password revision code = 65534
>>                supported
>>        not     enabled
>>        not     locked
>>                frozen
>>        not     expired: security count
>>                supported: enhanced erase
>>        84min for SECURITY ERASE UNIT. 84min for ENHANCED SECURITY ERASE
>> UNIT.
>> Logical Unit WWN Device Identifier: 50014ee101cedc24
>>        NAA             : 5
>>        IEEE OUI        : 0014ee
>>        Unique ID       : 101cedc24
>> Checksum: correct
>>
>> /dev/md0:
>>        Version : 0.90
>>  Creation Time : Mon Jul 27 22:38:26 2009
>>     Raid Level : raid1
>>     Array Size : 102336 (99.95 MiB 104.79 MB)
>>  Used Dev Size : 102336 (99.95 MiB 104.79 MB)
>>   Raid Devices : 4
>>  Total Devices : 4
>> Preferred Minor : 0
>>    Persistence : Superblock is persistent
>>
>>    Update Time : Mon Sep 21 11:02:14 2009
>>          State : clean
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 0
>>  Spare Devices : 0
>>
>>           UUID : 9037def2:035b0d10:bfe78010:bc810f04
>>         Events : 0.44
>>
>>    Number   Major   Minor   RaidDevice State
>>       0       8        1        0      active sync   /dev/sda1
>>       1       8       17        1      active sync   /dev/sdb1
>>       2       8       33        2      active sync   /dev/sdc1
>>       3       8       49        3      active sync   /dev/sdd1
>> /dev/md1:
>>        Version : 0.90
>>  Creation Time : Mon Jul 27 22:38:28 2009
>>     Raid Level : raid1
>>     Array Size : 4095936 (3.91 GiB 4.19 GB)
>>  Used Dev Size : 4095936 (3.91 GiB 4.19 GB)
>>   Raid Devices : 4
>>  Total Devices : 4
>> Preferred Minor : 1
>>    Persistence : Superblock is persistent
>>
>>    Update Time : Mon Sep 21 11:02:14 2009
>>          State : clean
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 0
>>  Spare Devices : 0
>>
>>           UUID : 642c2833:bf033a7d:bfe78010:bc810f04
>>         Events : 0.44
>>
>>    Number   Major   Minor   RaidDevice State
>>       0       8        2        0      active sync   /dev/sda2
>>       1       8       18        1      active sync   /dev/sdb2
>>       2       8       34        2      active sync   /dev/sdc2
>>       3       8       50        3      active sync   /dev/sdd2
>> /dev/md2:
>>        Version : 0.90
>>  Creation Time : Mon Jul 27 22:38:30 2009
>>     Raid Level : raid1
>>     Array Size : 484185536 (461.76 GiB 495.81 GB)
>>  Used Dev Size : 484185536 (461.76 GiB 495.81 GB)
>>   Raid Devices : 4
>>  Total Devices : 4
>> Preferred Minor : 2
>>    Persistence : Superblock is persistent
>>
>>    Update Time : Mon Sep 21 11:02:24 2009
>>          State : clean
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 0
>>  Spare Devices : 0
>>
>>           UUID : ce106d52:0c0373f7:bfe78010:bc810f04
>>         Events : 0.66
>>
>>    Number   Major   Minor   RaidDevice State
>>       0       8        3        0      active sync   /dev/sda3
>>       1       8       19        1      active sync   /dev/sdb3
>>       2       8       35        2      active sync   /dev/sdc3
>>       3       8       51        3      active sync   /dev/sdd3
>>
>>
>>     
>
>
>
>   


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-22 14:43   ` Lee Howard
@ 2009-09-22 14:48     ` Majed B.
  2009-09-22 14:59       ` Lee Howard
  0 siblings, 1 reply; 14+ messages in thread
From: Majed B. @ 2009-09-22 14:48 UTC (permalink / raw)
  To: Lee Howard; +Cc: linux-raid

I must have missed that part. It may not work for your case, but worth trying.

Perhaps Neil Brown, or someone involved could shed some light on this.

If I remember correctly, those soft lockups were harmless anyway.

On Tue, Sep 22, 2009 at 5:43 PM, Lee Howard <faxguy@howardsilvan.com> wrote:
> Thanks Majed,
>
> I've looked at the patch and it modifies md/raid5.c.  However, I'm using
> RAID1.  Am I ignorant for doubting that the patch to raid5.c has any affect
> on RAID1?
>
> Thanks,
>
> Lee.
>
>
> Majed B. wrote:
>>
>> Hello,
>>
>> Check out the thread with the subject: BUG: soft lockup in
>> [md4_raid5:21137]
>>
>> I think the guys have already fixed this in 2.6.32. The patch is
>> mentioned in one of the replies, if you wish to try it.
>>
>> On Mon, Sep 21, 2009 at 9:08 PM, Lee Howard <faxguy@howardsilvan.com>
>> wrote:
>>
>>>
>>> Linux software RAID maintainers and developers,
>>>
>>> Two months ago I wrote to the Linux kernel mailing list regarding a
>>> condition expressed as "BUG: soft lockup - CPU#3 stuck for 61s!".  I
>>> initially battled this recurring problem in both Fedora 10 and Fedora 11.
>>>  Rafael J. Wysocki suggested that I update the kernel (to 2.6.31-rc4 or
>>> later) and see if the problem resurfaced.  I then used kernel
>>> 2.6.31-0.94.rc4.fc12.x86_64 and found that the problem still continued,
>>> but
>>> noticeably only when the md data-check process was run.
>>>
>>> You can read the last post to the LKML thread (with links to the entire
>>> thread) here:
>>>
>>> http://lkml.org/lkml/2009/8/6/387
>>>
>>> The md data-check is being run in Fedora 11 by
>>> /etc/cron.weekly/raid-check
>>> which is a little shell script that looks like this:
>>>
>>> ------------------------------------------------------
>>> #!/bin/bash
>>>
>>> for dev in `grep "^md.*: active" /proc/mdstat | cut -f 1 -d ' '`; do
>>>      [ -f /sys/block/$dev/md/sync_action ] && \
>>>              echo "check" > /sys/block/$dev/md/sync_action
>>> done
>>> ------------------------------------------------------
>>>
>>> I have disabled this weekly data-check, and since doing so have not
>>> encountered any soft lockup (or any other problem, for that matter).
>>>
>>> For reference, you can see in this thread here that I am not the only one
>>> to
>>> have this problem:
>>>
>>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/212684
>>>
>>> It may important to note that I am running a 4-disk RAID1 array on 500GB
>>> Western-Digial SATA drives.  I've attached the output of 'lspci -vv' as
>>> "lspci.out".  I've attached the output of 'hdparm -I /dev/sda' as
>>> "hdparm.out".  /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd are all
>>> identical
>>> drive types.  I've attached the output of 'for i in 0 1 2; do mdadm
>>> --detail
>>> /dev/md$i; done' as "mdadm.out".
>>>
>>> Please let me know if this is a configuration problem, a kernel bug, or
>>> something else.  Please let me know how to fix this problem so that I can
>>> safely re-enable md data-check.
>>>
>>> Thanks,
>>>
>>> Lee.
>>>
>>> 00:00.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a2)
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR+ FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0
>>>       Capabilities: [44] HyperTransport: Slave or Primary Interface
>>>               Command: BaseUnitID=0 UnitCnt=15 MastHost- DefDir- DUL-
>>>               Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO-
>>> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>>>               Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut-
>>> LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
>>>               Link Control 1: CFlE- CST- CFE- <LkFail+ Init- EOC+ TXO+
>>> <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
>>>               Link Config 1: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut-
>>> LWI=8bit
>>> DwFcInEn- LWO=8bit DwFcOutEn-
>>>               Revision ID: 1.03
>>>               Link Frequency 0: 1.0GHz
>>>               Link Error 0: <Prot- <Ovfl- <EOC- CTLTm-
>>>               Link Frequency Capability 0: 200MHz+ 300MHz+ 400MHz+
>>> 500MHz+
>>> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>>>               Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA-
>>> UIDRD-
>>>               Link Frequency 1: 200MHz
>>>               Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
>>>               Link Frequency Capability 1: 200MHz- 300MHz- 400MHz-
>>> 500MHz-
>>> 600MHz- 800MHz- 1.0GHz- 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>>>               Error Handling: PFlE+ OFlE+ PFE- OFE- EOCFE- RFE- CRCFE-
>>> SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
>>>               Prefetchable memory behind bridge Upper: 00-00
>>>               Bus Number: 00
>>>       Capabilities: [dc] HyperTransport: MSI Mapping Enable+ Fixed-
>>>               Mapping Address Base: 00000000fee00000
>>>
>>> 00:01.0 ISA bridge: nVidia Corporation MCP55 LPC Bridge (rev a3)
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0
>>>       Region 0: I/O ports at 2f00 [size=128]
>>>
>>> 00:01.1 SMBus: nVidia Corporation MCP55 SMBus (rev a3)
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Interrupt: pin A routed to IRQ 10
>>>       Region 0: I/O ports at dc00 [size=64]
>>>       Region 4: I/O ports at 2d00 [size=64]
>>>       Region 5: I/O ports at 2e00 [size=64]
>>>       Capabilities: [44] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Kernel driver in use: nForce2_smbus
>>>       Kernel modules: i2c-nforce2
>>>
>>> 00:02.0 USB Controller: nVidia Corporation MCP55 USB Controller (rev a1)
>>> (prog-if 10 [OHCI])
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0 (750ns min, 250ns max)
>>>       Interrupt: pin A routed to IRQ 22
>>>       Region 0: Memory at feafb000 (32-bit, non-prefetchable) [size=4K]
>>>       Capabilities: [44] Power Management version 2
>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Kernel driver in use: ohci_hcd
>>>
>>> 00:02.1 USB Controller: nVidia Corporation MCP55 USB Controller (rev a2)
>>> (prog-if 20 [EHCI])
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0 (750ns min, 250ns max)
>>>       Interrupt: pin B routed to IRQ 23
>>>       Region 0: Memory at feafac00 (32-bit, non-prefetchable) [size=256]
>>>       Capabilities: [44] Debug port: BAR=1 offset=0098
>>>       Capabilities: [80] Power Management version 2
>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Kernel driver in use: ehci_hcd
>>>
>>> 00:04.0 IDE interface: nVidia Corporation MCP55 IDE (rev a1) (prog-if 8a
>>> [Master SecP PriP])
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0 (750ns min, 250ns max)
>>>       Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable)
>>> [size=8]
>>>       Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
>>> [size=1]
>>>       Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable)
>>> [size=8]
>>>       Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
>>> [size=1]
>>>       Region 4: I/O ports at ffa0 [size=16]
>>>       Capabilities: [44] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Kernel driver in use: pata_amd
>>>       Kernel modules: ata_generic, pata_acpi, pata_amd
>>>
>>> 00:05.0 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
>>> (prog-if 85 [Master SecO PriO])
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0 (750ns min, 250ns max)
>>>       Interrupt: pin A routed to IRQ 21
>>>       Region 0: I/O ports at d480 [size=8]
>>>       Region 1: I/O ports at d400 [size=4]
>>>       Region 2: I/O ports at d080 [size=8]
>>>       Region 3: I/O ports at d000 [size=4]
>>>       Region 4: I/O ports at cc00 [size=16]
>>>       Region 5: Memory at feaf9000 (32-bit, non-prefetchable) [size=4K]
>>>       Capabilities: [44] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>>               Address: 0000000000000000  Data: 0000
>>>       Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>>       Kernel driver in use: sata_nv
>>>       Kernel modules: ata_generic, pata_acpi, sata_nv
>>>
>>> 00:05.1 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
>>> (prog-if 85 [Master SecO PriO])
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0 (750ns min, 250ns max)
>>>       Interrupt: pin B routed to IRQ 20
>>>       Region 0: I/O ports at c880 [size=8]
>>>       Region 1: I/O ports at c800 [size=4]
>>>       Region 2: I/O ports at c480 [size=8]
>>>       Region 3: I/O ports at c400 [size=4]
>>>       Region 4: I/O ports at c080 [size=16]
>>>       Region 5: Memory at feaf8000 (32-bit, non-prefetchable) [size=4K]
>>>       Capabilities: [44] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>>               Address: 0000000000000000  Data: 0000
>>>       Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>>       Kernel driver in use: sata_nv
>>>       Kernel modules: ata_generic, pata_acpi, sata_nv
>>>
>>> 00:05.2 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
>>> (prog-if 85 [Master SecO PriO])
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0 (750ns min, 250ns max)
>>>       Interrupt: pin C routed to IRQ 23
>>>       Region 0: I/O ports at c000 [size=8]
>>>       Region 1: I/O ports at bc00 [size=4]
>>>       Region 2: I/O ports at b880 [size=8]
>>>       Region 3: I/O ports at b800 [size=4]
>>>       Region 4: I/O ports at b480 [size=16]
>>>       Region 5: Memory at feaf7000 (32-bit, non-prefetchable) [size=4K]
>>>       Capabilities: [44] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>>               Address: 0000000000000000  Data: 0000
>>>       Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>>       Kernel driver in use: sata_nv
>>>       Kernel modules: ata_generic, pata_acpi, sata_nv
>>>
>>> 00:06.0 PCI bridge: nVidia Corporation MCP55 PCI bridge (rev a2) (prog-if
>>> 01
>>> [Subtractive decode])
>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0
>>>       Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>>>       I/O behind bridge: 0000e000-0000efff
>>>       Memory behind bridge: feb00000-febfffff
>>>       Prefetchable memory behind bridge: f8000000-fbffffff
>>>       Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort-
>>> <TAbort- <MAbort- <SERR- <PERR-
>>>       BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
>>>               PriDiscTmr- SecDiscTmr+ DiscTmrStat- DiscTmrSERREn-
>>>       Capabilities: [b8] Subsystem: nVidia Corporation Device cb84
>>>       Capabilities: [8c] HyperTransport: MSI Mapping Enable- Fixed-
>>>               Mapping Address Base: 00000000fee00000
>>>
>>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx+
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0 (250ns min, 5000ns max)
>>>       Interrupt: pin A routed to IRQ 30
>>>       Region 0: Memory at feaf6000 (32-bit, non-prefetchable) [size=4K]
>>>       Region 1: I/O ports at b400 [size=8]
>>>       Region 2: Memory at feafa800 (32-bit, non-prefetchable) [size=256]
>>>       Region 3: Memory at feafa400 (32-bit, non-prefetchable) [size=16]
>>>       Capabilities: [44] Power Management version 2
>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>>>       Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>>>               Vector table: BAR=2 offset=00000000
>>>               PBA: BAR=3 offset=00000000
>>>       Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>>>               Address: 00000000fee0400c  Data: 4199
>>>               Masking: 000000fe  Pending: 00000000
>>>       Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>>>       Kernel driver in use: forcedeth
>>>       Kernel modules: forcedeth
>>>
>>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>>>       Subsystem: nVidia Corporation Device cb84
>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx+
>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0 (250ns min, 5000ns max)
>>>       Interrupt: pin A routed to IRQ 31
>>>       Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
>>>       Region 1: I/O ports at b080 [size=8]
>>>       Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256]
>>>       Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16]
>>>       Capabilities: [44] Power Management version 2
>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>>>       Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>>>               Vector table: BAR=2 offset=00000000
>>>               PBA: BAR=3 offset=00000000
>>>       Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>>>               Address: 00000000fee0200c  Data: 41a1
>>>               Masking: 000000fe  Pending: 00000000
>>>       Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>>>       Kernel driver in use: forcedeth
>>>       Kernel modules: forcedeth
>>>
>>> 00:0a.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>> (prog-if 00 [Normal decode])
>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx+
>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0, Cache Line Size: 64 bytes
>>>       Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- <SERR- <PERR-
>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>       Capabilities: [48] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>               Address: 00000000fee0f00c  Data: 4149
>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>               Mapping Address Base: 00000000fee00000
>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>> <64ns,
>>> L1 <1us
>>>                       ExtTag- RBE+ FLReset-
>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>> Unsupported-
>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>> TransPend-
>>>               LnkCap: Port #5, Speed 2.5GT/s, Width x8, ASPM L0s L1,
>>> Latency L0 <512ns, L1 <4us
>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>> CommClk-
>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>               LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
>>> DLActive- BWMgmt- ABWMgmt-
>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>> Surpise-
>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>> HPIrq- LinkChg-
>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>> Interlock-
>>>                       Changed: MRL- PresDet- LinkState-
>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>> CRSVisible-
>>>               RootCap: CRSVisible-
>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>       Capabilities: [100] Virtual Channel <?>
>>>       Kernel driver in use: pcieport-driver
>>>       Kernel modules: shpchp
>>>
>>> 00:0b.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>> (prog-if 00 [Normal decode])
>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx+
>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0, Cache Line Size: 64 bytes
>>>       Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- <SERR- <PERR-
>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>       Capabilities: [48] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>               Address: 00000000fee0f00c  Data: 4151
>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>               Mapping Address Base: 00000000fee00000
>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>> <64ns,
>>> L1 <1us
>>>                       ExtTag- RBE+ FLReset-
>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>> Unsupported-
>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>> TransPend-
>>>               LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>> Latency L0 <512ns, L1 <4us
>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>> CommClk-
>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>>> DLActive- BWMgmt- ABWMgmt-
>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>> Surpise-
>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>> HPIrq- LinkChg-
>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>> Interlock-
>>>                       Changed: MRL- PresDet- LinkState-
>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>> CRSVisible-
>>>               RootCap: CRSVisible-
>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>       Capabilities: [100] Virtual Channel <?>
>>>       Kernel driver in use: pcieport-driver
>>>       Kernel modules: shpchp
>>>
>>> 00:0c.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>> (prog-if 00 [Normal decode])
>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx+
>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0, Cache Line Size: 64 bytes
>>>       Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- <SERR- <PERR-
>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>       Capabilities: [48] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>               Address: 00000000fee0f00c  Data: 4159
>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>               Mapping Address Base: 00000000fee00000
>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>> <64ns,
>>> L1 <1us
>>>                       ExtTag- RBE+ FLReset-
>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>> Unsupported-
>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>> TransPend-
>>>               LnkCap: Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>> Latency L0 <512ns, L1 <4us
>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>> CommClk-
>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>>> DLActive- BWMgmt- ABWMgmt-
>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>> Surpise-
>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>> HPIrq- LinkChg-
>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>> Interlock-
>>>                       Changed: MRL- PresDet- LinkState-
>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>> CRSVisible-
>>>               RootCap: CRSVisible-
>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>       Capabilities: [100] Virtual Channel <?>
>>>       Kernel driver in use: pcieport-driver
>>>       Kernel modules: shpchp
>>>
>>> 00:0d.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>> (prog-if 00 [Normal decode])
>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx+
>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0, Cache Line Size: 64 bytes
>>>       Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- <SERR- <PERR-
>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>       Capabilities: [48] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>               Address: 00000000fee0f00c  Data: 4161
>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>               Mapping Address Base: 00000000fee00000
>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>> <64ns,
>>> L1 <1us
>>>                       ExtTag- RBE+ FLReset-
>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>> Unsupported-
>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>> TransPend-
>>>               LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>> Latency L0 <512ns, L1 <4us
>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>> CommClk-
>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>>> DLActive- BWMgmt- ABWMgmt-
>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>> Surpise-
>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>> HPIrq- LinkChg-
>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>> Interlock-
>>>                       Changed: MRL- PresDet- LinkState-
>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>> CRSVisible-
>>>               RootCap: CRSVisible-
>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>       Capabilities: [100] Virtual Channel <?>
>>>       Kernel driver in use: pcieport-driver
>>>       Kernel modules: shpchp
>>>
>>> 00:0e.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>> (prog-if 00 [Normal decode])
>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx+
>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0, Cache Line Size: 64 bytes
>>>       Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- <SERR- <PERR-
>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>       Capabilities: [48] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>               Address: 00000000fee0f00c  Data: 4169
>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>               Mapping Address Base: 00000000fee00000
>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>> <64ns,
>>> L1 <1us
>>>                       ExtTag- RBE+ FLReset-
>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>> Unsupported-
>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>> TransPend-
>>>               LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>> Latency L0 <512ns, L1 <4us
>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>> CommClk-
>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>               LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
>>> DLActive- BWMgmt- ABWMgmt-
>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>> Surpise-
>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>> HPIrq- LinkChg-
>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>> Interlock-
>>>                       Changed: MRL- PresDet- LinkState-
>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>> CRSVisible-
>>>               RootCap: CRSVisible-
>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>       Capabilities: [100] Virtual Channel <?>
>>>       Kernel driver in use: pcieport-driver
>>>       Kernel modules: shpchp
>>>
>>> 00:0f.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>> (prog-if 00 [Normal decode])
>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx+
>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Latency: 0, Cache Line Size: 64 bytes
>>>       Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- <SERR- <PERR-
>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>       Capabilities: [48] Power Management version 2
>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>               Address: 00000000fee0f00c  Data: 4171
>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>               Mapping Address Base: 00000000fee00000
>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>> <64ns,
>>> L1 <1us
>>>                       ExtTag- RBE+ FLReset-
>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>> Unsupported-
>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>> TransPend-
>>>               LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1,
>>> Latency L0 <512ns, L1 <4us
>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>> CommClk-
>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>               LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+
>>> DLActive- BWMgmt- ABWMgmt-
>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>> Surpise-
>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>> HPIrq- LinkChg-
>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>> Interlock-
>>>                       Changed: MRL- PresDet- LinkState-
>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>> CRSVisible-
>>>               RootCap: CRSVisible-
>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>       Capabilities: [100] Virtual Channel <?>
>>>       Kernel driver in use: pcieport-driver
>>>       Kernel modules: shpchp
>>>
>>> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>> Athlon64, Sempron] HyperTransport Configuration
>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Capabilities: [80] HyperTransport: Host or Secondary Interface
>>>               Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+
>>> Slave- <EOCErr- DUL-
>>>               Link Control: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO-
>>> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>>>               Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut-
>>> LWI=16bit
>>> DwFcInEn- LWO=16bit DwFcOutEn-
>>>               Revision ID: 3.00
>>>               Link Frequency: 1.0GHz
>>>               Link Error: <Prot- <Ovfl- <EOC- CTLTm-
>>>               Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz-
>>> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz+ 1.4GHz- 1.6GHz- Vend-
>>>               Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+
>>> UIDRD- ExtRS- UCnfE-
>>>
>>> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>> Athlon64, Sempron] Address Map
>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>
>>> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>> Athlon64, Sempron] DRAM Controller
>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Kernel modules: amd64_edac_mod
>>>
>>> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>> Athlon64, Sempron] Miscellaneous Control
>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       Capabilities: [f0] Secure device <?>
>>>
>>> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>> Athlon64, Sempron] Link Control
>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>
>>> 01:0a.0 VGA compatible controller: XGI Technology Inc. (eXtreme Graphics
>>> Innovation) Volari Z7/Z9/Z9s (prog-if 00 [VGA controller])
>>>       Subsystem: XGI Technology Inc. (eXtreme Graphics Innovation) Volari
>>> Z7/Z9/Z9s
>>>       Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>> Stepping- SERR- FastB2B- DisINTx-
>>>       Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>       BIST result: 00
>>>       Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
>>>       Region 1: Memory at febc0000 (32-bit, non-prefetchable) [size=256K]
>>>       Region 2: I/O ports at ec00 [size=128]
>>>       Expansion ROM at <unassigned> [disabled]
>>>       Capabilities: [40] Power Management version 2
>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>
>>>
>>>
>>> /dev/sda:
>>>
>>> ATA device, with non-removable media
>>>       Model Number:       WDC WD5000AAKS-00A7B2
>>>       Serial Number:      WD-WCASZ0321108
>>>       Firmware Revision:  01.03B01
>>>       Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev
>>> 2.5
>>> Standards:
>>>       Supported: 8 7 6 5
>>>       Likely used: 8
>>> Configuration:
>>>       Logical         max     current
>>>       cylinders       16383   16383
>>>       heads           16      16
>>>       sectors/track   63      63
>>>       --
>>>       CHS current addressable sectors:   16514064
>>>       LBA    user addressable sectors:  268435455
>>>       LBA48  user addressable sectors:  976773168
>>>       Logical/Physical Sector size:           512 bytes
>>>       device size with M = 1024*1024:      476940 MBytes
>>>       device size with M = 1000*1000:      500107 MBytes (500 GB)
>>>       cache/buffer size  = 16384 KBytes
>>> Capabilities:
>>>       LBA, IORDY(can be disabled)
>>>       Queue depth: 32
>>>       Standby timer values: spec'd by Standard, with device specific
>>> minimum
>>>       R/W multiple sector transfer: Max = 16  Current = 16
>>>       Recommended acoustic management value: 128, current value: 254
>>>       DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
>>>            Cycle time: min=120ns recommended=120ns
>>>       PIO: pio0 pio1 pio2 pio3 pio4
>>>            Cycle time: no flow control=120ns  IORDY flow control=120ns
>>> Commands/features:
>>>       Enabled Supported:
>>>          *    SMART feature set
>>>               Security Mode feature set
>>>          *    Power Management feature set
>>>          *    Write cache
>>>          *    Look-ahead
>>>          *    Host Protected Area feature set
>>>          *    WRITE_BUFFER command
>>>          *    READ_BUFFER command
>>>          *    NOP cmd
>>>          *    DOWNLOAD_MICROCODE
>>>               Power-Up In Standby feature set
>>>          *    SET_FEATURES required to spinup after power up
>>>               SET_MAX security extension
>>>               Automatic Acoustic Management feature set
>>>          *    48-bit Address feature set
>>>          *    Device Configuration Overlay feature set
>>>          *    Mandatory FLUSH_CACHE
>>>          *    FLUSH_CACHE_EXT
>>>          *    SMART error logging
>>>          *    SMART self-test
>>>          *    General Purpose Logging feature set
>>>          *    64-bit World wide name
>>>          *    Segmented DOWNLOAD_MICROCODE
>>>          *    Gen1 signaling speed (1.5Gb/s)
>>>          *    Gen2 signaling speed (3.0Gb/s)
>>>          *    Native Command Queueing (NCQ)
>>>          *    Host-initiated interface power management
>>>          *    Phy event counters
>>>               DMA Setup Auto-Activate optimization
>>>          *    Software settings preservation
>>>          *    SMART Command Transport (SCT) feature set
>>>          *    SCT Long Sector Access (AC1)
>>>          *    SCT LBA Segment Access (AC2)
>>>          *    SCT Error Recovery Control (AC3)
>>>          *    SCT Features Control (AC4)
>>>          *    SCT Data Tables (AC5)
>>>               unknown 206[12] (vendor specific)
>>>               unknown 206[13] (vendor specific)
>>> Security:
>>>       Master password revision code = 65534
>>>               supported
>>>       not     enabled
>>>       not     locked
>>>               frozen
>>>       not     expired: security count
>>>               supported: enhanced erase
>>>       84min for SECURITY ERASE UNIT. 84min for ENHANCED SECURITY ERASE
>>> UNIT.
>>> Logical Unit WWN Device Identifier: 50014ee101cedc24
>>>       NAA             : 5
>>>       IEEE OUI        : 0014ee
>>>       Unique ID       : 101cedc24
>>> Checksum: correct
>>>
>>> /dev/md0:
>>>       Version : 0.90
>>>  Creation Time : Mon Jul 27 22:38:26 2009
>>>    Raid Level : raid1
>>>    Array Size : 102336 (99.95 MiB 104.79 MB)
>>>  Used Dev Size : 102336 (99.95 MiB 104.79 MB)
>>>  Raid Devices : 4
>>>  Total Devices : 4
>>> Preferred Minor : 0
>>>   Persistence : Superblock is persistent
>>>
>>>   Update Time : Mon Sep 21 11:02:14 2009
>>>         State : clean
>>>  Active Devices : 4
>>> Working Devices : 4
>>>  Failed Devices : 0
>>>  Spare Devices : 0
>>>
>>>          UUID : 9037def2:035b0d10:bfe78010:bc810f04
>>>        Events : 0.44
>>>
>>>   Number   Major   Minor   RaidDevice State
>>>      0       8        1        0      active sync   /dev/sda1
>>>      1       8       17        1      active sync   /dev/sdb1
>>>      2       8       33        2      active sync   /dev/sdc1
>>>      3       8       49        3      active sync   /dev/sdd1
>>> /dev/md1:
>>>       Version : 0.90
>>>  Creation Time : Mon Jul 27 22:38:28 2009
>>>    Raid Level : raid1
>>>    Array Size : 4095936 (3.91 GiB 4.19 GB)
>>>  Used Dev Size : 4095936 (3.91 GiB 4.19 GB)
>>>  Raid Devices : 4
>>>  Total Devices : 4
>>> Preferred Minor : 1
>>>   Persistence : Superblock is persistent
>>>
>>>   Update Time : Mon Sep 21 11:02:14 2009
>>>         State : clean
>>>  Active Devices : 4
>>> Working Devices : 4
>>>  Failed Devices : 0
>>>  Spare Devices : 0
>>>
>>>          UUID : 642c2833:bf033a7d:bfe78010:bc810f04
>>>        Events : 0.44
>>>
>>>   Number   Major   Minor   RaidDevice State
>>>      0       8        2        0      active sync   /dev/sda2
>>>      1       8       18        1      active sync   /dev/sdb2
>>>      2       8       34        2      active sync   /dev/sdc2
>>>      3       8       50        3      active sync   /dev/sdd2
>>> /dev/md2:
>>>       Version : 0.90
>>>  Creation Time : Mon Jul 27 22:38:30 2009
>>>    Raid Level : raid1
>>>    Array Size : 484185536 (461.76 GiB 495.81 GB)
>>>  Used Dev Size : 484185536 (461.76 GiB 495.81 GB)
>>>  Raid Devices : 4
>>>  Total Devices : 4
>>> Preferred Minor : 2
>>>   Persistence : Superblock is persistent
>>>
>>>   Update Time : Mon Sep 21 11:02:24 2009
>>>         State : clean
>>>  Active Devices : 4
>>> Working Devices : 4
>>>  Failed Devices : 0
>>>  Spare Devices : 0
>>>
>>>          UUID : ce106d52:0c0373f7:bfe78010:bc810f04
>>>        Events : 0.66
>>>
>>>   Number   Major   Minor   RaidDevice State
>>>      0       8        3        0      active sync   /dev/sda3
>>>      1       8       19        1      active sync   /dev/sdb3
>>>      2       8       35        2      active sync   /dev/sdc3
>>>      3       8       51        3      active sync   /dev/sdd3
>>>
>>>
>>>
>>
>>
>>
>>
>
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-22 14:48     ` Majed B.
@ 2009-09-22 14:59       ` Lee Howard
  2009-09-22 15:13         ` Majed B.
  2009-09-22 15:19         ` Robin Hill
  0 siblings, 2 replies; 14+ messages in thread
From: Lee Howard @ 2009-09-22 14:59 UTC (permalink / raw)
  To: Majed B.; +Cc: linux-raid

Majed B. wrote:
> I must have missed that part. It may not work for your case, but worth trying.
>
> Perhaps Neil Brown, or someone involved could shed some light on this.
>
> If I remember correctly, those soft lockups were harmless anyway.
>   

Not harmless for production use.  Yes, data is not harmed, and yes, the 
problem state does recover when the data-check finishes, but during the 
data-check the system is virtually unresponsive and all other use of the 
system is stalled.

Thanks,

Lee.

> On Tue, Sep 22, 2009 at 5:43 PM, Lee Howard <faxguy@howardsilvan.com> wrote:
>   
>> Thanks Majed,
>>
>> I've looked at the patch and it modifies md/raid5.c.  However, I'm using
>> RAID1.  Am I ignorant for doubting that the patch to raid5.c has any affect
>> on RAID1?
>>
>> Thanks,
>>
>> Lee.
>>
>>
>> Majed B. wrote:
>>     
>>> Hello,
>>>
>>> Check out the thread with the subject: BUG: soft lockup in
>>> [md4_raid5:21137]
>>>
>>> I think the guys have already fixed this in 2.6.32. The patch is
>>> mentioned in one of the replies, if you wish to try it.
>>>
>>> On Mon, Sep 21, 2009 at 9:08 PM, Lee Howard <faxguy@howardsilvan.com>
>>> wrote:
>>>
>>>       
>>>> Linux software RAID maintainers and developers,
>>>>
>>>> Two months ago I wrote to the Linux kernel mailing list regarding a
>>>> condition expressed as "BUG: soft lockup - CPU#3 stuck for 61s!".  I
>>>> initially battled this recurring problem in both Fedora 10 and Fedora 11.
>>>>  Rafael J. Wysocki suggested that I update the kernel (to 2.6.31-rc4 or
>>>> later) and see if the problem resurfaced.  I then used kernel
>>>> 2.6.31-0.94.rc4.fc12.x86_64 and found that the problem still continued,
>>>> but
>>>> noticeably only when the md data-check process was run.
>>>>
>>>> You can read the last post to the LKML thread (with links to the entire
>>>> thread) here:
>>>>
>>>> http://lkml.org/lkml/2009/8/6/387
>>>>
>>>> The md data-check is being run in Fedora 11 by
>>>> /etc/cron.weekly/raid-check
>>>> which is a little shell script that looks like this:
>>>>
>>>> ------------------------------------------------------
>>>> #!/bin/bash
>>>>
>>>> for dev in `grep "^md.*: active" /proc/mdstat | cut -f 1 -d ' '`; do
>>>>      [ -f /sys/block/$dev/md/sync_action ] && \
>>>>              echo "check" > /sys/block/$dev/md/sync_action
>>>> done
>>>> ------------------------------------------------------
>>>>
>>>> I have disabled this weekly data-check, and since doing so have not
>>>> encountered any soft lockup (or any other problem, for that matter).
>>>>
>>>> For reference, you can see in this thread here that I am not the only one
>>>> to
>>>> have this problem:
>>>>
>>>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/212684
>>>>
>>>> It may important to note that I am running a 4-disk RAID1 array on 500GB
>>>> Western-Digial SATA drives.  I've attached the output of 'lspci -vv' as
>>>> "lspci.out".  I've attached the output of 'hdparm -I /dev/sda' as
>>>> "hdparm.out".  /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd are all
>>>> identical
>>>> drive types.  I've attached the output of 'for i in 0 1 2; do mdadm
>>>> --detail
>>>> /dev/md$i; done' as "mdadm.out".
>>>>
>>>> Please let me know if this is a configuration problem, a kernel bug, or
>>>> something else.  Please let me know how to fix this problem so that I can
>>>> safely re-enable md data-check.
>>>>
>>>> Thanks,
>>>>
>>>> Lee.
>>>>
>>>> 00:00.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a2)
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR+ FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0
>>>>       Capabilities: [44] HyperTransport: Slave or Primary Interface
>>>>               Command: BaseUnitID=0 UnitCnt=15 MastHost- DefDir- DUL-
>>>>               Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO-
>>>> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>>>>               Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut-
>>>> LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
>>>>               Link Control 1: CFlE- CST- CFE- <LkFail+ Init- EOC+ TXO+
>>>> <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
>>>>               Link Config 1: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut-
>>>> LWI=8bit
>>>> DwFcInEn- LWO=8bit DwFcOutEn-
>>>>               Revision ID: 1.03
>>>>               Link Frequency 0: 1.0GHz
>>>>               Link Error 0: <Prot- <Ovfl- <EOC- CTLTm-
>>>>               Link Frequency Capability 0: 200MHz+ 300MHz+ 400MHz+
>>>> 500MHz+
>>>> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>>>>               Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA-
>>>> UIDRD-
>>>>               Link Frequency 1: 200MHz
>>>>               Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
>>>>               Link Frequency Capability 1: 200MHz- 300MHz- 400MHz-
>>>> 500MHz-
>>>> 600MHz- 800MHz- 1.0GHz- 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>>>>               Error Handling: PFlE+ OFlE+ PFE- OFE- EOCFE- RFE- CRCFE-
>>>> SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
>>>>               Prefetchable memory behind bridge Upper: 00-00
>>>>               Bus Number: 00
>>>>       Capabilities: [dc] HyperTransport: MSI Mapping Enable+ Fixed-
>>>>               Mapping Address Base: 00000000fee00000
>>>>
>>>> 00:01.0 ISA bridge: nVidia Corporation MCP55 LPC Bridge (rev a3)
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0
>>>>       Region 0: I/O ports at 2f00 [size=128]
>>>>
>>>> 00:01.1 SMBus: nVidia Corporation MCP55 SMBus (rev a3)
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Interrupt: pin A routed to IRQ 10
>>>>       Region 0: I/O ports at dc00 [size=64]
>>>>       Region 4: I/O ports at 2d00 [size=64]
>>>>       Region 5: I/O ports at 2e00 [size=64]
>>>>       Capabilities: [44] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Kernel driver in use: nForce2_smbus
>>>>       Kernel modules: i2c-nforce2
>>>>
>>>> 00:02.0 USB Controller: nVidia Corporation MCP55 USB Controller (rev a1)
>>>> (prog-if 10 [OHCI])
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0 (750ns min, 250ns max)
>>>>       Interrupt: pin A routed to IRQ 22
>>>>       Region 0: Memory at feafb000 (32-bit, non-prefetchable) [size=4K]
>>>>       Capabilities: [44] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Kernel driver in use: ohci_hcd
>>>>
>>>> 00:02.1 USB Controller: nVidia Corporation MCP55 USB Controller (rev a2)
>>>> (prog-if 20 [EHCI])
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0 (750ns min, 250ns max)
>>>>       Interrupt: pin B routed to IRQ 23
>>>>       Region 0: Memory at feafac00 (32-bit, non-prefetchable) [size=256]
>>>>       Capabilities: [44] Debug port: BAR=1 offset=0098
>>>>       Capabilities: [80] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Kernel driver in use: ehci_hcd
>>>>
>>>> 00:04.0 IDE interface: nVidia Corporation MCP55 IDE (rev a1) (prog-if 8a
>>>> [Master SecP PriP])
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0 (750ns min, 250ns max)
>>>>       Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable)
>>>> [size=8]
>>>>       Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
>>>> [size=1]
>>>>       Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable)
>>>> [size=8]
>>>>       Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
>>>> [size=1]
>>>>       Region 4: I/O ports at ffa0 [size=16]
>>>>       Capabilities: [44] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Kernel driver in use: pata_amd
>>>>       Kernel modules: ata_generic, pata_acpi, pata_amd
>>>>
>>>> 00:05.0 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
>>>> (prog-if 85 [Master SecO PriO])
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0 (750ns min, 250ns max)
>>>>       Interrupt: pin A routed to IRQ 21
>>>>       Region 0: I/O ports at d480 [size=8]
>>>>       Region 1: I/O ports at d400 [size=4]
>>>>       Region 2: I/O ports at d080 [size=8]
>>>>       Region 3: I/O ports at d000 [size=4]
>>>>       Region 4: I/O ports at cc00 [size=16]
>>>>       Region 5: Memory at feaf9000 (32-bit, non-prefetchable) [size=4K]
>>>>       Capabilities: [44] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>>>               Address: 0000000000000000  Data: 0000
>>>>       Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>>>       Kernel driver in use: sata_nv
>>>>       Kernel modules: ata_generic, pata_acpi, sata_nv
>>>>
>>>> 00:05.1 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
>>>> (prog-if 85 [Master SecO PriO])
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0 (750ns min, 250ns max)
>>>>       Interrupt: pin B routed to IRQ 20
>>>>       Region 0: I/O ports at c880 [size=8]
>>>>       Region 1: I/O ports at c800 [size=4]
>>>>       Region 2: I/O ports at c480 [size=8]
>>>>       Region 3: I/O ports at c400 [size=4]
>>>>       Region 4: I/O ports at c080 [size=16]
>>>>       Region 5: Memory at feaf8000 (32-bit, non-prefetchable) [size=4K]
>>>>       Capabilities: [44] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>>>               Address: 0000000000000000  Data: 0000
>>>>       Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>>>       Kernel driver in use: sata_nv
>>>>       Kernel modules: ata_generic, pata_acpi, sata_nv
>>>>
>>>> 00:05.2 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a3)
>>>> (prog-if 85 [Master SecO PriO])
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0 (750ns min, 250ns max)
>>>>       Interrupt: pin C routed to IRQ 23
>>>>       Region 0: I/O ports at c000 [size=8]
>>>>       Region 1: I/O ports at bc00 [size=4]
>>>>       Region 2: I/O ports at b880 [size=8]
>>>>       Region 3: I/O ports at b800 [size=4]
>>>>       Region 4: I/O ports at b480 [size=16]
>>>>       Region 5: Memory at feaf7000 (32-bit, non-prefetchable) [size=4K]
>>>>       Capabilities: [44] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>>>               Address: 0000000000000000  Data: 0000
>>>>       Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>>>       Kernel driver in use: sata_nv
>>>>       Kernel modules: ata_generic, pata_acpi, sata_nv
>>>>
>>>> 00:06.0 PCI bridge: nVidia Corporation MCP55 PCI bridge (rev a2) (prog-if
>>>> 01
>>>> [Subtractive decode])
>>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0
>>>>       Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>>>>       I/O behind bridge: 0000e000-0000efff
>>>>       Memory behind bridge: feb00000-febfffff
>>>>       Prefetchable memory behind bridge: f8000000-fbffffff
>>>>       Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort-
>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>       BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
>>>>               PriDiscTmr- SecDiscTmr+ DiscTmrStat- DiscTmrSERREn-
>>>>       Capabilities: [b8] Subsystem: nVidia Corporation Device cb84
>>>>       Capabilities: [8c] HyperTransport: MSI Mapping Enable- Fixed-
>>>>               Mapping Address Base: 00000000fee00000
>>>>
>>>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0 (250ns min, 5000ns max)
>>>>       Interrupt: pin A routed to IRQ 30
>>>>       Region 0: Memory at feaf6000 (32-bit, non-prefetchable) [size=4K]
>>>>       Region 1: I/O ports at b400 [size=8]
>>>>       Region 2: Memory at feafa800 (32-bit, non-prefetchable) [size=256]
>>>>       Region 3: Memory at feafa400 (32-bit, non-prefetchable) [size=16]
>>>>       Capabilities: [44] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>>>>       Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>>>>               Vector table: BAR=2 offset=00000000
>>>>               PBA: BAR=3 offset=00000000
>>>>       Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>>>>               Address: 00000000fee0400c  Data: 4199
>>>>               Masking: 000000fe  Pending: 00000000
>>>>       Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>>>>       Kernel driver in use: forcedeth
>>>>       Kernel modules: forcedeth
>>>>
>>>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>>>>       Subsystem: nVidia Corporation Device cb84
>>>>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0 (250ns min, 5000ns max)
>>>>       Interrupt: pin A routed to IRQ 31
>>>>       Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
>>>>       Region 1: I/O ports at b080 [size=8]
>>>>       Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256]
>>>>       Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16]
>>>>       Capabilities: [44] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>>>>       Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>>>>               Vector table: BAR=2 offset=00000000
>>>>               PBA: BAR=3 offset=00000000
>>>>       Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>>>>               Address: 00000000fee0200c  Data: 41a1
>>>>               Masking: 000000fe  Pending: 00000000
>>>>       Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>>>>       Kernel driver in use: forcedeth
>>>>       Kernel modules: forcedeth
>>>>
>>>> 00:0a.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>>> (prog-if 00 [Normal decode])
>>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0, Cache Line Size: 64 bytes
>>>>       Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>       Capabilities: [48] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>               Address: 00000000fee0f00c  Data: 4149
>>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>               Mapping Address Base: 00000000fee00000
>>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>> <64ns,
>>>> L1 <1us
>>>>                       ExtTag- RBE+ FLReset-
>>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>> Unsupported-
>>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>> TransPend-
>>>>               LnkCap: Port #5, Speed 2.5GT/s, Width x8, ASPM L0s L1,
>>>> Latency L0 <512ns, L1 <4us
>>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>> CommClk-
>>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>               LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
>>>> DLActive- BWMgmt- ABWMgmt-
>>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>> Surpise-
>>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>> HPIrq- LinkChg-
>>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>> Interlock-
>>>>                       Changed: MRL- PresDet- LinkState-
>>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>>> CRSVisible-
>>>>               RootCap: CRSVisible-
>>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>       Capabilities: [100] Virtual Channel <?>
>>>>       Kernel driver in use: pcieport-driver
>>>>       Kernel modules: shpchp
>>>>
>>>> 00:0b.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>>> (prog-if 00 [Normal decode])
>>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0, Cache Line Size: 64 bytes
>>>>       Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
>>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>       Capabilities: [48] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>               Address: 00000000fee0f00c  Data: 4151
>>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>               Mapping Address Base: 00000000fee00000
>>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>> <64ns,
>>>> L1 <1us
>>>>                       ExtTag- RBE+ FLReset-
>>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>> Unsupported-
>>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>> TransPend-
>>>>               LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>>> Latency L0 <512ns, L1 <4us
>>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>> CommClk-
>>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>>>> DLActive- BWMgmt- ABWMgmt-
>>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>> Surpise-
>>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>> HPIrq- LinkChg-
>>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>> Interlock-
>>>>                       Changed: MRL- PresDet- LinkState-
>>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>>> CRSVisible-
>>>>               RootCap: CRSVisible-
>>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>       Capabilities: [100] Virtual Channel <?>
>>>>       Kernel driver in use: pcieport-driver
>>>>       Kernel modules: shpchp
>>>>
>>>> 00:0c.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>>> (prog-if 00 [Normal decode])
>>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0, Cache Line Size: 64 bytes
>>>>       Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
>>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>       Capabilities: [48] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>               Address: 00000000fee0f00c  Data: 4159
>>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>               Mapping Address Base: 00000000fee00000
>>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>> <64ns,
>>>> L1 <1us
>>>>                       ExtTag- RBE+ FLReset-
>>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>> Unsupported-
>>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>> TransPend-
>>>>               LnkCap: Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>>> Latency L0 <512ns, L1 <4us
>>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>> CommClk-
>>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>>>> DLActive- BWMgmt- ABWMgmt-
>>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>> Surpise-
>>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>> HPIrq- LinkChg-
>>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>> Interlock-
>>>>                       Changed: MRL- PresDet- LinkState-
>>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>>> CRSVisible-
>>>>               RootCap: CRSVisible-
>>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>       Capabilities: [100] Virtual Channel <?>
>>>>       Kernel driver in use: pcieport-driver
>>>>       Kernel modules: shpchp
>>>>
>>>> 00:0d.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>>> (prog-if 00 [Normal decode])
>>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0, Cache Line Size: 64 bytes
>>>>       Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
>>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>       Capabilities: [48] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>               Address: 00000000fee0f00c  Data: 4161
>>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>               Mapping Address Base: 00000000fee00000
>>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>> <64ns,
>>>> L1 <1us
>>>>                       ExtTag- RBE+ FLReset-
>>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>> Unsupported-
>>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>> TransPend-
>>>>               LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>>> Latency L0 <512ns, L1 <4us
>>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>> CommClk-
>>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>>>> DLActive- BWMgmt- ABWMgmt-
>>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>> Surpise-
>>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>> HPIrq- LinkChg-
>>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>> Interlock-
>>>>                       Changed: MRL- PresDet- LinkState-
>>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>>> CRSVisible-
>>>>               RootCap: CRSVisible-
>>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>       Capabilities: [100] Virtual Channel <?>
>>>>       Kernel driver in use: pcieport-driver
>>>>       Kernel modules: shpchp
>>>>
>>>> 00:0e.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>>> (prog-if 00 [Normal decode])
>>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0, Cache Line Size: 64 bytes
>>>>       Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
>>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>       Capabilities: [48] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>               Address: 00000000fee0f00c  Data: 4169
>>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>               Mapping Address Base: 00000000fee00000
>>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>> <64ns,
>>>> L1 <1us
>>>>                       ExtTag- RBE+ FLReset-
>>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>> Unsupported-
>>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>> TransPend-
>>>>               LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>>> Latency L0 <512ns, L1 <4us
>>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>> CommClk-
>>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>               LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
>>>> DLActive- BWMgmt- ABWMgmt-
>>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>> Surpise-
>>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>> HPIrq- LinkChg-
>>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>> Interlock-
>>>>                       Changed: MRL- PresDet- LinkState-
>>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>>> CRSVisible-
>>>>               RootCap: CRSVisible-
>>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>       Capabilities: [100] Virtual Channel <?>
>>>>       Kernel driver in use: pcieport-driver
>>>>       Kernel modules: shpchp
>>>>
>>>> 00:0f.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a3)
>>>> (prog-if 00 [Normal decode])
>>>>       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Latency: 0, Cache Line Size: 64 bytes
>>>>       Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
>>>>       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>       BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>       Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>       Capabilities: [48] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>       Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>               Address: 00000000fee0f00c  Data: 4171
>>>>       Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>               Mapping Address Base: 00000000fee00000
>>>>       Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>> <64ns,
>>>> L1 <1us
>>>>                       ExtTag- RBE+ FLReset-
>>>>               DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>> Unsupported-
>>>>                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>> TransPend-
>>>>               LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1,
>>>> Latency L0 <512ns, L1 <4us
>>>>                       ClockPM- Surprise- LLActRep+ BwNot-
>>>>               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>> CommClk-
>>>>                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>               LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+
>>>> DLActive- BWMgmt- ABWMgmt-
>>>>               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>> Surpise-
>>>>                       Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-
>>>>               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>> HPIrq- LinkChg-
>>>>                       Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>> Interlock-
>>>>                       Changed: MRL- PresDet- LinkState-
>>>>               RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
>>>> CRSVisible-
>>>>               RootCap: CRSVisible-
>>>>               RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>       Capabilities: [100] Virtual Channel <?>
>>>>       Kernel driver in use: pcieport-driver
>>>>       Kernel modules: shpchp
>>>>
>>>> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>> Athlon64, Sempron] HyperTransport Configuration
>>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Capabilities: [80] HyperTransport: Host or Secondary Interface
>>>>               Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+
>>>> Slave- <EOCErr- DUL-
>>>>               Link Control: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO-
>>>> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>>>>               Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut-
>>>> LWI=16bit
>>>> DwFcInEn- LWO=16bit DwFcOutEn-
>>>>               Revision ID: 3.00
>>>>               Link Frequency: 1.0GHz
>>>>               Link Error: <Prot- <Ovfl- <EOC- CTLTm-
>>>>               Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz-
>>>> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz+ 1.4GHz- 1.6GHz- Vend-
>>>>               Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+
>>>> UIDRD- ExtRS- UCnfE-
>>>>
>>>> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>> Athlon64, Sempron] Address Map
>>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>
>>>> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>> Athlon64, Sempron] DRAM Controller
>>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Kernel modules: amd64_edac_mod
>>>>
>>>> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>> Athlon64, Sempron] Miscellaneous Control
>>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       Capabilities: [f0] Secure device <?>
>>>>
>>>> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>> Athlon64, Sempron] Link Control
>>>>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>
>>>> 01:0a.0 VGA compatible controller: XGI Technology Inc. (eXtreme Graphics
>>>> Innovation) Volari Z7/Z9/Z9s (prog-if 00 [VGA controller])
>>>>       Subsystem: XGI Technology Inc. (eXtreme Graphics Innovation) Volari
>>>> Z7/Z9/Z9s
>>>>       Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>       Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>       BIST result: 00
>>>>       Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
>>>>       Region 1: Memory at febc0000 (32-bit, non-prefetchable) [size=256K]
>>>>       Region 2: I/O ports at ec00 [size=128]
>>>>       Expansion ROM at <unassigned> [disabled]
>>>>       Capabilities: [40] Power Management version 2
>>>>               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>               Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>
>>>>
>>>>
>>>> /dev/sda:
>>>>
>>>> ATA device, with non-removable media
>>>>       Model Number:       WDC WD5000AAKS-00A7B2
>>>>       Serial Number:      WD-WCASZ0321108
>>>>       Firmware Revision:  01.03B01
>>>>       Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev
>>>> 2.5
>>>> Standards:
>>>>       Supported: 8 7 6 5
>>>>       Likely used: 8
>>>> Configuration:
>>>>       Logical         max     current
>>>>       cylinders       16383   16383
>>>>       heads           16      16
>>>>       sectors/track   63      63
>>>>       --
>>>>       CHS current addressable sectors:   16514064
>>>>       LBA    user addressable sectors:  268435455
>>>>       LBA48  user addressable sectors:  976773168
>>>>       Logical/Physical Sector size:           512 bytes
>>>>       device size with M = 1024*1024:      476940 MBytes
>>>>       device size with M = 1000*1000:      500107 MBytes (500 GB)
>>>>       cache/buffer size  = 16384 KBytes
>>>> Capabilities:
>>>>       LBA, IORDY(can be disabled)
>>>>       Queue depth: 32
>>>>       Standby timer values: spec'd by Standard, with device specific
>>>> minimum
>>>>       R/W multiple sector transfer: Max = 16  Current = 16
>>>>       Recommended acoustic management value: 128, current value: 254
>>>>       DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
>>>>            Cycle time: min=120ns recommended=120ns
>>>>       PIO: pio0 pio1 pio2 pio3 pio4
>>>>            Cycle time: no flow control=120ns  IORDY flow control=120ns
>>>> Commands/features:
>>>>       Enabled Supported:
>>>>          *    SMART feature set
>>>>               Security Mode feature set
>>>>          *    Power Management feature set
>>>>          *    Write cache
>>>>          *    Look-ahead
>>>>          *    Host Protected Area feature set
>>>>          *    WRITE_BUFFER command
>>>>          *    READ_BUFFER command
>>>>          *    NOP cmd
>>>>          *    DOWNLOAD_MICROCODE
>>>>               Power-Up In Standby feature set
>>>>          *    SET_FEATURES required to spinup after power up
>>>>               SET_MAX security extension
>>>>               Automatic Acoustic Management feature set
>>>>          *    48-bit Address feature set
>>>>          *    Device Configuration Overlay feature set
>>>>          *    Mandatory FLUSH_CACHE
>>>>          *    FLUSH_CACHE_EXT
>>>>          *    SMART error logging
>>>>          *    SMART self-test
>>>>          *    General Purpose Logging feature set
>>>>          *    64-bit World wide name
>>>>          *    Segmented DOWNLOAD_MICROCODE
>>>>          *    Gen1 signaling speed (1.5Gb/s)
>>>>          *    Gen2 signaling speed (3.0Gb/s)
>>>>          *    Native Command Queueing (NCQ)
>>>>          *    Host-initiated interface power management
>>>>          *    Phy event counters
>>>>               DMA Setup Auto-Activate optimization
>>>>          *    Software settings preservation
>>>>          *    SMART Command Transport (SCT) feature set
>>>>          *    SCT Long Sector Access (AC1)
>>>>          *    SCT LBA Segment Access (AC2)
>>>>          *    SCT Error Recovery Control (AC3)
>>>>          *    SCT Features Control (AC4)
>>>>          *    SCT Data Tables (AC5)
>>>>               unknown 206[12] (vendor specific)
>>>>               unknown 206[13] (vendor specific)
>>>> Security:
>>>>       Master password revision code = 65534
>>>>               supported
>>>>       not     enabled
>>>>       not     locked
>>>>               frozen
>>>>       not     expired: security count
>>>>               supported: enhanced erase
>>>>       84min for SECURITY ERASE UNIT. 84min for ENHANCED SECURITY ERASE
>>>> UNIT.
>>>> Logical Unit WWN Device Identifier: 50014ee101cedc24
>>>>       NAA             : 5
>>>>       IEEE OUI        : 0014ee
>>>>       Unique ID       : 101cedc24
>>>> Checksum: correct
>>>>
>>>> /dev/md0:
>>>>       Version : 0.90
>>>>  Creation Time : Mon Jul 27 22:38:26 2009
>>>>    Raid Level : raid1
>>>>    Array Size : 102336 (99.95 MiB 104.79 MB)
>>>>  Used Dev Size : 102336 (99.95 MiB 104.79 MB)
>>>>  Raid Devices : 4
>>>>  Total Devices : 4
>>>> Preferred Minor : 0
>>>>   Persistence : Superblock is persistent
>>>>
>>>>   Update Time : Mon Sep 21 11:02:14 2009
>>>>         State : clean
>>>>  Active Devices : 4
>>>> Working Devices : 4
>>>>  Failed Devices : 0
>>>>  Spare Devices : 0
>>>>
>>>>          UUID : 9037def2:035b0d10:bfe78010:bc810f04
>>>>        Events : 0.44
>>>>
>>>>   Number   Major   Minor   RaidDevice State
>>>>      0       8        1        0      active sync   /dev/sda1
>>>>      1       8       17        1      active sync   /dev/sdb1
>>>>      2       8       33        2      active sync   /dev/sdc1
>>>>      3       8       49        3      active sync   /dev/sdd1
>>>> /dev/md1:
>>>>       Version : 0.90
>>>>  Creation Time : Mon Jul 27 22:38:28 2009
>>>>    Raid Level : raid1
>>>>    Array Size : 4095936 (3.91 GiB 4.19 GB)
>>>>  Used Dev Size : 4095936 (3.91 GiB 4.19 GB)
>>>>  Raid Devices : 4
>>>>  Total Devices : 4
>>>> Preferred Minor : 1
>>>>   Persistence : Superblock is persistent
>>>>
>>>>   Update Time : Mon Sep 21 11:02:14 2009
>>>>         State : clean
>>>>  Active Devices : 4
>>>> Working Devices : 4
>>>>  Failed Devices : 0
>>>>  Spare Devices : 0
>>>>
>>>>          UUID : 642c2833:bf033a7d:bfe78010:bc810f04
>>>>        Events : 0.44
>>>>
>>>>   Number   Major   Minor   RaidDevice State
>>>>      0       8        2        0      active sync   /dev/sda2
>>>>      1       8       18        1      active sync   /dev/sdb2
>>>>      2       8       34        2      active sync   /dev/sdc2
>>>>      3       8       50        3      active sync   /dev/sdd2
>>>> /dev/md2:
>>>>       Version : 0.90
>>>>  Creation Time : Mon Jul 27 22:38:30 2009
>>>>    Raid Level : raid1
>>>>    Array Size : 484185536 (461.76 GiB 495.81 GB)
>>>>  Used Dev Size : 484185536 (461.76 GiB 495.81 GB)
>>>>  Raid Devices : 4
>>>>  Total Devices : 4
>>>> Preferred Minor : 2
>>>>   Persistence : Superblock is persistent
>>>>
>>>>   Update Time : Mon Sep 21 11:02:24 2009
>>>>         State : clean
>>>>  Active Devices : 4
>>>> Working Devices : 4
>>>>  Failed Devices : 0
>>>>  Spare Devices : 0
>>>>
>>>>          UUID : ce106d52:0c0373f7:bfe78010:bc810f04
>>>>        Events : 0.66
>>>>
>>>>   Number   Major   Minor   RaidDevice State
>>>>      0       8        3        0      active sync   /dev/sda3
>>>>      1       8       19        1      active sync   /dev/sdb3
>>>>      2       8       35        2      active sync   /dev/sdc3
>>>>      3       8       51        3      active sync   /dev/sdd3
>>>>
>>>>
>>>>
>>>>         
>>>
>>>
>>>       
>>     
>
>
>
>   


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-22 14:59       ` Lee Howard
@ 2009-09-22 15:13         ` Majed B.
  2009-09-22 15:19         ` Robin Hill
  1 sibling, 0 replies; 14+ messages in thread
From: Majed B. @ 2009-09-22 15:13 UTC (permalink / raw)
  To: Lee Howard; +Cc: linux-raid

Have you tried attaching the checking process to a specific CPU/Core?
It's called Affinity.

Also, did you try to reduce the numbers in: /proc/sys/dev/raid/speed_limit_min

Keep it as low as possible, and md should drop to that number (I think
it's in kB) when the system is being utilized, then go back to
/proc/sys/dev/raid/speed_limit_max when it's idle.

On Tue, Sep 22, 2009 at 5:59 PM, Lee Howard <faxguy@howardsilvan.com> wrote:
> Majed B. wrote:
>>
>> I must have missed that part. It may not work for your case, but worth
>> trying.
>>
>> Perhaps Neil Brown, or someone involved could shed some light on this.
>>
>> If I remember correctly, those soft lockups were harmless anyway.
>>
>
> Not harmless for production use.  Yes, data is not harmed, and yes, the
> problem state does recover when the data-check finishes, but during the
> data-check the system is virtually unresponsive and all other use of the
> system is stalled.
>
> Thanks,
>
> Lee.
>
>> On Tue, Sep 22, 2009 at 5:43 PM, Lee Howard <faxguy@howardsilvan.com>
>> wrote:
>>
>>>
>>> Thanks Majed,
>>>
>>> I've looked at the patch and it modifies md/raid5.c.  However, I'm using
>>> RAID1.  Am I ignorant for doubting that the patch to raid5.c has any
>>> affect
>>> on RAID1?
>>>
>>> Thanks,
>>>
>>> Lee.
>>>
>>>
>>> Majed B. wrote:
>>>
>>>>
>>>> Hello,
>>>>
>>>> Check out the thread with the subject: BUG: soft lockup in
>>>> [md4_raid5:21137]
>>>>
>>>> I think the guys have already fixed this in 2.6.32. The patch is
>>>> mentioned in one of the replies, if you wish to try it.
>>>>
>>>> On Mon, Sep 21, 2009 at 9:08 PM, Lee Howard <faxguy@howardsilvan.com>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> Linux software RAID maintainers and developers,
>>>>>
>>>>> Two months ago I wrote to the Linux kernel mailing list regarding a
>>>>> condition expressed as "BUG: soft lockup - CPU#3 stuck for 61s!".  I
>>>>> initially battled this recurring problem in both Fedora 10 and Fedora
>>>>> 11.
>>>>>  Rafael J. Wysocki suggested that I update the kernel (to 2.6.31-rc4 or
>>>>> later) and see if the problem resurfaced.  I then used kernel
>>>>> 2.6.31-0.94.rc4.fc12.x86_64 and found that the problem still continued,
>>>>> but
>>>>> noticeably only when the md data-check process was run.
>>>>>
>>>>> You can read the last post to the LKML thread (with links to the entire
>>>>> thread) here:
>>>>>
>>>>> http://lkml.org/lkml/2009/8/6/387
>>>>>
>>>>> The md data-check is being run in Fedora 11 by
>>>>> /etc/cron.weekly/raid-check
>>>>> which is a little shell script that looks like this:
>>>>>
>>>>> ------------------------------------------------------
>>>>> #!/bin/bash
>>>>>
>>>>> for dev in `grep "^md.*: active" /proc/mdstat | cut -f 1 -d ' '`; do
>>>>>     [ -f /sys/block/$dev/md/sync_action ] && \
>>>>>             echo "check" > /sys/block/$dev/md/sync_action
>>>>> done
>>>>> ------------------------------------------------------
>>>>>
>>>>> I have disabled this weekly data-check, and since doing so have not
>>>>> encountered any soft lockup (or any other problem, for that matter).
>>>>>
>>>>> For reference, you can see in this thread here that I am not the only
>>>>> one
>>>>> to
>>>>> have this problem:
>>>>>
>>>>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/212684
>>>>>
>>>>> It may important to note that I am running a 4-disk RAID1 array on
>>>>> 500GB
>>>>> Western-Digial SATA drives.  I've attached the output of 'lspci -vv' as
>>>>> "lspci.out".  I've attached the output of 'hdparm -I /dev/sda' as
>>>>> "hdparm.out".  /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd are all
>>>>> identical
>>>>> drive types.  I've attached the output of 'for i in 0 1 2; do mdadm
>>>>> --detail
>>>>> /dev/md$i; done' as "mdadm.out".
>>>>>
>>>>> Please let me know if this is a configuration problem, a kernel bug, or
>>>>> something else.  Please let me know how to fix this problem so that I
>>>>> can
>>>>> safely re-enable md data-check.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Lee.
>>>>>
>>>>> 00:00.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a2)
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR+ FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0
>>>>>      Capabilities: [44] HyperTransport: Slave or Primary Interface
>>>>>              Command: BaseUnitID=0 UnitCnt=15 MastHost- DefDir- DUL-
>>>>>              Link Control 0: CFlE+ CST- CFE- <LkFail- Init+ EOC- TXO-
>>>>> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>>>>>              Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut-
>>>>> LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
>>>>>              Link Control 1: CFlE- CST- CFE- <LkFail+ Init- EOC+ TXO+
>>>>> <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
>>>>>              Link Config 1: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut-
>>>>> LWI=8bit
>>>>> DwFcInEn- LWO=8bit DwFcOutEn-
>>>>>              Revision ID: 1.03
>>>>>              Link Frequency 0: 1.0GHz
>>>>>              Link Error 0: <Prot- <Ovfl- <EOC- CTLTm-
>>>>>              Link Frequency Capability 0: 200MHz+ 300MHz+ 400MHz+
>>>>> 500MHz+
>>>>> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>>>>>              Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA-
>>>>> UIDRD-
>>>>>              Link Frequency 1: 200MHz
>>>>>              Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
>>>>>              Link Frequency Capability 1: 200MHz- 300MHz- 400MHz-
>>>>> 500MHz-
>>>>> 600MHz- 800MHz- 1.0GHz- 1.2GHz- 1.4GHz- 1.6GHz- Vend-
>>>>>              Error Handling: PFlE+ OFlE+ PFE- OFE- EOCFE- RFE- CRCFE-
>>>>> SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
>>>>>              Prefetchable memory behind bridge Upper: 00-00
>>>>>              Bus Number: 00
>>>>>      Capabilities: [dc] HyperTransport: MSI Mapping Enable+ Fixed-
>>>>>              Mapping Address Base: 00000000fee00000
>>>>>
>>>>> 00:01.0 ISA bridge: nVidia Corporation MCP55 LPC Bridge (rev a3)
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0
>>>>>      Region 0: I/O ports at 2f00 [size=128]
>>>>>
>>>>> 00:01.1 SMBus: nVidia Corporation MCP55 SMBus (rev a3)
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Interrupt: pin A routed to IRQ 10
>>>>>      Region 0: I/O ports at dc00 [size=64]
>>>>>      Region 4: I/O ports at 2d00 [size=64]
>>>>>      Region 5: I/O ports at 2e00 [size=64]
>>>>>      Capabilities: [44] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0-,D1-,D2-,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Kernel driver in use: nForce2_smbus
>>>>>      Kernel modules: i2c-nforce2
>>>>>
>>>>> 00:02.0 USB Controller: nVidia Corporation MCP55 USB Controller (rev
>>>>> a1)
>>>>> (prog-if 10 [OHCI])
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0 (750ns min, 250ns max)
>>>>>      Interrupt: pin A routed to IRQ 22
>>>>>      Region 0: Memory at feafb000 (32-bit, non-prefetchable) [size=4K]
>>>>>      Capabilities: [44] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Kernel driver in use: ohci_hcd
>>>>>
>>>>> 00:02.1 USB Controller: nVidia Corporation MCP55 USB Controller (rev
>>>>> a2)
>>>>> (prog-if 20 [EHCI])
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0 (750ns min, 250ns max)
>>>>>      Interrupt: pin B routed to IRQ 23
>>>>>      Region 0: Memory at feafac00 (32-bit, non-prefetchable) [size=256]
>>>>>      Capabilities: [44] Debug port: BAR=1 offset=0098
>>>>>      Capabilities: [80] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Kernel driver in use: ehci_hcd
>>>>>
>>>>> 00:04.0 IDE interface: nVidia Corporation MCP55 IDE (rev a1) (prog-if
>>>>> 8a
>>>>> [Master SecP PriP])
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0 (750ns min, 250ns max)
>>>>>      Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable)
>>>>> [size=8]
>>>>>      Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
>>>>> [size=1]
>>>>>      Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable)
>>>>> [size=8]
>>>>>      Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
>>>>> [size=1]
>>>>>      Region 4: I/O ports at ffa0 [size=16]
>>>>>      Capabilities: [44] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Kernel driver in use: pata_amd
>>>>>      Kernel modules: ata_generic, pata_acpi, pata_amd
>>>>>
>>>>> 00:05.0 IDE interface: nVidia Corporation MCP55 SATA Controller (rev
>>>>> a3)
>>>>> (prog-if 85 [Master SecO PriO])
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0 (750ns min, 250ns max)
>>>>>      Interrupt: pin A routed to IRQ 21
>>>>>      Region 0: I/O ports at d480 [size=8]
>>>>>      Region 1: I/O ports at d400 [size=4]
>>>>>      Region 2: I/O ports at d080 [size=8]
>>>>>      Region 3: I/O ports at d000 [size=4]
>>>>>      Region 4: I/O ports at cc00 [size=16]
>>>>>      Region 5: Memory at feaf9000 (32-bit, non-prefetchable) [size=4K]
>>>>>      Capabilities: [44] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>>>>              Address: 0000000000000000  Data: 0000
>>>>>      Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>>>>      Kernel driver in use: sata_nv
>>>>>      Kernel modules: ata_generic, pata_acpi, sata_nv
>>>>>
>>>>> 00:05.1 IDE interface: nVidia Corporation MCP55 SATA Controller (rev
>>>>> a3)
>>>>> (prog-if 85 [Master SecO PriO])
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0 (750ns min, 250ns max)
>>>>>      Interrupt: pin B routed to IRQ 20
>>>>>      Region 0: I/O ports at c880 [size=8]
>>>>>      Region 1: I/O ports at c800 [size=4]
>>>>>      Region 2: I/O ports at c480 [size=8]
>>>>>      Region 3: I/O ports at c400 [size=4]
>>>>>      Region 4: I/O ports at c080 [size=16]
>>>>>      Region 5: Memory at feaf8000 (32-bit, non-prefetchable) [size=4K]
>>>>>      Capabilities: [44] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>>>>              Address: 0000000000000000  Data: 0000
>>>>>      Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>>>>      Kernel driver in use: sata_nv
>>>>>      Kernel modules: ata_generic, pata_acpi, sata_nv
>>>>>
>>>>> 00:05.2 IDE interface: nVidia Corporation MCP55 SATA Controller (rev
>>>>> a3)
>>>>> (prog-if 85 [Master SecO PriO])
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0 (750ns min, 250ns max)
>>>>>      Interrupt: pin C routed to IRQ 23
>>>>>      Region 0: I/O ports at c000 [size=8]
>>>>>      Region 1: I/O ports at bc00 [size=4]
>>>>>      Region 2: I/O ports at b880 [size=8]
>>>>>      Region 3: I/O ports at b800 [size=4]
>>>>>      Region 4: I/O ports at b480 [size=16]
>>>>>      Region 5: Memory at feaf7000 (32-bit, non-prefetchable) [size=4K]
>>>>>      Capabilities: [44] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Capabilities: [b0] MSI: Mask- 64bit+ Count=1/4 Enable-
>>>>>              Address: 0000000000000000  Data: 0000
>>>>>      Capabilities: [cc] HyperTransport: MSI Mapping Enable- Fixed+
>>>>>      Kernel driver in use: sata_nv
>>>>>      Kernel modules: ata_generic, pata_acpi, sata_nv
>>>>>
>>>>> 00:06.0 PCI bridge: nVidia Corporation MCP55 PCI bridge (rev a2)
>>>>> (prog-if
>>>>> 01
>>>>> [Subtractive decode])
>>>>>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0
>>>>>      Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>>>>>      I/O behind bridge: 0000e000-0000efff
>>>>>      Memory behind bridge: feb00000-febfffff
>>>>>      Prefetchable memory behind bridge: f8000000-fbffffff
>>>>>      Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort-
>>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>>      BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
>>>>>              PriDiscTmr- SecDiscTmr+ DiscTmrStat- DiscTmrSERREn-
>>>>>      Capabilities: [b8] Subsystem: nVidia Corporation Device cb84
>>>>>      Capabilities: [8c] HyperTransport: MSI Mapping Enable- Fixed-
>>>>>              Mapping Address Base: 00000000fee00000
>>>>>
>>>>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0 (250ns min, 5000ns max)
>>>>>      Interrupt: pin A routed to IRQ 30
>>>>>      Region 0: Memory at feaf6000 (32-bit, non-prefetchable) [size=4K]
>>>>>      Region 1: I/O ports at b400 [size=8]
>>>>>      Region 2: Memory at feafa800 (32-bit, non-prefetchable) [size=256]
>>>>>      Region 3: Memory at feafa400 (32-bit, non-prefetchable) [size=16]
>>>>>      Capabilities: [44] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>>>>>      Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>>>>>              Vector table: BAR=2 offset=00000000
>>>>>              PBA: BAR=3 offset=00000000
>>>>>      Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>>>>>              Address: 00000000fee0400c  Data: 4199
>>>>>              Masking: 000000fe  Pending: 00000000
>>>>>      Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>>>>>      Kernel driver in use: forcedeth
>>>>>      Kernel modules: forcedeth
>>>>>
>>>>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
>>>>>      Subsystem: nVidia Corporation Device cb84
>>>>>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0 (250ns min, 5000ns max)
>>>>>      Interrupt: pin A routed to IRQ 31
>>>>>      Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
>>>>>      Region 1: I/O ports at b080 [size=8]
>>>>>      Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256]
>>>>>      Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16]
>>>>>      Capabilities: [44] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable+ DSel=0 DScale=0 PME-
>>>>>      Capabilities: [70] MSI-X: Enable- Mask- TabSize=8
>>>>>              Vector table: BAR=2 offset=00000000
>>>>>              PBA: BAR=3 offset=00000000
>>>>>      Capabilities: [50] MSI: Mask+ 64bit+ Count=1/8 Enable+
>>>>>              Address: 00000000fee0200c  Data: 41a1
>>>>>              Masking: 000000fe  Pending: 00000000
>>>>>      Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
>>>>>      Kernel driver in use: forcedeth
>>>>>      Kernel modules: forcedeth
>>>>>
>>>>> 00:0a.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev
>>>>> a3)
>>>>> (prog-if 00 [Normal decode])
>>>>>      Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0, Cache Line Size: 64 bytes
>>>>>      Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>>>>>      Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>>      BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>>              PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>>      Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>>      Capabilities: [48] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>>              Address: 00000000fee0f00c  Data: 4149
>>>>>      Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>>              Mapping Address Base: 00000000fee00000
>>>>>      Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>>              DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>>> <64ns,
>>>>> L1 <1us
>>>>>                      ExtTag- RBE+ FLReset-
>>>>>              DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>>> Unsupported-
>>>>>                      RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>>                      MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>>              DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>>> TransPend-
>>>>>              LnkCap: Port #5, Speed 2.5GT/s, Width x8, ASPM L0s L1,
>>>>> Latency L0 <512ns, L1 <4us
>>>>>                      ClockPM- Surprise- LLActRep+ BwNot-
>>>>>              LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>>> CommClk-
>>>>>                      ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>>              LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
>>>>> DLActive- BWMgmt- ABWMgmt-
>>>>>              SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>>> Surpise-
>>>>>                      Slot #  0, PowerLimit 0.000000; Interlock-
>>>>> NoCompl-
>>>>>              SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>>> HPIrq- LinkChg-
>>>>>                      Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>>              SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>>> Interlock-
>>>>>                      Changed: MRL- PresDet- LinkState-
>>>>>              RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
>>>>> PMEIntEna-
>>>>> CRSVisible-
>>>>>              RootCap: CRSVisible-
>>>>>              RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>>      Capabilities: [100] Virtual Channel <?>
>>>>>      Kernel driver in use: pcieport-driver
>>>>>      Kernel modules: shpchp
>>>>>
>>>>> 00:0b.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev
>>>>> a3)
>>>>> (prog-if 00 [Normal decode])
>>>>>      Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0, Cache Line Size: 64 bytes
>>>>>      Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
>>>>>      Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>>      BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>>              PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>>      Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>>      Capabilities: [48] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>>              Address: 00000000fee0f00c  Data: 4151
>>>>>      Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>>              Mapping Address Base: 00000000fee00000
>>>>>      Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>>              DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>>> <64ns,
>>>>> L1 <1us
>>>>>                      ExtTag- RBE+ FLReset-
>>>>>              DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>>> Unsupported-
>>>>>                      RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>>                      MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>>              DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>>> TransPend-
>>>>>              LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>>>> Latency L0 <512ns, L1 <4us
>>>>>                      ClockPM- Surprise- LLActRep+ BwNot-
>>>>>              LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>>> CommClk-
>>>>>                      ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>>              LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>>>>> DLActive- BWMgmt- ABWMgmt-
>>>>>              SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>>> Surpise-
>>>>>                      Slot #  0, PowerLimit 0.000000; Interlock-
>>>>> NoCompl-
>>>>>              SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>>> HPIrq- LinkChg-
>>>>>                      Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>>              SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>>> Interlock-
>>>>>                      Changed: MRL- PresDet- LinkState-
>>>>>              RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
>>>>> PMEIntEna-
>>>>> CRSVisible-
>>>>>              RootCap: CRSVisible-
>>>>>              RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>>      Capabilities: [100] Virtual Channel <?>
>>>>>      Kernel driver in use: pcieport-driver
>>>>>      Kernel modules: shpchp
>>>>>
>>>>> 00:0c.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev
>>>>> a3)
>>>>> (prog-if 00 [Normal decode])
>>>>>      Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0, Cache Line Size: 64 bytes
>>>>>      Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
>>>>>      Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>>      BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>>              PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>>      Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>>      Capabilities: [48] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>>              Address: 00000000fee0f00c  Data: 4159
>>>>>      Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>>              Mapping Address Base: 00000000fee00000
>>>>>      Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>>              DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>>> <64ns,
>>>>> L1 <1us
>>>>>                      ExtTag- RBE+ FLReset-
>>>>>              DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>>> Unsupported-
>>>>>                      RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>>                      MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>>              DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>>> TransPend-
>>>>>              LnkCap: Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>>>> Latency L0 <512ns, L1 <4us
>>>>>                      ClockPM- Surprise- LLActRep+ BwNot-
>>>>>              LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>>> CommClk-
>>>>>                      ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>>              LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>>>>> DLActive- BWMgmt- ABWMgmt-
>>>>>              SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>>> Surpise-
>>>>>                      Slot #  0, PowerLimit 0.000000; Interlock-
>>>>> NoCompl-
>>>>>              SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>>> HPIrq- LinkChg-
>>>>>                      Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>>              SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>>> Interlock-
>>>>>                      Changed: MRL- PresDet- LinkState-
>>>>>              RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
>>>>> PMEIntEna-
>>>>> CRSVisible-
>>>>>              RootCap: CRSVisible-
>>>>>              RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>>      Capabilities: [100] Virtual Channel <?>
>>>>>      Kernel driver in use: pcieport-driver
>>>>>      Kernel modules: shpchp
>>>>>
>>>>> 00:0d.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev
>>>>> a3)
>>>>> (prog-if 00 [Normal decode])
>>>>>      Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0, Cache Line Size: 64 bytes
>>>>>      Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
>>>>>      Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>>      BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>>              PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>>      Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>>      Capabilities: [48] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>>              Address: 00000000fee0f00c  Data: 4161
>>>>>      Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>>              Mapping Address Base: 00000000fee00000
>>>>>      Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>>              DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>>> <64ns,
>>>>> L1 <1us
>>>>>                      ExtTag- RBE+ FLReset-
>>>>>              DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>>> Unsupported-
>>>>>                      RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>>                      MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>>              DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>>> TransPend-
>>>>>              LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>>>> Latency L0 <512ns, L1 <4us
>>>>>                      ClockPM- Surprise- LLActRep+ BwNot-
>>>>>              LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>>> CommClk-
>>>>>                      ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>>              LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
>>>>> DLActive- BWMgmt- ABWMgmt-
>>>>>              SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>>> Surpise-
>>>>>                      Slot #  0, PowerLimit 0.000000; Interlock-
>>>>> NoCompl-
>>>>>              SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>>> HPIrq- LinkChg-
>>>>>                      Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>>              SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>>> Interlock-
>>>>>                      Changed: MRL- PresDet- LinkState-
>>>>>              RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
>>>>> PMEIntEna-
>>>>> CRSVisible-
>>>>>              RootCap: CRSVisible-
>>>>>              RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>>      Capabilities: [100] Virtual Channel <?>
>>>>>      Kernel driver in use: pcieport-driver
>>>>>      Kernel modules: shpchp
>>>>>
>>>>> 00:0e.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev
>>>>> a3)
>>>>> (prog-if 00 [Normal decode])
>>>>>      Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0, Cache Line Size: 64 bytes
>>>>>      Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
>>>>>      Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>>      BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>>              PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>>      Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>>      Capabilities: [48] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>>              Address: 00000000fee0f00c  Data: 4169
>>>>>      Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>>              Mapping Address Base: 00000000fee00000
>>>>>      Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>>              DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>>> <64ns,
>>>>> L1 <1us
>>>>>                      ExtTag- RBE+ FLReset-
>>>>>              DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>>> Unsupported-
>>>>>                      RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>>                      MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>>              DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>>> TransPend-
>>>>>              LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1,
>>>>> Latency L0 <512ns, L1 <4us
>>>>>                      ClockPM- Surprise- LLActRep+ BwNot-
>>>>>              LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>>> CommClk-
>>>>>                      ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>>              LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+
>>>>> DLActive- BWMgmt- ABWMgmt-
>>>>>              SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>>> Surpise-
>>>>>                      Slot #  0, PowerLimit 0.000000; Interlock-
>>>>> NoCompl-
>>>>>              SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>>> HPIrq- LinkChg-
>>>>>                      Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>>              SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>>> Interlock-
>>>>>                      Changed: MRL- PresDet- LinkState-
>>>>>              RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
>>>>> PMEIntEna-
>>>>> CRSVisible-
>>>>>              RootCap: CRSVisible-
>>>>>              RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>>      Capabilities: [100] Virtual Channel <?>
>>>>>      Kernel driver in use: pcieport-driver
>>>>>      Kernel modules: shpchp
>>>>>
>>>>> 00:0f.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev
>>>>> a3)
>>>>> (prog-if 00 [Normal decode])
>>>>>      Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx+
>>>>>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Latency: 0, Cache Line Size: 64 bytes
>>>>>      Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
>>>>>      Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- <SERR- <PERR-
>>>>>      BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>>              PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>>      Capabilities: [40] Subsystem: nVidia Corporation Device 0000
>>>>>      Capabilities: [48] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>      Capabilities: [50] MSI: Mask- 64bit+ Count=1/2 Enable+
>>>>>              Address: 00000000fee0f00c  Data: 4171
>>>>>      Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
>>>>>              Mapping Address Base: 00000000fee00000
>>>>>      Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
>>>>>              DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
>>>>> <64ns,
>>>>> L1 <1us
>>>>>                      ExtTag- RBE+ FLReset-
>>>>>              DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
>>>>> Unsupported-
>>>>>                      RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>>                      MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>>              DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
>>>>> TransPend-
>>>>>              LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1,
>>>>> Latency L0 <512ns, L1 <4us
>>>>>                      ClockPM- Surprise- LLActRep+ BwNot-
>>>>>              LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
>>>>> CommClk-
>>>>>                      ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>>>>              LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+
>>>>> DLActive- BWMgmt- ABWMgmt-
>>>>>              SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
>>>>> Surpise-
>>>>>                      Slot #  0, PowerLimit 0.000000; Interlock-
>>>>> NoCompl-
>>>>>              SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
>>>>> HPIrq- LinkChg-
>>>>>                      Control: AttnInd Off, PwrInd On, Power- Interlock-
>>>>>              SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
>>>>> Interlock-
>>>>>                      Changed: MRL- PresDet- LinkState-
>>>>>              RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal-
>>>>> PMEIntEna-
>>>>> CRSVisible-
>>>>>              RootCap: CRSVisible-
>>>>>              RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>>>>      Capabilities: [100] Virtual Channel <?>
>>>>>      Kernel driver in use: pcieport-driver
>>>>>      Kernel modules: shpchp
>>>>>
>>>>> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>>> Athlon64, Sempron] HyperTransport Configuration
>>>>>      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Capabilities: [80] HyperTransport: Host or Secondary Interface
>>>>>              Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+
>>>>> Slave- <EOCErr- DUL-
>>>>>              Link Control: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO-
>>>>> <CRCErr=0 IsocEn- LSEn+ ExtCTL- 64b-
>>>>>              Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut-
>>>>> LWI=16bit
>>>>> DwFcInEn- LWO=16bit DwFcOutEn-
>>>>>              Revision ID: 3.00
>>>>>              Link Frequency: 1.0GHz
>>>>>              Link Error: <Prot- <Ovfl- <EOC- CTLTm-
>>>>>              Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz-
>>>>> 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz+ 1.4GHz- 1.6GHz- Vend-
>>>>>              Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+
>>>>> UIDRD- ExtRS- UCnfE-
>>>>>
>>>>> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>>> Athlon64, Sempron] Address Map
>>>>>      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>
>>>>> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>>> Athlon64, Sempron] DRAM Controller
>>>>>      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Kernel modules: amd64_edac_mod
>>>>>
>>>>> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>>> Athlon64, Sempron] Miscellaneous Control
>>>>>      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      Capabilities: [f0] Secure device <?>
>>>>>
>>>>> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>>>>> Athlon64, Sempron] Link Control
>>>>>      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>
>>>>> 01:0a.0 VGA compatible controller: XGI Technology Inc. (eXtreme
>>>>> Graphics
>>>>> Innovation) Volari Z7/Z9/Z9s (prog-if 00 [VGA controller])
>>>>>      Subsystem: XGI Technology Inc. (eXtreme Graphics Innovation)
>>>>> Volari
>>>>> Z7/Z9/Z9s
>>>>>      Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
>>>>> ParErr-
>>>>> Stepping- SERR- FastB2B- DisINTx-
>>>>>      Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
>>>>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>>>      BIST result: 00
>>>>>      Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
>>>>>      Region 1: Memory at febc0000 (32-bit, non-prefetchable)
>>>>> [size=256K]
>>>>>      Region 2: I/O ports at ec00 [size=128]
>>>>>      Expansion ROM at <unassigned> [disabled]
>>>>>      Capabilities: [40] Power Management version 2
>>>>>              Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
>>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>>              Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>>
>>>>>
>>>>>
>>>>> /dev/sda:
>>>>>
>>>>> ATA device, with non-removable media
>>>>>      Model Number:       WDC WD5000AAKS-00A7B2
>>>>>      Serial Number:      WD-WCASZ0321108
>>>>>      Firmware Revision:  01.03B01
>>>>>      Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA
>>>>> Rev
>>>>> 2.5
>>>>> Standards:
>>>>>      Supported: 8 7 6 5
>>>>>      Likely used: 8
>>>>> Configuration:
>>>>>      Logical         max     current
>>>>>      cylinders       16383   16383
>>>>>      heads           16      16
>>>>>      sectors/track   63      63
>>>>>      --
>>>>>      CHS current addressable sectors:   16514064
>>>>>      LBA    user addressable sectors:  268435455
>>>>>      LBA48  user addressable sectors:  976773168
>>>>>      Logical/Physical Sector size:           512 bytes
>>>>>      device size with M = 1024*1024:      476940 MBytes
>>>>>      device size with M = 1000*1000:      500107 MBytes (500 GB)
>>>>>      cache/buffer size  = 16384 KBytes
>>>>> Capabilities:
>>>>>      LBA, IORDY(can be disabled)
>>>>>      Queue depth: 32
>>>>>      Standby timer values: spec'd by Standard, with device specific
>>>>> minimum
>>>>>      R/W multiple sector transfer: Max = 16  Current = 16
>>>>>      Recommended acoustic management value: 128, current value: 254
>>>>>      DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
>>>>>           Cycle time: min=120ns recommended=120ns
>>>>>      PIO: pio0 pio1 pio2 pio3 pio4
>>>>>           Cycle time: no flow control=120ns  IORDY flow control=120ns
>>>>> Commands/features:
>>>>>      Enabled Supported:
>>>>>         *    SMART feature set
>>>>>              Security Mode feature set
>>>>>         *    Power Management feature set
>>>>>         *    Write cache
>>>>>         *    Look-ahead
>>>>>         *    Host Protected Area feature set
>>>>>         *    WRITE_BUFFER command
>>>>>         *    READ_BUFFER command
>>>>>         *    NOP cmd
>>>>>         *    DOWNLOAD_MICROCODE
>>>>>              Power-Up In Standby feature set
>>>>>         *    SET_FEATURES required to spinup after power up
>>>>>              SET_MAX security extension
>>>>>              Automatic Acoustic Management feature set
>>>>>         *    48-bit Address feature set
>>>>>         *    Device Configuration Overlay feature set
>>>>>         *    Mandatory FLUSH_CACHE
>>>>>         *    FLUSH_CACHE_EXT
>>>>>         *    SMART error logging
>>>>>         *    SMART self-test
>>>>>         *    General Purpose Logging feature set
>>>>>         *    64-bit World wide name
>>>>>         *    Segmented DOWNLOAD_MICROCODE
>>>>>         *    Gen1 signaling speed (1.5Gb/s)
>>>>>         *    Gen2 signaling speed (3.0Gb/s)
>>>>>         *    Native Command Queueing (NCQ)
>>>>>         *    Host-initiated interface power management
>>>>>         *    Phy event counters
>>>>>              DMA Setup Auto-Activate optimization
>>>>>         *    Software settings preservation
>>>>>         *    SMART Command Transport (SCT) feature set
>>>>>         *    SCT Long Sector Access (AC1)
>>>>>         *    SCT LBA Segment Access (AC2)
>>>>>         *    SCT Error Recovery Control (AC3)
>>>>>         *    SCT Features Control (AC4)
>>>>>         *    SCT Data Tables (AC5)
>>>>>              unknown 206[12] (vendor specific)
>>>>>              unknown 206[13] (vendor specific)
>>>>> Security:
>>>>>      Master password revision code = 65534
>>>>>              supported
>>>>>      not     enabled
>>>>>      not     locked
>>>>>              frozen
>>>>>      not     expired: security count
>>>>>              supported: enhanced erase
>>>>>      84min for SECURITY ERASE UNIT. 84min for ENHANCED SECURITY ERASE
>>>>> UNIT.
>>>>> Logical Unit WWN Device Identifier: 50014ee101cedc24
>>>>>      NAA             : 5
>>>>>      IEEE OUI        : 0014ee
>>>>>      Unique ID       : 101cedc24
>>>>> Checksum: correct
>>>>>
>>>>> /dev/md0:
>>>>>      Version : 0.90
>>>>>  Creation Time : Mon Jul 27 22:38:26 2009
>>>>>   Raid Level : raid1
>>>>>   Array Size : 102336 (99.95 MiB 104.79 MB)
>>>>>  Used Dev Size : 102336 (99.95 MiB 104.79 MB)
>>>>>  Raid Devices : 4
>>>>>  Total Devices : 4
>>>>> Preferred Minor : 0
>>>>>  Persistence : Superblock is persistent
>>>>>
>>>>>  Update Time : Mon Sep 21 11:02:14 2009
>>>>>        State : clean
>>>>>  Active Devices : 4
>>>>> Working Devices : 4
>>>>>  Failed Devices : 0
>>>>>  Spare Devices : 0
>>>>>
>>>>>         UUID : 9037def2:035b0d10:bfe78010:bc810f04
>>>>>       Events : 0.44
>>>>>
>>>>>  Number   Major   Minor   RaidDevice State
>>>>>     0       8        1        0      active sync   /dev/sda1
>>>>>     1       8       17        1      active sync   /dev/sdb1
>>>>>     2       8       33        2      active sync   /dev/sdc1
>>>>>     3       8       49        3      active sync   /dev/sdd1
>>>>> /dev/md1:
>>>>>      Version : 0.90
>>>>>  Creation Time : Mon Jul 27 22:38:28 2009
>>>>>   Raid Level : raid1
>>>>>   Array Size : 4095936 (3.91 GiB 4.19 GB)
>>>>>  Used Dev Size : 4095936 (3.91 GiB 4.19 GB)
>>>>>  Raid Devices : 4
>>>>>  Total Devices : 4
>>>>> Preferred Minor : 1
>>>>>  Persistence : Superblock is persistent
>>>>>
>>>>>  Update Time : Mon Sep 21 11:02:14 2009
>>>>>        State : clean
>>>>>  Active Devices : 4
>>>>> Working Devices : 4
>>>>>  Failed Devices : 0
>>>>>  Spare Devices : 0
>>>>>
>>>>>         UUID : 642c2833:bf033a7d:bfe78010:bc810f04
>>>>>       Events : 0.44
>>>>>
>>>>>  Number   Major   Minor   RaidDevice State
>>>>>     0       8        2        0      active sync   /dev/sda2
>>>>>     1       8       18        1      active sync   /dev/sdb2
>>>>>     2       8       34        2      active sync   /dev/sdc2
>>>>>     3       8       50        3      active sync   /dev/sdd2
>>>>> /dev/md2:
>>>>>      Version : 0.90
>>>>>  Creation Time : Mon Jul 27 22:38:30 2009
>>>>>   Raid Level : raid1
>>>>>   Array Size : 484185536 (461.76 GiB 495.81 GB)
>>>>>  Used Dev Size : 484185536 (461.76 GiB 495.81 GB)
>>>>>  Raid Devices : 4
>>>>>  Total Devices : 4
>>>>> Preferred Minor : 2
>>>>>  Persistence : Superblock is persistent
>>>>>
>>>>>  Update Time : Mon Sep 21 11:02:24 2009
>>>>>        State : clean
>>>>>  Active Devices : 4
>>>>> Working Devices : 4
>>>>>  Failed Devices : 0
>>>>>  Spare Devices : 0
>>>>>
>>>>>         UUID : ce106d52:0c0373f7:bfe78010:bc810f04
>>>>>       Events : 0.66
>>>>>
>>>>>  Number   Major   Minor   RaidDevice State
>>>>>     0       8        3        0      active sync   /dev/sda3
>>>>>     1       8       19        1      active sync   /dev/sdb3
>>>>>     2       8       35        2      active sync   /dev/sdc3
>>>>>     3       8       51        3      active sync   /dev/sdd3
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-22 14:59       ` Lee Howard
  2009-09-22 15:13         ` Majed B.
@ 2009-09-22 15:19         ` Robin Hill
  2009-09-22 19:35           ` Gabriele Trombetti
  1 sibling, 1 reply; 14+ messages in thread
From: Robin Hill @ 2009-09-22 15:19 UTC (permalink / raw)
  To: linux-raid

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

On Tue Sep 22, 2009 at 07:59:45AM -0700, Lee Howard wrote:

> Majed B. wrote:
> > I must have missed that part. It may not work for your case, but worth trying.
> >
> > Perhaps Neil Brown, or someone involved could shed some light on this.
> >
> > If I remember correctly, those soft lockups were harmless anyway.
> >   
> 
> Not harmless for production use.  Yes, data is not harmed, and yes, the 
> problem state does recover when the data-check finishes, but during the 
> data-check the system is virtually unresponsive and all other use of the 
> system is stalled.
> 
Are you sure this is caused by these soft lockups, and that you're not
just running with too high a /sys/block/mdX/md/sync_speed_max setting?
I've had issues with this on some servers, where the I/O demand for the
sync/check is causing the system to become totally unresponsive.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-22 15:19         ` Robin Hill
@ 2009-09-22 19:35           ` Gabriele Trombetti
  2009-09-23  0:16             ` Majed B.
  0 siblings, 1 reply; 14+ messages in thread
From: Gabriele Trombetti @ 2009-09-22 19:35 UTC (permalink / raw)
  To: linux-raid

Robin Hill wrote:
> On Tue Sep 22, 2009 at 07:59:45AM -0700, Lee Howard wrote:
>
>   
>> Majed B. wrote:
>>     
>>> I must have missed that part. It may not work for your case, but worth trying.
>>>
>>> Perhaps Neil Brown, or someone involved could shed some light on this.
>>>
>>> If I remember correctly, those soft lockups were harmless anyway.
>>>   
>>>       
>> Not harmless for production use.  Yes, data is not harmed, and yes, the 
>> problem state does recover when the data-check finishes, but during the 
>> data-check the system is virtually unresponsive and all other use of the 
>> system is stalled.
>>
>>     
> Are you sure this is caused by these soft lockups, and that you're not
> just running with too high a /sys/block/mdX/md/sync_speed_max setting?
> I've had issues with this on some servers, where the I/O demand for the
> sync/check is causing the system to become totally unresponsive.
>   

That's correct for me in the sense that lowering sync_speed_max solves
the problem, see my post, however I'd call it a bug if a value of
sync_speed_max too high starves the system forever. The resync is
supposed to be less prioritarian than normal I/O disk operations, but it
doesn't happen this way. Also note that lowering the value of
stripe_cache_size also solves the problem: how would this fit into your
reasoning?

(BTW I have not checked the mentioned patch yet, I'm not sure I can do
that in a short time because our servers are into production now)


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-22 19:35           ` Gabriele Trombetti
@ 2009-09-23  0:16             ` Majed B.
  2009-09-23  1:05               ` Guy Watkins
  0 siblings, 1 reply; 14+ messages in thread
From: Majed B. @ 2009-09-23  0:16 UTC (permalink / raw)
  To: Gabriele Trombetti; +Cc: linux-raid

Why would you lower the max value? You should keep the min value as
low as possible and md would drop to that automatically if there are
applications demanding access to the array.

On Tue, Sep 22, 2009 at 10:35 PM, Gabriele Trombetti
<gabriele.trombetti@itb.cnr.it> wrote:
> Robin Hill wrote:
>>
>> On Tue Sep 22, 2009 at 07:59:45AM -0700, Lee Howard wrote:
>>
>>
>>>
>>> Majed B. wrote:
>>>
>>>>
>>>> I must have missed that part. It may not work for your case, but worth
>>>> trying.
>>>>
>>>> Perhaps Neil Brown, or someone involved could shed some light on this.
>>>>
>>>> If I remember correctly, those soft lockups were harmless anyway.
>>>>
>>>
>>> Not harmless for production use.  Yes, data is not harmed, and yes, the
>>> problem state does recover when the data-check finishes, but during the
>>> data-check the system is virtually unresponsive and all other use of the
>>> system is stalled.
>>>
>>>
>>
>> Are you sure this is caused by these soft lockups, and that you're not
>> just running with too high a /sys/block/mdX/md/sync_speed_max setting?
>> I've had issues with this on some servers, where the I/O demand for the
>> sync/check is causing the system to become totally unresponsive.
>>
>
> That's correct for me in the sense that lowering sync_speed_max solves
> the problem, see my post, however I'd call it a bug if a value of
> sync_speed_max too high starves the system forever. The resync is
> supposed to be less prioritarian than normal I/O disk operations, but it
> doesn't happen this way. Also note that lowering the value of
> stripe_cache_size also solves the problem: how would this fit into your
> reasoning?
>
> (BTW I have not checked the mentioned patch yet, I'm not sure I can do
> that in a short time because our servers are into production now)
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 14+ messages in thread

* RE: md data-check causes soft lockup
  2009-09-23  0:16             ` Majed B.
@ 2009-09-23  1:05               ` Guy Watkins
  0 siblings, 0 replies; 14+ messages in thread
From: Guy Watkins @ 2009-09-23  1:05 UTC (permalink / raw)
  To: 'Majed B.', 'Gabriele Trombetti'; +Cc: 'linux-raid'

But if the applications are locked out, they can't demand anything.  I have
seen the same on my Linux server, but only with the 2.6 kernel.  The same
hardware with a 2.4 kernel was fine.  I have not seen this myself for at
least 1 year, I assumed it was fixed.

When I was locked out my putty session would not respond.  I don't think it
timed out, but recovered when the rebuild/resync was done.

} -----Original Message-----
} From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
} owner@vger.kernel.org] On Behalf Of Majed B.
} Sent: Tuesday, September 22, 2009 8:17 PM
} To: Gabriele Trombetti
} Cc: linux-raid
} Subject: Re: md data-check causes soft lockup
} 
} Why would you lower the max value? You should keep the min value as
} low as possible and md would drop to that automatically if there are
} applications demanding access to the array.
} 
} On Tue, Sep 22, 2009 at 10:35 PM, Gabriele Trombetti
} <gabriele.trombetti@itb.cnr.it> wrote:
} > Robin Hill wrote:
} >>
} >> On Tue Sep 22, 2009 at 07:59:45AM -0700, Lee Howard wrote:
} >>
} >>
} >>>
} >>> Majed B. wrote:
} >>>
} >>>>
} >>>> I must have missed that part. It may not work for your case, but
} worth
} >>>> trying.
} >>>>
} >>>> Perhaps Neil Brown, or someone involved could shed some light on
} this.
} >>>>
} >>>> If I remember correctly, those soft lockups were harmless anyway.
} >>>>
} >>>
} >>> Not harmless for production use.  Yes, data is not harmed, and yes,
} the
} >>> problem state does recover when the data-check finishes, but during
} the
} >>> data-check the system is virtually unresponsive and all other use of
} the
} >>> system is stalled.
} >>>
} >>>
} >>
} >> Are you sure this is caused by these soft lockups, and that you're not
} >> just running with too high a /sys/block/mdX/md/sync_speed_max setting?
} >> I've had issues with this on some servers, where the I/O demand for the
} >> sync/check is causing the system to become totally unresponsive.
} >>
} >
} > That's correct for me in the sense that lowering sync_speed_max solves
} > the problem, see my post, however I'd call it a bug if a value of
} > sync_speed_max too high starves the system forever. The resync is
} > supposed to be less prioritarian than normal I/O disk operations, but it
} > doesn't happen this way. Also note that lowering the value of
} > stripe_cache_size also solves the problem: how would this fit into your
} > reasoning?
} >
} > (BTW I have not checked the mentioned patch yet, I'm not sure I can do
} > that in a short time because our servers are into production now)
} >
} > --
} > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
} > the body of a message to majordomo@vger.kernel.org
} > More majordomo info at  http://vger.kernel.org/majordomo-info.html
} >
} 
} 
} 
} --
}        Majed B.
} --
} To unsubscribe from this list: send the line "unsubscribe linux-raid" in
} the body of a message to majordomo@vger.kernel.org
} More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-21 18:08 md data-check causes soft lockup Lee Howard
  2009-09-21 18:54 ` Majed B.
  2009-09-21 19:13 ` kwick
@ 2009-09-25  6:54 ` Neil Brown
  2009-09-25 11:01   ` kwick
  2 siblings, 1 reply; 14+ messages in thread
From: Neil Brown @ 2009-09-25  6:54 UTC (permalink / raw)
  To: Lee Howard; +Cc: dan.j.williams, mingo, linux-raid

On Monday September 21, faxguy@howardsilvan.com wrote:
> Linux software RAID maintainers and developers,
> 
> Two months ago I wrote to the Linux kernel mailing list regarding a 
> condition expressed as "BUG: soft lockup - CPU#3 stuck for 61s!".  I 
> initially battled this recurring problem in both Fedora 10 and Fedora 
> 11.  Rafael J. Wysocki suggested that I update the kernel (to 2.6.31-rc4 
> or later) and see if the problem resurfaced.  I then used kernel 
> 2.6.31-0.94.rc4.fc12.x86_64 and found that the problem still continued, 
> but noticeably only when the md data-check process was run.
> 
> You can read the last post to the LKML thread (with links to the entire 
> thread) here:
> 
> http://lkml.org/lkml/2009/8/6/387

Thanks for the report.

It looks like the difference between the cpu/RAM speed and the drive
speed is small enough that the CPU gets stuck comparing lots of blocks
for multiple seconds.

This patch should fix it.  I'll see that it goes upstream.

NeilBrown

--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1703,6 +1703,7 @@ static void raid1d(mddev_t *mddev)
 				generic_make_request(bio);
 			}
 		}
+		cond_resched();
 	}
 	if (unplug)
 		unplug_slaves(mddev);

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-25  6:54 ` Neil Brown
@ 2009-09-25 11:01   ` kwick
  2009-09-25 11:23     ` NeilBrown
  0 siblings, 1 reply; 14+ messages in thread
From: kwick @ 2009-09-25 11:01 UTC (permalink / raw)
  To: linux-raid

Neil Brown wrote:
> On Monday September 21, faxguy@howardsilvan.com wrote:
>> http://lkml.org/lkml/2009/8/6/387
>>     
>
> Thanks for the report.
>
> This patch...
Dear Neil,

as you can see I reported a similar problem for raid-6 in this thread.
Your patch apparently covers raid-1. Is it because this other patch
http://osdir.com/ml/linux-raid/2009-09/msg00371.html
covers my case? it's all fixed then?

Thank you

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: md data-check causes soft lockup
  2009-09-25 11:01   ` kwick
@ 2009-09-25 11:23     ` NeilBrown
  0 siblings, 0 replies; 14+ messages in thread
From: NeilBrown @ 2009-09-25 11:23 UTC (permalink / raw)
  To: kwick; +Cc: linux-raid

On Fri, September 25, 2009 9:01 pm, kwick wrote:
> Neil Brown wrote:
>> On Monday September 21, faxguy@howardsilvan.com wrote:
>>> http://lkml.org/lkml/2009/8/6/387
>>>
>>
>> Thanks for the report.
>>
>> This patch...
> Dear Neil,
>
> as you can see I reported a similar problem for raid-6 in this thread.
> Your patch apparently covers raid-1. Is it because this other patch
> http://osdir.com/ml/linux-raid/2009-09/msg00371.html
> covers my case? it's all fixed then?
>

Yes and yes .... unless there is a similar problem with raid10...
OK, I just added the same fix for RAID10 so it should all be fixed
(once these patches get upstream).

Thanks,
NeilBrown


> Thank you
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2009-09-25 11:23 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-09-21 18:08 md data-check causes soft lockup Lee Howard
2009-09-21 18:54 ` Majed B.
2009-09-22 14:43   ` Lee Howard
2009-09-22 14:48     ` Majed B.
2009-09-22 14:59       ` Lee Howard
2009-09-22 15:13         ` Majed B.
2009-09-22 15:19         ` Robin Hill
2009-09-22 19:35           ` Gabriele Trombetti
2009-09-23  0:16             ` Majed B.
2009-09-23  1:05               ` Guy Watkins
2009-09-21 19:13 ` kwick
2009-09-25  6:54 ` Neil Brown
2009-09-25 11:01   ` kwick
2009-09-25 11:23     ` NeilBrown

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.