pci-passthrough in pvops causing offline raid

pci-passthrough in pvops causing offline raid

[Mark Adams]

Hi All,

Running xen 4.0.1-rc6, debian squeeze 2.6.32-21.

In a voip setup, where I have forwarded the onboard NIC interfaces
through to domU using the following grub config:

module  /vmlinuz-2.6.32-5-xen-amd64 placeholder root=UUID=25c3ac79-6850-498d-afcf-ea42970e94fd ro  quiet xen-pciback.permissive xen-pciback.hide=(02:00.0)(03:00.0) pci=resource_alignment=02:00.0;03:00.0

I'm having a serious issue where the raid card goes offline after an
indefinate period of time. Sometimes runs fine for a week, other times 1
day before I get "offline device" errors. Rebooting the machine fixes it
straight away, and everything is back online.

What in the Xen pciback is causing the raid card to go offline? The
only devices hidden are the 2 onboard NIC's.

I know that this issue is with Xen, as I had this running on a different
server (same xen setup) and it had the same issues, which I initially
thought were to do with the raid card.

Is there known issues in this kernel and xen version with pciback? I'm
going to update to the current package versions this evening (4.0.1-1
and 2.6.32-27) however would appreciate if anyone has any other insight
into this issue, or even just a note to say it is a bug that has been
fixed in current versions!

Thanks,
Mark

Re: pci-passthrough in pvops causing offline raid

[Olivier Hanesse]

[-- Attachment #1.1: Type: text/plain, Size: 1677 bytes --]

Hello,

What is the module of your raid card ?
If it is "megaraid_sas", please try upgrading the version of that module
(see others post in the ML).

Regards

2010/11/11 Mark Adams <mark@campbell-lange.net>

> Hi All,
>
> Running xen 4.0.1-rc6, debian squeeze 2.6.32-21.
>
> In a voip setup, where I have forwarded the onboard NIC interfaces
> through to domU using the following grub config:
>
> module  /vmlinuz-2.6.32-5-xen-amd64 placeholder
> root=UUID=25c3ac79-6850-498d-afcf-ea42970e94fd ro  quiet
> xen-pciback.permissive xen-pciback.hide=(02:00.0)(03:00.0)
> pci=resource_alignment=02:00.0;03:00.0
>
> I'm having a serious issue where the raid card goes offline after an
> indefinate period of time. Sometimes runs fine for a week, other times 1
> day before I get "offline device" errors. Rebooting the machine fixes it
> straight away, and everything is back online.
>
> What in the Xen pciback is causing the raid card to go offline? The
> only devices hidden are the 2 onboard NIC's.
>
> I know that this issue is with Xen, as I had this running on a different
> server (same xen setup) and it had the same issues, which I initially
> thought were to do with the raid card.
>
> Is there known issues in this kernel and xen version with pciback? I'm
> going to update to the current package versions this evening (4.0.1-1
> and 2.6.32-27) however would appreciate if anyone has any other insight
> into this issue, or even just a note to say it is a bug that has been
> fixed in current versions!
>
> Thanks,
> Mark
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

[-- Attachment #1.2: Type: text/html, Size: 2215 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: Re: [Xen-users] pci-passthrough in pvops causing offline raid

[Mark Adams]

Hi - It's not megaraid, its an Areca card (arcmsr)

This is definately something to do with the pciback. Anyone else got any
ideas or views on this? The domU is an HVM debian-squeeze instance.

On Thu, Nov 11, 2010 at 12:13:31PM +0100, Olivier Hanesse wrote:
> Hello,
> 
> What is the module of your raid card ?
> If it is "megaraid_sas", please try upgrading the version of that module
> (see others post in the ML).
> 
> Regards
> 
> 2010/11/11 Mark Adams <mark@campbell-lange.net>
> 
> > Hi All,
> >
> > Running xen 4.0.1-rc6, debian squeeze 2.6.32-21.
> >
> > In a voip setup, where I have forwarded the onboard NIC interfaces
> > through to domU using the following grub config:
> >
> > module  /vmlinuz-2.6.32-5-xen-amd64 placeholder
> > root=UUID=25c3ac79-6850-498d-afcf-ea42970e94fd ro  quiet
> > xen-pciback.permissive xen-pciback.hide=(02:00.0)(03:00.0)
> > pci=resource_alignment=02:00.0;03:00.0
> >
> > I'm having a serious issue where the raid card goes offline after an
> > indefinate period of time. Sometimes runs fine for a week, other times 1
> > day before I get "offline device" errors. Rebooting the machine fixes it
> > straight away, and everything is back online.
> >
> > What in the Xen pciback is causing the raid card to go offline? The
> > only devices hidden are the 2 onboard NIC's.
> >
> > I know that this issue is with Xen, as I had this running on a different
> > server (same xen setup) and it had the same issues, which I initially
> > thought were to do with the raid card.
> >
> > Is there known issues in this kernel and xen version with pciback? I'm
> > going to update to the current package versions this evening (4.0.1-1
> > and 2.6.32-27) however would appreciate if anyone has any other insight
> > into this issue, or even just a note to say it is a bug that has been
> > fixed in current versions!
> >
> > Thanks,
> > Mark
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users
> >

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Konrad Rzeszutek Wilk]

On Thu, Nov 11, 2010 at 10:24:17AM +0000, Mark Adams wrote:
> Hi All,
> 
> Running xen 4.0.1-rc6, debian squeeze 2.6.32-21.
> 
> In a voip setup, where I have forwarded the onboard NIC interfaces
> through to domU using the following grub config:
> 
> module  /vmlinuz-2.6.32-5-xen-amd64 placeholder root=UUID=25c3ac79-6850-498d-afcf-ea42970e94fd ro  quiet xen-pciback.permissive xen-pciback.hide=(02:00.0)(03:00.0) pci=resource_alignment=02:00.0;03:00.0
> 
> I'm having a serious issue where the raid card goes offline after an
> indefinate period of time. Sometimes runs fine for a week, other times 1
> day before I get "offline device" errors. Rebooting the machine fixes it
> straight away, and everything is back online.
> 
> What in the Xen pciback is causing the raid card to go offline? The
> only devices hidden are the 2 onboard NIC's.

You need to give more details. Is the RAID card a 3Ware? An LSI? Do you
run with an IOMMU? When the RAID card goes offline, do you see a stop of
IRQs going to the device? Are the IRQs for the RAID card sent to all of your
CPUs or just a specific one? Are you pinning your guests to specific CPUs?
Does the issue disappear if you don't passthrough the NIC interfaces? If so have
you run this setup for "a week" to make sure?
> 
> I know that this issue is with Xen, as I had this running on a different
> server (same xen setup) and it had the same issues, which I initially
> thought were to do with the raid card.

So you never ran this setup on this kernel (2.6.32-5) without the Xen hypervisor?

> 
> Is there known issues in this kernel and xen version with pciback? I'm

No. It all works perfectly :-)

> going to update to the current package versions this evening (4.0.1-1
> and 2.6.32-27) however would appreciate if anyone has any other insight
> into this issue, or even just a note to say it is a bug that has been
> fixed in current versions!

Well, there were issues with the LSI cards having a hidden PCI device. But those
are pretty obvious as you can't even use it correctly. There is also
a problem with 3Ware 9506 IDE card - which on my box stops sending IRQs
on the IOAPIC it has been assigned (28) and instead uses another one (17).
Not sure if this is just the PCI card using the wrong PCI interrupt pin on the
card and it ends up poking the wrong IOAPIC.

Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Mark Adams]

On Thu, Nov 11, 2010 at 11:53:40AM -0500, Konrad Rzeszutek Wilk wrote:
> On Thu, Nov 11, 2010 at 10:24:17AM +0000, Mark Adams wrote:
> > Hi All,
> > 
> > Running xen 4.0.1-rc6, debian squeeze 2.6.32-21.
> > 
> > In a voip setup, where I have forwarded the onboard NIC interfaces
> > through to domU using the following grub config:
> > 
> > module  /vmlinuz-2.6.32-5-xen-amd64 placeholder root=UUID=25c3ac79-6850-498d-afcf-ea42970e94fd ro  quiet xen-pciback.permissive xen-pciback.hide=(02:00.0)(03:00.0) pci=resource_alignment=02:00.0;03:00.0
> > 
> > I'm having a serious issue where the raid card goes offline after an
> > indefinate period of time. Sometimes runs fine for a week, other times 1
> > day before I get "offline device" errors. Rebooting the machine fixes it
> > straight away, and everything is back online.
> > 
> > What in the Xen pciback is causing the raid card to go offline? The
> > only devices hidden are the 2 onboard NIC's.
> 
> You need to give more details. Is the RAID card a 3Ware? An LSI? Do you
> run with an IOMMU? When the RAID card goes offline, do you see a stop of
> IRQs going to the device? Are the IRQs for the RAID card sent to all of your
> CPUs or just a specific one? Are you pinning your guests to specific CPUs?
> Does the issue disappear if you don't passthrough the NIC interfaces? If so have
> you run this setup for "a week" to make sure?

It is an Areca 1220. I can't see anything when the device goes offline
apart from 

    [77324.264270] sd 0:0:0:1: rejecting I/O to offline device
    [77334.005854] sd 0:0:0:0: rejecting I/O to offline device

Unfortunately nothing get's logged because there is nothing to write to
anymore. I'm not sure how I can see the IRQs otherwise. There is no
pinning being done at all, and the machine was running for a few months
OK before the pciback was added.

Is my kernel module line correct above? are the xen-pciback.permissive
and resource_alignment options required? Also I am passing through the
onboard NIC's - is this something that should be avoided or is it ok to
do?

> > 
> > I know that this issue is with Xen, as I had this running on a different
> > server (same xen setup) and it had the same issues, which I initially
> > thought were to do with the raid card.
> 
> So you never ran this setup on this kernel (2.6.32-5) without the Xen hypervisor?

no, its always had the hypervisor - but it was running ok before the
pciback options were added. This week, it's seemed to happen
approximately every 24 hours.

> 
> > 
> > Is there known issues in this kernel and xen version with pciback? I'm
> 
> No. It all works perfectly :-)
> 
> > going to update to the current package versions this evening (4.0.1-1
> > and 2.6.32-27) however would appreciate if anyone has any other insight
> > into this issue, or even just a note to say it is a bug that has been
> > fixed in current versions!
> 
> Well, there were issues with the LSI cards having a hidden PCI device. But those
> are pretty obvious as you can't even use it correctly. There is also
> a problem with 3Ware 9506 IDE card - which on my box stops sending IRQs
> on the IOAPIC it has been assigned (28) and instead uses another one (17).
> Not sure if this is just the PCI card using the wrong PCI interrupt pin on the
> card and it ends up poking the wrong IOAPIC.

Thanks,
Mark

Re: pci-passthrough in pvops causing offline raid

[Konrad Rzeszutek Wilk]

On Thu, Nov 11, 2010 at 05:38:50PM +0000, Mark Adams wrote:
> On Thu, Nov 11, 2010 at 11:53:40AM -0500, Konrad Rzeszutek Wilk wrote:
> > On Thu, Nov 11, 2010 at 10:24:17AM +0000, Mark Adams wrote:
> > > Hi All,
> > > 
> > > Running xen 4.0.1-rc6, debian squeeze 2.6.32-21.
> > > 
> > > In a voip setup, where I have forwarded the onboard NIC interfaces
> > > through to domU using the following grub config:
> > > 
> > > module  /vmlinuz-2.6.32-5-xen-amd64 placeholder root=UUID=25c3ac79-6850-498d-afcf-ea42970e94fd ro  quiet xen-pciback.permissive xen-pciback.hide=(02:00.0)(03:00.0) pci=resource_alignment=02:00.0;03:00.0
> > > 
> > > I'm having a serious issue where the raid card goes offline after an
> > > indefinate period of time. Sometimes runs fine for a week, other times 1
> > > day before I get "offline device" errors. Rebooting the machine fixes it
> > > straight away, and everything is back online.
> > > 
> > > What in the Xen pciback is causing the raid card to go offline? The
> > > only devices hidden are the 2 onboard NIC's.
> > 
> > You need to give more details. Is the RAID card a 3Ware? An LSI? Do you
> > run with an IOMMU? When the RAID card goes offline, do you see a stop of
> > IRQs going to the device? Are the IRQs for the RAID card sent to all of your
> > CPUs or just a specific one? Are you pinning your guests to specific CPUs?
> > Does the issue disappear if you don't passthrough the NIC interfaces? If so have
> > you run this setup for "a week" to make sure?
> 
> It is an Areca 1220. I can't see anything when the device goes offline
> apart from 
> 
>     [77324.264270] sd 0:0:0:1: rejecting I/O to offline device
>     [77334.005854] sd 0:0:0:0: rejecting I/O to offline device

That is it? No other details from the driver? Did you poke at the driver (modinfo)
to see if there are any options to increase its verbosity.

> 
> Unfortunately nothing get's logged because there is nothing to write to
> anymore. I'm not sure how I can see the IRQs otherwise. There is no

cat /proc/interrupts

> pinning being done at all, and the machine was running for a few months
> OK before the pciback was added.

Ok, what about your NICs? Are they on-board? Are they sharing the IRQ
with the card? You should be able to see this by looking at /proc/interrupts.
Which NICs are they? lspci can you help you there. As of matter of fact, run
lspci -vvv and send that.
> 
> Is my kernel module line correct above? are the xen-pciback.permissive
> and resource_alignment options required? Also I am passing through the

Not always. The resource_alignment only if the BARs (look at lspci output) are
not page-aligned. If you have no idea what I am talking about then the answer
is yes.

> onboard NIC's - is this something that should be avoided or is it ok to
> do?

It is fine. That is the first thing I test..

> 
> > > 
> > > I know that this issue is with Xen, as I had this running on a different
> > > server (same xen setup) and it had the same issues, which I initially
> > > thought were to do with the raid card.
> > 
> > So you never ran this setup on this kernel (2.6.32-5) without the Xen hypervisor?
> 
> no, its always had the hypervisor - but it was running ok before the
> pciback options were added. This week, it's seemed to happen
> approximately every 24 hours.

When this hang occurs, can you do 'xm debug-key Q', 'xm debug-key i', 'xm debug-key z'.
Then run 'xm dmesg' and provide that to me?

Is your boot disk on the same disk as the RAID?

Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Mark Adams]

[-- Attachment #1: Type: text/plain, Size: 4363 bytes --]

On Thu, Nov 11, 2010 at 12:58:09PM -0500, Konrad Rzeszutek Wilk wrote:
> On Thu, Nov 11, 2010 at 05:38:50PM +0000, Mark Adams wrote:
> > On Thu, Nov 11, 2010 at 11:53:40AM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Thu, Nov 11, 2010 at 10:24:17AM +0000, Mark Adams wrote:
> > > > Hi All,
> > > > 
> > > > Running xen 4.0.1-rc6, debian squeeze 2.6.32-21.
> > > > 
> > > > In a voip setup, where I have forwarded the onboard NIC interfaces
> > > > through to domU using the following grub config:
> > > > 
> > > > module  /vmlinuz-2.6.32-5-xen-amd64 placeholder root=UUID=25c3ac79-6850-498d-afcf-ea42970e94fd ro  quiet xen-pciback.permissive xen-pciback.hide=(02:00.0)(03:00.0) pci=resource_alignment=02:00.0;03:00.0
> > > > 
> > > > I'm having a serious issue where the raid card goes offline after an
> > > > indefinate period of time. Sometimes runs fine for a week, other times 1
> > > > day before I get "offline device" errors. Rebooting the machine fixes it
> > > > straight away, and everything is back online.
> > > > 
> > > > What in the Xen pciback is causing the raid card to go offline? The
> > > > only devices hidden are the 2 onboard NIC's.
> > > 
> > > You need to give more details. Is the RAID card a 3Ware? An LSI? Do you
> > > run with an IOMMU? When the RAID card goes offline, do you see a stop of
> > > IRQs going to the device? Are the IRQs for the RAID card sent to all of your
> > > CPUs or just a specific one? Are you pinning your guests to specific CPUs?
> > > Does the issue disappear if you don't passthrough the NIC interfaces? If so have
> > > you run this setup for "a week" to make sure?
> > 
> > It is an Areca 1220. I can't see anything when the device goes offline
> > apart from 
> > 
> >     [77324.264270] sd 0:0:0:1: rejecting I/O to offline device
> >     [77334.005854] sd 0:0:0:0: rejecting I/O to offline device
> 
> That is it? No other details from the driver? Did you poke at the driver (modinfo)
> to see if there are any options to increase its verbosity.

I can't do anything once its happened, everything is offline so I have
no utils...
> 
> > 
> > Unfortunately nothing get's logged because there is nothing to write to
> > anymore. I'm not sure how I can see the IRQs otherwise. There is no
> 
> cat /proc/interrupts
> 
> > pinning being done at all, and the machine was running for a few months
> > OK before the pciback was added.
> 
> Ok, what about your NICs? Are they on-board? Are they sharing the IRQ
> with the card? You should be able to see this by looking at /proc/interrupts.
> Which NICs are they? lspci can you help you there. As of matter of fact, run
> lspci -vvv and send that.

It is the onboard nics, they are Intel 82574L. I can see the arcmsr
line, but not anything for the NICS (because they are hidden?)

39:    1126249          0          0          0          0          0         0          0  xen-pirq-ioapic-level  arcmsr

Nothing else is on 1126249

see lspci.txt attached.

> > 
> > Is my kernel module line correct above? are the xen-pciback.permissive
> > and resource_alignment options required? Also I am passing through the
> 
> Not always. The resource_alignment only if the BARs (look at lspci output) are
> not page-aligned. If you have no idea what I am talking about then the answer
> is yes.
> 
> > onboard NIC's - is this something that should be avoided or is it ok to
> > do?
> 
> It is fine. That is the first thing I test..
> 
> > 
> > > > 
> > > > I know that this issue is with Xen, as I had this running on a different
> > > > server (same xen setup) and it had the same issues, which I initially
> > > > thought were to do with the raid card.
> > > 
> > > So you never ran this setup on this kernel (2.6.32-5) without the Xen hypervisor?
> > 
> > no, its always had the hypervisor - but it was running ok before the
> > pciback options were added. This week, it's seemed to happen
> > approximately every 24 hours.
> 
> When this hang occurs, can you do 'xm debug-key Q', 'xm debug-key i', 'xm debug-key z'.
> Then run 'xm dmesg' and provide that to me?

I can try this, but It probably won't work as the device is will not be
readable.
> 
> Is your boot disk on the same disk as the RAID?

There are 2 raids, a Raid1 for the OS (/boot / /var /tmp /usr) and a
raid5 for VM's - They both dissapear at the same time so it appears the
card is dissapearing..


[-- Attachment #2: lspci.txt --]
[-- Type: text/plain, Size: 94196 bytes --]

00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit-
		Address: 00000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>

00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=0f, subordinate=0f, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee005f8  Data: 0000
		Masking: 00000003  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet+ LinkState-
		RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=0e, subordinate=0e, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00618  Data: 0000
		Masking: 00000003  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #4, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet+ LinkState-
		RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=08, subordinate=0d, sec-latency=0
	I/O behind bridge: 0000d000-0000efff
	Memory behind bridge: f9f00000-fbefffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00638  Data: 0000
		Masking: 00000003  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #3, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=05, subordinate=07, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: f9e00000-f9efffff
	Prefetchable memory behind bridge: 00000000f7c00000-00000000f7ffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00658  Data: 0000
		Masking: 00000003  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #5, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+ ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:13.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 13) (prog-if 20 [IO(X)-APIC])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Region 0: Memory at fec8a000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [6c] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB

00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB

00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB

00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 13) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 43
	Region 0: Memory at f8ed4000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin B routed to IRQ 44
	Region 0: Memory at f8ed8000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin C routed to IRQ 45
	Region 0: Memory at f8edc000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin D routed to IRQ 46
	Region 0: Memory at f8ee0000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 43
	Region 0: Memory at f8ee4000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin B routed to IRQ 44
	Region 0: Memory at f8ee8000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin C routed to IRQ 45
	Region 0: Memory at f8eec000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin D routed to IRQ 46
	Region 0: Memory at f8ef0000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 (prog-if 00 [UHCI])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 16
	Region 4: I/O ports at 7400 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 (prog-if 00 [UHCI])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 21
	Region 4: I/O ports at 7480 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 (prog-if 00 [UHCI])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin D routed to IRQ 19
	Region 4: I/O ports at 7800 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 (prog-if 20 [EHCI])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at f8ef6000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ehci_hcd
	Kernel modules: ehci-hcd

00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
	Subsystem: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 22
	Region 0: Memory at f8ef8000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE- FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=0 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
	Capabilities: [130 v1] Root Complex Link
		Desc:	PortNumber=0f ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: HDA Intel
	Kernel modules: snd-hda-intel

00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 00001000-00001fff
	Memory behind bridge: 80000000-801fffff
	Prefetchable memory behind bridge: 0000000080200000-00000000803fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00678  Data: 0000
	Capabilities: [90] Subsystem: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Root Complex Link
		Desc:	PortNumber=01 ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: f9d00000-f9dfffff
	Prefetchable memory behind bridge: 0000000080400000-00000000805fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #5, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00698  Data: 0000
	Capabilities: [90] Subsystem: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Root Complex Link
		Desc:	PortNumber=05 ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000b000-0000bfff
	Memory behind bridge: f9c00000-f9cfffff
	Prefetchable memory behind bridge: 0000000080600000-00000000807fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #6, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee006b8  Data: 0000
	Capabilities: [90] Subsystem: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Root Complex Link
		Desc:	PortNumber=06 ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 (prog-if 00 [UHCI])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 23
	Region 4: I/O ports at 7880 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 (prog-if 00 [UHCI])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 19
	Region 4: I/O ports at 7c00 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 (prog-if 00 [UHCI])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin C routed to IRQ 18
	Region 4: I/O ports at 8000 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 (prog-if 20 [EHCI])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 23
	Region 0: Memory at f8efc000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ehci_hcd
	Kernel modules: ehci-hcd

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
	I/O behind bridge: 0000a000-0000afff
	Memory behind bridge: f8f00000-f9bfffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR+
	BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] Subsystem: Intel Corporation 82801 PCI Bridge

00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
	Subsystem: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>
	Kernel modules: iTCO_wdt

00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1 (prog-if 8f [Master SecP SecO PriP PriO])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 19
	Region 0: I/O ports at 8c00 [size=8]
	Region 1: I/O ports at 8880 [size=4]
	Region 2: I/O ports at 8800 [size=8]
	Region 3: I/O ports at 8480 [size=4]
	Region 4: I/O ports at 8400 [size=16]
	Region 5: I/O ports at 8080 [size=16]
	Capabilities: [70] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [b0] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ata_piix
	Kernel modules: ata_generic, ata_piix

00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
	Subsystem: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at f8efe000 (64-bit, non-prefetchable) [size=256]
	Region 4: I/O ports at 0400 [size=32]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c-i801

00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2 (prog-if 85 [Master SecO PriO])
	Subsystem: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 19
	Region 0: I/O ports at 9c00 [size=8]
	Region 1: I/O ports at 9880 [size=4]
	Region 2: I/O ports at 9800 [size=8]
	Region 3: I/O ports at 9480 [size=4]
	Region 4: I/O ports at 9400 [size=16]
	Region 5: I/O ports at 9080 [size=16]
	Capabilities: [70] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [b0] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ata_piix
	Kernel modules: ata_generic, ata_piix

01:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10) (prog-if 00 [VGA controller])
	Subsystem: ASPEED Technology, Inc. ASPEED Graphics Family
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at f9000000 (32-bit, non-prefetchable) [size=8M]
	Region 1: Memory at f8fe0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at a880 [size=128]
	Expansion ROM at <unassigned> [disabled]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

01:05.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0) (prog-if 10 [OHCI])
	Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64 (8000ns max), Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 21
	Region 0: Memory at f9bfe000 (32-bit, non-prefetchable) [size=2K]
	Region 1: I/O ports at ac00 [size=128]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA PME(D0-,D1-,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: firewire_ohci
	Kernel modules: firewire-ohci

02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Intel Corporation Device 0000
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 17
	Region 0: Memory at f9c00000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at bc00 [size=32]
	Region 3: Memory at f9c20000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 00000000fee00c98  Data: 0000
	Capabilities: [e0] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: pciback
	Kernel modules: e1000e

03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Intel Corporation Device 0000
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at f9de0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at cc00 [size=32]
	Region 3: Memory at f9ddc000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 00000000fee00cb8  Data: 0000
	Capabilities: [e0] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: pciback
	Kernel modules: e1000e

05:00.0 PCI bridge: Intel Corporation 80333 Segment-A PCI Express-to-PCI Express Bridge (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=05, secondary=07, subordinate=07, sec-latency=64
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: f9e00000-f9efffff
	Prefetchable memory behind bridge: 00000000f7c00000-00000000f7ffffff
	Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [44] Express (v1) PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [5c] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [6c] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [d8] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=133MHz
		Status: Dev=05:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
		Upstream: Capacity=65535 CommitmentLimit=65535
		Downstream: Capacity=65535 CommitmentLimit=65535
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [300 v1] Power Budgeting <?>
	Kernel modules: shpchp

05:00.2 PCI bridge: Intel Corporation 80333 Segment-B PCI Express-to-PCI Express Bridge (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=05, secondary=06, subordinate=06, sec-latency=64
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [44] Express (v1) PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [5c] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [6c] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [d8] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=133MHz
		Status: Dev=05:00.2 64bit- 133MHz- SCD- USC- SCO- SRD-
		Upstream: Capacity=65535 CommitmentLimit=65535
		Downstream: Capacity=65535 CommitmentLimit=65535
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [300 v1] Power Budgeting <?>
	Kernel modules: shpchp

07:0e.0 RAID bus controller: Areca Technology Corp. ARC-1220 8-Port PCI-Express to SATA RAID Controller
	Subsystem: Areca Technology Corp. ARC-1220 8-Port PCI-Express to SATA RAID Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping+ SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64 (32000ns min), Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 39
	Region 0: Memory at f9eff000 (32-bit, non-prefetchable) [size=4K]
	Region 2: Memory at f7c00000 (32-bit, prefetchable) [size=4M]
	Expansion ROM at f9ee0000 [disabled] [size=64K]
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [d0] MSI: Enable- Count=1/2 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [e0] PCI-X non-bridge device
		Command: DPERE+ ERO- RBC=1024 OST=8
		Status: Dev=07:0e.0 64bit+ 133MHz+ SCD- USC- DC=bridge DMMRBC=1024 DMOST=4 DMCRS=32 RSCEM- 266MHz- 533MHz-
	Kernel driver in use: arcmsr
	Kernel modules: arcmsr

08:00.0 PCI bridge: Integrated Device Technology, Inc. PES12N3A PCI Express Switch (rev 0e) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=08, secondary=09, subordinate=0d, sec-latency=0
	I/O behind bridge: 0000d000-0000efff
	Memory behind bridge: f9f00000-fbefffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Upstream Port, MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-SlotPowerLimit 25.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [c0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 05, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [200 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=4
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=02 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32+ WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
			Port Arbitration Table <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

09:02.0 PCI bridge: Integrated Device Technology, Inc. PES12N3A PCI Express Switch (rev 0e) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=09, secondary=0c, subordinate=0d, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: faf00000-fbefffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Downstream Port (Slot-), MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
	Capabilities: [c0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee006d8  Data: 0000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 05, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [200 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Kernel driver in use: pcieport
	Kernel modules: shpchp

09:04.0 PCI bridge: Integrated Device Technology, Inc. PES12N3A PCI Express Switch (rev 0e) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=09, secondary=0a, subordinate=0b, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: f9f00000-faefffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Downstream Port (Slot-), MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #4, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
	Capabilities: [c0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee006f8  Data: 0000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 05, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [200 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Kernel driver in use: pcieport
	Kernel modules: shpchp

0a:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Quad Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 26
	Region 0: Memory at fa780000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fa000000 (32-bit, non-prefetchable) [size=4M]
	Region 2: I/O ports at d880 [size=32]
	Region 3: Memory at f9ffc000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #4, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
		DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-5d-54-44
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 8, Function Dependency Link: 00
		VF offset: 384, stride: 2, Device ID: 10ca
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000f9f00000 (64-bit, non-prefetchable)
		Region 3: Memory at 00000000f9f20000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: igb
	Kernel modules: igb

0a:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Quad Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin B routed to IRQ 25
	Region 0: Memory at faee0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fa800000 (32-bit, non-prefetchable) [size=4M]
	Region 2: I/O ports at dc00 [size=32]
	Region 3: Memory at fa7fc000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #4, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
		DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-5d-54-44
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 8, Function Dependency Link: 01
		VF offset: 384, stride: 2, Device ID: 10ca
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000f9f40000 (64-bit, non-prefetchable)
		Region 3: Memory at 00000000f9f60000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: igb
	Kernel modules: igb

0c:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Quad Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 27
	Region 0: Memory at fb780000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fb000000 (32-bit, non-prefetchable) [size=4M]
	Region 2: I/O ports at e880 [size=32]
	Region 3: Memory at faffc000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
		DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-5d-54-40
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 8, Function Dependency Link: 00
		VF offset: 384, stride: 2, Device ID: 10ca
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000faf00000 (64-bit, non-prefetchable)
		Region 3: Memory at 00000000faf20000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: igb
	Kernel modules: igb

0c:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Quad Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin B routed to IRQ 29
	Region 0: Memory at fbee0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fb800000 (32-bit, non-prefetchable) [size=4M]
	Region 2: I/O ports at ec00 [size=32]
	Region 3: Memory at fb7fc000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
		DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-5d-54-40
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 8, Function Dependency Link: 01
		VF offset: 384, stride: 2, Device ID: 10ca
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000faf40000 (64-bit, non-prefetchable)
		Region 3: Memory at 00000000faf60000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: igb
	Kernel modules: igb


[-- Attachment #3: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: pci-passthrough in pvops causing offline raid

[Mark Adams]

On Thu, Nov 11, 2010 at 06:13:29PM +0000, Mark Adams wrote:
> On Thu, Nov 11, 2010 at 12:58:09PM -0500, Konrad Rzeszutek Wilk wrote:
> > On Thu, Nov 11, 2010 at 05:38:50PM +0000, Mark Adams wrote:
> > > On Thu, Nov 11, 2010 at 11:53:40AM -0500, Konrad Rzeszutek Wilk wrote:
> > > > On Thu, Nov 11, 2010 at 10:24:17AM +0000, Mark Adams wrote:
> > > > > Hi All,
> > > > > 
> > > > > Running xen 4.0.1-rc6, debian squeeze 2.6.32-21.
> > > > > 
> > > > > In a voip setup, where I have forwarded the onboard NIC interfaces
> > > > > through to domU using the following grub config:
> > > > > 
> > > > > module  /vmlinuz-2.6.32-5-xen-amd64 placeholder root=UUID=25c3ac79-6850-498d-afcf-ea42970e94fd ro  quiet xen-pciback.permissive xen-pciback.hide=(02:00.0)(03:00.0) pci=resource_alignment=02:00.0;03:00.0
> > > > > 
> > > > > I'm having a serious issue where the raid card goes offline after an
> > > > > indefinate period of time. Sometimes runs fine for a week, other times 1
> > > > > day before I get "offline device" errors. Rebooting the machine fixes it
> > > > > straight away, and everything is back online.
> > > > > 
> > > > > What in the Xen pciback is causing the raid card to go offline? The
> > > > > only devices hidden are the 2 onboard NIC's.
> > > > 
> > > > You need to give more details. Is the RAID card a 3Ware? An LSI? Do you
> > > > run with an IOMMU? When the RAID card goes offline, do you see a stop of
> > > > IRQs going to the device? Are the IRQs for the RAID card sent to all of your
> > > > CPUs or just a specific one? Are you pinning your guests to specific CPUs?
> > > > Does the issue disappear if you don't passthrough the NIC interfaces? If so have
> > > > you run this setup for "a week" to make sure?
> > > 
> > > It is an Areca 1220. I can't see anything when the device goes offline
> > > apart from 
> > > 
> > >     [77324.264270] sd 0:0:0:1: rejecting I/O to offline device
> > >     [77334.005854] sd 0:0:0:0: rejecting I/O to offline device
> > 
> > That is it? No other details from the driver? Did you poke at the driver (modinfo)
> > to see if there are any options to increase its verbosity.
> 
> I can't do anything once its happened, everything is offline so I have
> no utils...
> > 
> > > 
> > > Unfortunately nothing get's logged because there is nothing to write to
> > > anymore. I'm not sure how I can see the IRQs otherwise. There is no
> > 
> > cat /proc/interrupts
> > 
> > > pinning being done at all, and the machine was running for a few months
> > > OK before the pciback was added.
> > 
> > Ok, what about your NICs? Are they on-board? Are they sharing the IRQ
> > with the card? You should be able to see this by looking at /proc/interrupts.
> > Which NICs are they? lspci can you help you there. As of matter of fact, run
> > lspci -vvv and send that.
> 
> It is the onboard nics, they are Intel 82574L. I can see the arcmsr
> line, but not anything for the NICS (because they are hidden?)
> 
> 39:    1126249          0          0          0          0          0         0          0  xen-pirq-ioapic-level  arcmsr
> 
> Nothing else is on 1126249
> 
> see lspci.txt attached.
> 

I've just noticed this at the end of xm dmesg

(XEN) msi.c:715: MSI is already in use on device 02:00.0
(XEN) msi.c:715: MSI is already in use on device 02:00.0
(XEN) msi.c:715: MSI is already in use on device 02:00.0

Something else trying to use the device being exported? (the nics are
02:00.0 and 03:00.0)

Re: pci-passthrough in pvops causing offline raid

[Konrad Rzeszutek Wilk]

> I've just noticed this at the end of xm dmesg
> 
> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> 
> Something else trying to use the device being exported? (the nics are
> 02:00.0 and 03:00.0)

Hmm, looks like it, but it should not have happend. Can you attach
the output of 'xm dmesg' and also do the 'xm debug-keys ..' that I
asked for in the previous e-mail?

Jan, the fixes for the MSI you did, they weren't for 4.0.1 right? Just
for unstable?

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Mark Adams]

[-- Attachment #1: Type: text/plain, Size: 846 bytes --]

See attached

On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> > I've just noticed this at the end of xm dmesg
> > 
> > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > 
> > Something else trying to use the device being exported? (the nics are
> > 02:00.0 and 03:00.0)
> 
> Hmm, looks like it, but it should not have happend. Can you attach
> the output of 'xm dmesg' and also do the 'xm debug-keys ..' that I
> asked for in the previous e-mail?
> 
> Jan, the fixes for the MSI you did, they weren't for 4.0.1 right? Just
> for unstable?
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users


[-- Attachment #2: dmesg_debugi.txt --]
[-- Type: text/plain, Size: 16384 bytes --]

ype=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   8 affinity:00000000,00000000,00000000,00000001 vec:60 type=IO-APIC-edge    status=00000010 in-flight=0 domain-list=0:  8(----),
(XEN)    IRQ:   9 affinity:00000000,00000000,00000000,00000001 vec:68 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0:  9(----),
(XEN)    IRQ:  10 affinity:00000000,00000000,00000000,00000001 vec:70 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  11 affinity:00000000,00000000,00000000,00000001 vec:78 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  12 affinity:00000000,00000000,00000000,00000001 vec:88 type=IO-APIC-edge    status=00000010 in-flight=0 domain-list=0: 12(----),
(XEN)    IRQ:  13 affinity:00000000,00000000,00000000,000000ff vec:90 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  14 affinity:00000000,00000000,00000000,00000001 vec:98 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  15 affinity:00000000,00000000,00000000,00000001 vec:a0 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  16 affinity:00000000,00000000,00000000,00000001 vec:b0 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 16(----),3: 16(--M-),
(XEN)    IRQ:  17 affinity:00000000,00000000,00000000,00000008 vec:a8 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=3: 17(--M-),
(XEN)    IRQ:  18 affinity:00000000,00000000,00000000,00000001 vec:91 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 18(----),
(XEN)    IRQ:  19 affinity:00000000,00000000,00000000,00000001 vec:a9 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 19(----),
(XEN)    IRQ:  21 affinity:00000000,00000000,00000000,00000001 vec:a1 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 21(----),
(XEN)    IRQ:  22 affinity:00000000,00000000,00000000,00000010 vec:93 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 22(----),
(XEN)    IRQ:  23 affinity:00000000,00000000,00000000,00000001 vec:b1 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 23(----),
(XEN)    IRQ:  25 affinity:00000000,00000000,00000000,000000ff vec:a2 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  26 affinity:00000000,00000000,00000000,000000ff vec:4a type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  27 affinity:00000000,00000000,00000000,000000ff vec:41 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  29 affinity:00000000,00000000,00000000,000000ff vec:b9 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  39 affinity:00000000,00000000,00000000,00000001 vec:99 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 39(----),
(XEN)    IRQ:  43 affinity:00000000,00000000,00000000,000000ff vec:33 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  44 affinity:00000000,00000000,00000000,000000ff vec:43 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  45 affinity:00000000,00000000,00000000,000000ff vec:53 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  46 affinity:00000000,00000000,00000000,000000ff vec:63 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  48 affinity:00000000,00000000,00000000,00000001 vec:28 type=DMA_MSI         status=00000000 mapped, unbound
(XEN)    IRQ:  49 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b8 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  50 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c0 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  51 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c8 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  52 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d0 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  53 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d8 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  54 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:21 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  55 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:29 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  56 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:31 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  57 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:39 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  58 affinity:00000000,00000000,00000000,00000002 vec:49 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:294(----),
(XEN)    IRQ:  59 affinity:00000000,00000000,00000000,00000002 vec:51 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:293(----),
(XEN)    IRQ:  60 affinity:00000000,00000000,00000000,00000002 vec:59 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:292(----),
(XEN)    IRQ:  61 affinity:00000000,00000000,00000000,00000002 vec:61 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:291(----),
(XEN)    IRQ:  62 affinity:00000000,00000000,00000000,00000002 vec:69 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:290(----),
(XEN)    IRQ:  63 affinity:00000000,00000000,00000000,00000002 vec:71 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:289(----),
(XEN)    IRQ:  64 affinity:00000000,00000000,00000000,00000002 vec:79 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:288(----),
(XEN)    IRQ:  65 affinity:00000000,00000000,00000000,00000002 vec:81 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:287(----),
(XEN)    IRQ:  66 affinity:00000000,00000000,00000000,00000002 vec:89 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:286(----),
(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c1 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c9 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d1 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  70 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d9 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  71 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:22 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  72 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  73 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:32 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  74 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  75 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  76 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  77 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:5a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  78 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:62 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  79 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:6a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  80 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:72 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  81 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:7a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  82 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:8a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  83 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:92 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  84 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:9a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  85 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:aa type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  86 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b2 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  87 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ba type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  88 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c2 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  89 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ca type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  90 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d2 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  91 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:da type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  92 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:23 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  93 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2b type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  94 affinity:00000000,00000000,00000000,00000008 vec:3b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:258(----),
(XEN)    IRQ:  95 affinity:00000000,00000000,00000000,00000008 vec:4b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:257(----),
(XEN)    IRQ:  96 affinity:00000000,00000000,00000000,00000008 vec:5b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:256(----),
(XEN)    IRQ:  97 affinity:00000000,00000000,00000000,00000008 vec:6b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:255(----),
(XEN)    IRQ:  98 affinity:00000000,00000000,00000000,00000008 vec:73 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:254(----),
(XEN)    IRQ:  99 affinity:00000000,00000000,00000000,00000008 vec:7b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:253(----),
(XEN)    IRQ: 100 affinity:00000000,00000000,00000000,00000008 vec:83 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:252(----),
(XEN)    IRQ: 101 affinity:00000000,00000000,00000000,00000008 vec:8b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:251(----),
(XEN)    IRQ: 102 affinity:00000000,00000000,00000000,00000001 vec:9b type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ: 103 affinity:00000000,00000000,00000000,00000001 vec:a3 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ: 104 affinity:00000000,00000000,00000000,00000002 vec:ab type=PCI-MSI         status=00000010 in-flight=0 domain-list=3: 77(--M-),
(XEN)    IRQ: 105 affinity:00000000,00000000,00000000,00000002 vec:b3 type=PCI-MSI         status=00000010 in-flight=0 domain-list=3: 76(--M-),
(XEN)    IRQ: 106 affinity:00000000,00000000,00000000,00000008 vec:bb type=PCI-MSI         status=00000050 in-flight=0 domain-list=3: 75(--M-),
(XEN) IO-APIC interrupt information:
(XEN)     IRQ  0 Vec240:
(XEN)       Apic 0x00, Pin  2: vector=240, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:255
(XEN)     IRQ  1 Vec 48:
(XEN)       Apic 0x00, Pin  1: vector=48, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  3 Vec 56:
(XEN)       Apic 0x00, Pin  3: vector=56, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:255
(XEN)     IRQ  4 Vec 64:
(XEN)       Apic 0x00, Pin  4: vector=64, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:255
(XEN)     IRQ  5 Vec 72:
(XEN)       Apic 0x00, Pin  5: vector=72, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  6 Vec 80:
(XEN)       Apic 0x00, Pin  6: vector=80, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  7 Vec 88:
(XEN)       Apic 0x00, Pin  7: vector=88, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  8 Vec 96:
(XEN)       Apic 0x00, Pin  8: vector=96, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  9 Vec104:
(XEN)       Apic 0x00, Pin  9: vector=104, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 10 Vec112:
(XEN)       Apic 0x00, Pin 10: vector=112, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 11 Vec120:
(XEN)       Apic 0x00, Pin 11: vector=120, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 12 Vec136:
(XEN)       Apic 0x00, Pin 12: vector=136, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 13 Vec144:
(XEN)       Apic 0x00, Pin 13: vector=144, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:255
(XEN)     IRQ 14 Vec152:
(XEN)       Apic 0x00, Pin 14: vector=152, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 15 Vec160:
(XEN)       Apic 0x00, Pin 15: vector=160, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 16 Vec176:
(XEN)       Apic 0x00, Pin 16: vector=176, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 17 Vec168:
(XEN)       Apic 0x00, Pin 17: vector=168, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:8
(XEN)     IRQ 18 Vec145:
(XEN)       Apic 0x00, Pin 18: vector=145, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 19 Vec169:
(XEN)       Apic 0x00, Pin 19: vector=169, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 21 Vec161:
(XEN)       Apic 0x00, Pin 21: vector=161, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 22 Vec147:
(XEN)       Apic 0x00, Pin 22: vector=147, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:16
(XEN)     IRQ 23 Vec177:
(XEN)       Apic 0x00, Pin 23: vector=177, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 25 Vec162:
(XEN)       Apic 0x01, Pin  1: vector=162, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 26 Vec 74:
(XEN)       Apic 0x01, Pin  2: vector=74, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 27 Vec 65:
(XEN)       Apic 0x01, Pin  3: vector=65, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 29 Vec185:
(XEN)       Apic 0x01, Pin  5: vector=185, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 39 Vec153:
(XEN)       Apic 0x01, Pin 15: vector=153, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 43 Vec 51:
(XEN)       Apic 0x01, Pin 19: vector=51, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 44 Vec 67:
(XEN)       Apic 0x01, Pin 20: vector=67, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 45 Vec 83:
(XEN)       Apic 0x01, Pin 21: vector=83, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 46 Vec 99:
(XEN)       Apic 0x01, Pin 22: vector=99, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255

[-- Attachment #3: dmesg_debugQ.txt --]
[-- Type: text/plain, Size: 16384 bytes --]

 mapped, unbound
(XEN)    IRQ:  90 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d2 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  91 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:da type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  92 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:23 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  93 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2b type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  94 affinity:00000000,00000000,00000000,00000008 vec:3b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:258(----),
(XEN)    IRQ:  95 affinity:00000000,00000000,00000000,00000008 vec:4b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:257(----),
(XEN)    IRQ:  96 affinity:00000000,00000000,00000000,00000008 vec:5b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:256(----),
(XEN)    IRQ:  97 affinity:00000000,00000000,00000000,00000008 vec:6b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:255(----),
(XEN)    IRQ:  98 affinity:00000000,00000000,00000000,00000008 vec:73 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:254(----),
(XEN)    IRQ:  99 affinity:00000000,00000000,00000000,00000008 vec:7b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:253(----),
(XEN)    IRQ: 100 affinity:00000000,00000000,00000000,00000008 vec:83 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:252(----),
(XEN)    IRQ: 101 affinity:00000000,00000000,00000000,00000008 vec:8b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:251(----),
(XEN)    IRQ: 102 affinity:00000000,00000000,00000000,00000001 vec:9b type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ: 103 affinity:00000000,00000000,00000000,00000001 vec:a3 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ: 104 affinity:00000000,00000000,00000000,00000004 vec:ab type=PCI-MSI         status=00000010 in-flight=0 domain-list=3: 77(--M-),
(XEN)    IRQ: 105 affinity:00000000,00000000,00000000,00000004 vec:b3 type=PCI-MSI         status=00000010 in-flight=0 domain-list=3: 76(--M-),
(XEN)    IRQ: 106 affinity:00000000,00000000,00000000,00000008 vec:bb type=PCI-MSI         status=00000050 in-flight=0 domain-list=3: 75(--M-),
(XEN) IO-APIC interrupt information:
(XEN)     IRQ  0 Vec240:
(XEN)       Apic 0x00, Pin  2: vector=240, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:255
(XEN)     IRQ  1 Vec 48:
(XEN)       Apic 0x00, Pin  1: vector=48, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  3 Vec 56:
(XEN)       Apic 0x00, Pin  3: vector=56, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:255
(XEN)     IRQ  4 Vec 64:
(XEN)       Apic 0x00, Pin  4: vector=64, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:255
(XEN)     IRQ  5 Vec 72:
(XEN)       Apic 0x00, Pin  5: vector=72, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  6 Vec 80:
(XEN)       Apic 0x00, Pin  6: vector=80, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  7 Vec 88:
(XEN)       Apic 0x00, Pin  7: vector=88, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  8 Vec 96:
(XEN)       Apic 0x00, Pin  8: vector=96, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  9 Vec104:
(XEN)       Apic 0x00, Pin  9: vector=104, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 10 Vec112:
(XEN)       Apic 0x00, Pin 10: vector=112, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 11 Vec120:
(XEN)       Apic 0x00, Pin 11: vector=120, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 12 Vec136:
(XEN)       Apic 0x00, Pin 12: vector=136, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 13 Vec144:
(XEN)       Apic 0x00, Pin 13: vector=144, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:255
(XEN)     IRQ 14 Vec152:
(XEN)       Apic 0x00, Pin 14: vector=152, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 15 Vec160:
(XEN)       Apic 0x00, Pin 15: vector=160, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 16 Vec176:
(XEN)       Apic 0x00, Pin 16: vector=176, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 17 Vec168:
(XEN)       Apic 0x00, Pin 17: vector=168, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:8
(XEN)     IRQ 18 Vec145:
(XEN)       Apic 0x00, Pin 18: vector=145, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 19 Vec169:
(XEN)       Apic 0x00, Pin 19: vector=169, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 21 Vec161:
(XEN)       Apic 0x00, Pin 21: vector=161, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 22 Vec147:
(XEN)       Apic 0x00, Pin 22: vector=147, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:16
(XEN)     IRQ 23 Vec177:
(XEN)       Apic 0x00, Pin 23: vector=177, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 25 Vec162:
(XEN)       Apic 0x01, Pin  1: vector=162, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 26 Vec 74:
(XEN)       Apic 0x01, Pin  2: vector=74, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 27 Vec 65:
(XEN)       Apic 0x01, Pin  3: vector=65, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 29 Vec185:
(XEN)       Apic 0x01, Pin  5: vector=185, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 39 Vec153:
(XEN)       Apic 0x01, Pin 15: vector=153, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 43 Vec 51:
(XEN)       Apic 0x01, Pin 19: vector=51, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 44 Vec 67:
(XEN)       Apic 0x01, Pin 20: vector=67, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 45 Vec 83:
(XEN)       Apic 0x01, Pin 21: vector=83, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 46 Vec 99:
(XEN)       Apic 0x01, Pin 22: vector=99, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #1 registers: 24.
(XEN) number of IO-APIC #3 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #1......
(XEN) .... register #00: 04000000
(XEN) .......    : physical APIC id: 04
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00170020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0020
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 001 01  0    0    0   0   0    1    1    30
(XEN)  02 0FF 0F  0    0    0   0   0    1    1    F0
(XEN)  03 0FF 0F  1    0    0   0   0    1    1    38
(XEN)  04 0FF 0F  1    0    0   0   0    1    1    40
(XEN)  05 001 01  0    0    0   0   0    1    1    48
(XEN)  06 001 01  0    0    0   0   0    1    1    50
(XEN)  07 001 01  0    0    0   0   0    1    1    58
(XEN)  08 001 01  0    0    0   0   0    1    1    60
(XEN)  09 001 01  0    1    0   0   0    1    1    68
(XEN)  0a 001 01  0    0    0   0   0    1    1    70
(XEN)  0b 001 01  0    0    0   0   0    1    1    78
(XEN)  0c 001 01  0    0    0   0   0    1    1    88
(XEN)  0d 0FF 0F  1    0    0   0   0    1    1    90
(XEN)  0e 001 01  0    0    0   0   0    1    1    98
(XEN)  0f 001 01  0    0    0   0   0    1    1    A0
(XEN)  10 001 01  0    1    0   1   0    1    1    B0
(XEN)  11 008 08  0    1    0   1   0    1    1    A8
(XEN)  12 001 01  0    1    0   1   0    1    1    91
(XEN)  13 001 01  0    1    0   1   0    1    1    A9
(XEN)  14 04B 0B  1    0    0   0   0    0    2    05
(XEN)  15 001 01  0    1    0   1   0    1    1    A1
(XEN)  16 010 00  0    1    0   1   0    1    1    93
(XEN)  17 001 01  0    1    0   1   0    1    1    B1
(XEN) IO APIC #3......
(XEN) .... register #00: 06000000
(XEN) .......    : physical APIC id: 06
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00170020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 0FF 0F  1    1    0   1   0    1    1    A2
(XEN)  02 0FF 0F  1    1    0   1   0    1    1    4A
(XEN)  03 0FF 0F  1    1    0   1   0    1    1    41
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 0FF 0F  1    1    0   1   0    1    1    B9
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 001 01  0    1    0   1   0    1    1    99
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 0FF 0F  1    1    0   1   0    1    1    33
(XEN)  14 0FF 0F  1    1    0   1   0    1    1    43
(XEN)  15 0FF 0F  1    1    0   1   0    1    1    53
(XEN)  16 0FF 0F  1    1    0   1   0    1    1    63
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ48 -> 0:1
(XEN) IRQ56 -> 0:3
(XEN) IRQ64 -> 0:4
(XEN) IRQ72 -> 0:5
(XEN) IRQ80 -> 0:6
(XEN) IRQ88 -> 0:7
(XEN) IRQ96 -> 0:8
(XEN) IRQ104 -> 0:9
(XEN) IRQ112 -> 0:10
(XEN) IRQ120 -> 0:11
(XEN) IRQ136 -> 0:12
(XEN) IRQ144 -> 0:13
(XEN) IRQ152 -> 0:14
(XEN) IRQ160 -> 0:15
(XEN) IRQ176 -> 0:16
(XEN) IRQ168 -> 0:17
(XEN) IRQ145 -> 0:18
(XEN) IRQ169 -> 0:19
(XEN) IRQ161 -> 0:21
(XEN) IRQ147 -> 0:22
(XEN) IRQ177 -> 0:23
(XEN) IRQ162 -> 1:1
(XEN) IRQ74 -> 1:2
(XEN) IRQ65 -> 1:3
(XEN) IRQ185 -> 1:5
(XEN) IRQ153 -> 1:15
(XEN) IRQ51 -> 1:19
(XEN) IRQ67 -> 1:20
(XEN) IRQ83 -> 1:21
(XEN) IRQ99 -> 1:22
(XEN) .................................... done.
(XEN) ==== PCI devices ====
(XEN) ff:06.3 - dom 0   - MSIs < >
(XEN) ff:06.2 - dom 0   - MSIs < >
(XEN) ff:06.1 - dom 0   - MSIs < >
(XEN) ff:06.0 - dom 0   - MSIs < >
(XEN) ff:05.3 - dom 0   - MSIs < >
(XEN) ff:05.2 - dom 0   - MSIs < >
(XEN) ff:05.1 - dom 0   - MSIs < >
(XEN) ff:05.0 - dom 0   - MSIs < >
(XEN) ff:04.3 - dom 0   - MSIs < >
(XEN) ff:04.2 - dom 0   - MSIs < >
(XEN) ff:04.1 - dom 0   - MSIs < >
(XEN) ff:04.0 - dom 0   - MSIs < >
(XEN) ff:03.4 - dom 0   - MSIs < >
(XEN) ff:03.2 - dom 0   - MSIs < >
(XEN) ff:03.1 - dom 0   - MSIs < >
(XEN) ff:03.0 - dom 0   - MSIs < >
(XEN) ff:02.5 - dom 0   - MSIs < >
(XEN) ff:02.4 - dom 0   - MSIs < >
(XEN) ff:02.3 - dom 0   - MSIs < >
(XEN) ff:02.2 - dom 0   - MSIs < >
(XEN) ff:02.1 - dom 0   - MSIs < >
(XEN) ff:02.0 - dom 0   - MSIs < >
(XEN) ff:00.1 - dom 0   - MSIs < >
(XEN) ff:00.0 - dom 0   - MSIs < >
(XEN) fe:06.3 - dom 0   - MSIs < >
(XEN) fe:06.2 - dom 0   - MSIs < >
(XEN) fe:06.1 - dom 0   - MSIs < >
(XEN) fe:06.0 - dom 0   - MSIs < >
(XEN) fe:05.3 - dom 0   - MSIs < >
(XEN) fe:05.2 - dom 0   - MSIs < >
(XEN) fe:05.1 - dom 0   - MSIs < >
(XEN) fe:05.0 - dom 0   - MSIs < >
(XEN) fe:04.3 - dom 0   - MSIs < >
(XEN) fe:04.2 - dom 0   - MSIs < >
(XEN) fe:04.1 - dom 0   - MSIs < >
(XEN) fe:04.0 - dom 0   - MSIs < >
(XEN) fe:03.4 - dom 0   - MSIs < >
(XEN) fe:03.2 - dom 0   - MSIs < >
(XEN) fe:03.1 - dom 0   - MSIs < >
(XEN) fe:03.0 - dom 0   - MSIs < >
(XEN) fe:02.5 - dom 0   - MSIs < >
(XEN) fe:02.4 - dom 0   - MSIs < >
(XEN) fe:02.3 - dom 0   - MSIs < >
(XEN) fe:02.2 - dom 0   - MSIs < >
(XEN) fe:02.1 - dom 0   - MSIs < >
(XEN) fe:02.0 - dom 0   - MSIs < >
(XEN) fe:00.1 - dom 0   - MSIs < >
(XEN) fe:00.0 - dom 0   - MSIs < >
(XEN) 0c:00.1 - dom 0   - MSIs < 67 68 69 70 71 72 73 74 75 >
(XEN) 0c:00.0 - dom 0   - MSIs < 58 59 60 61 62 63 64 65 66 >
(XEN) 0a:00.1 - dom 0   - MSIs < 85 86 87 88 89 90 91 92 93 >
(XEN) 0a:00.0 - dom 0   - MSIs < 76 77 78 79 80 81 82 83 84 >
(XEN) 09:04.0 - dom 0   - MSIs < 57 >
(XEN) 09:02.0 - dom 0   - MSIs < 56 >
(XEN) 08:00.0 - dom 0   - MSIs < >
(XEN) 07:0e.0 - dom 0   - MSIs < >
(XEN) 05:00.2 - dom 0   - MSIs < >
(XEN) 05:00.0 - dom 0   - MSIs < >
(XEN) 03:00.0 - dom 3   - MSIs < 103 >
(XEN) 02:00.0 - dom 3   - MSIs < 102 104 105 106 >
(XEN) 01:05.7 - dom 0   - MSIs < >
(XEN) 01:05.6 - dom 0   - MSIs < >
(XEN) 01:05.5 - dom 0   - MSIs < >
(XEN) 01:05.4 - dom 0   - MSIs < >
(XEN) 01:05.3 - dom 0   - MSIs < >
(XEN) 01:05.2 - dom 0   - MSIs < >
(XEN) 01:05.1 - dom 0   - MSIs < >
(XEN) 01:05.0 - dom 0   - MSIs < >
(XEN) 01:00.7 - dom 0   - MSIs < >
(XEN) 01:00.6 - dom 0   - MSIs < >
(XEN) 01:00.5 - dom 0   - MSIs < >
(XEN) 01:00.4 - dom 0   - MSIs < >
(XEN) 01:00.3 - dom 0   - MSIs < >
(XEN) 01:00.2 - dom 0   - MSIs < >
(XEN) 01:00.1 - dom 0   - MSIs < >
(XEN) 01:00.0 - dom 0   - MSIs < >
(XEN) 00:1f.5 - dom 0   - MSIs < >
(XEN) 00:1f.3 - dom 0   - MSIs < >
(XEN) 00:1f.2 - dom 0   - MSIs < >
(XEN) 00:1f.0 - dom 0   - MSIs < >
(XEN) 00:1e.0 - dom 0   - MSIs < >
(XEN) 00:1d.7 - dom 0   - MSIs < >
(XEN) 00:1d.2 - dom 0   - MSIs < >
(XEN) 00:1d.1 - dom 0   - MSIs < >
(XEN) 00:1d.0 - dom 0   - MSIs < >
(XEN) 00:1c.5 - dom 0   - MSIs < 55 >
(XEN) 00:1c.4 - dom 0   - MSIs < 54 >
(XEN) 00:1c.0 - dom 0   - MSIs < 53 >
(XEN) 00:1b.0 - dom 0   - MSIs < >
(XEN) 00:1a.7 - dom 0   - MSIs < >
(XEN) 00:1a.2 - dom 0   - MSIs < >
(XEN) 00:1a.1 - dom 0   - MSIs < >
(XEN) 00:1a.0 - dom 0   - MSIs < >
(XEN) 00:16.7 - dom 0   - MSIs < 101 >
(XEN) 00:16.6 - dom 0   - MSIs < 100 >
(XEN) 00:16.5 - dom 0   - MSIs < 99 >
(XEN) 00:16.4 - dom 0   - MSIs < 98 >
(XEN) 00:16.3 - dom 0   - MSIs < 97 >
(XEN) 00:16.2 - dom 0   - MSIs < 96 >
(XEN) 00:16.1 - dom 0   - MSIs < 95 >
(XEN) 00:16.0 - dom 0   - MSIs < 94 >
(XEN) 00:14.3 - dom 0   - MSIs < >
(XEN) 00:14.2 - dom 0   - MSIs < >
(XEN) 00:14.1 - dom 0   - MSIs < >
(XEN) 00:14.0 - dom 0   - MSIs < >
(XEN) 00:13.0 - dom 0   - MSIs < >
(XEN) 00:07.0 - dom 0   - MSIs < 52 >
(XEN) 00:05.0 - dom 0   - MSIs < 51 >
(XEN) 00:03.0 - dom 0   - MSIs < 50 >
(XEN) 00:01.0 - dom 0   - MSIs < 49 >
(XEN) 00:00.0 - dom 0   - MSIs < >

[-- Attachment #4: dmesg_debugz.txt --]
[-- Type: text/plain, Size: 16384 bytes --]

 58 affinity:00000000,00000000,00000000,00000002 vec:49 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:294(----),
(XEN)    IRQ:  59 affinity:00000000,00000000,00000000,00000002 vec:51 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:293(----),
(XEN)    IRQ:  60 affinity:00000000,00000000,00000000,00000002 vec:59 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:292(----),
(XEN)    IRQ:  61 affinity:00000000,00000000,00000000,00000002 vec:61 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:291(----),
(XEN)    IRQ:  62 affinity:00000000,00000000,00000000,00000002 vec:69 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:290(----),
(XEN)    IRQ:  63 affinity:00000000,00000000,00000000,00000002 vec:71 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:289(----),
(XEN)    IRQ:  64 affinity:00000000,00000000,00000000,00000002 vec:79 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:288(----),
(XEN)    IRQ:  65 affinity:00000000,00000000,00000000,00000002 vec:81 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:287(----),
(XEN)    IRQ:  66 affinity:00000000,00000000,00000000,00000002 vec:89 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:286(----),
(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c1 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c9 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d1 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  70 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d9 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  71 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:22 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  72 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  73 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:32 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  74 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  75 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  76 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  77 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:5a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  78 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:62 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  79 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:6a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  80 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:72 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  81 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:7a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  82 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:8a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  83 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:92 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  84 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:9a type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  85 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:aa type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  86 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b2 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  87 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ba type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  88 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c2 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  89 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ca type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  90 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d2 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  91 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:da type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  92 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:23 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  93 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2b type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  94 affinity:00000000,00000000,00000000,00000008 vec:3b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:258(----),
(XEN)    IRQ:  95 affinity:00000000,00000000,00000000,00000008 vec:4b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:257(----),
(XEN)    IRQ:  96 affinity:00000000,00000000,00000000,00000008 vec:5b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:256(----),
(XEN)    IRQ:  97 affinity:00000000,00000000,00000000,00000008 vec:6b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:255(----),
(XEN)    IRQ:  98 affinity:00000000,00000000,00000000,00000008 vec:73 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:254(----),
(XEN)    IRQ:  99 affinity:00000000,00000000,00000000,00000008 vec:7b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:253(----),
(XEN)    IRQ: 100 affinity:00000000,00000000,00000000,00000008 vec:83 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:252(----),
(XEN)    IRQ: 101 affinity:00000000,00000000,00000000,00000008 vec:8b type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:251(----),
(XEN)    IRQ: 102 affinity:00000000,00000000,00000000,00000001 vec:9b type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ: 103 affinity:00000000,00000000,00000000,00000001 vec:a3 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ: 104 affinity:00000000,00000000,00000000,00000004 vec:ab type=PCI-MSI         status=00000010 in-flight=0 domain-list=3: 77(--M-),
(XEN)    IRQ: 105 affinity:00000000,00000000,00000000,00000004 vec:b3 type=PCI-MSI         status=00000010 in-flight=0 domain-list=3: 76(--M-),
(XEN)    IRQ: 106 affinity:00000000,00000000,00000000,00000008 vec:bb type=PCI-MSI         status=00000050 in-flight=0 domain-list=3: 75(--M-),
(XEN) IO-APIC interrupt information:
(XEN)     IRQ  0 Vec240:
(XEN)       Apic 0x00, Pin  2: vector=240, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:255
(XEN)     IRQ  1 Vec 48:
(XEN)       Apic 0x00, Pin  1: vector=48, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  3 Vec 56:
(XEN)       Apic 0x00, Pin  3: vector=56, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:255
(XEN)     IRQ  4 Vec 64:
(XEN)       Apic 0x00, Pin  4: vector=64, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:255
(XEN)     IRQ  5 Vec 72:
(XEN)       Apic 0x00, Pin  5: vector=72, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  6 Vec 80:
(XEN)       Apic 0x00, Pin  6: vector=80, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  7 Vec 88:
(XEN)       Apic 0x00, Pin  7: vector=88, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  8 Vec 96:
(XEN)       Apic 0x00, Pin  8: vector=96, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  9 Vec104:
(XEN)       Apic 0x00, Pin  9: vector=104, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 10 Vec112:
(XEN)       Apic 0x00, Pin 10: vector=112, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 11 Vec120:
(XEN)       Apic 0x00, Pin 11: vector=120, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 12 Vec136:
(XEN)       Apic 0x00, Pin 12: vector=136, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 13 Vec144:
(XEN)       Apic 0x00, Pin 13: vector=144, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:255
(XEN)     IRQ 14 Vec152:
(XEN)       Apic 0x00, Pin 14: vector=152, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 15 Vec160:
(XEN)       Apic 0x00, Pin 15: vector=160, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 16 Vec176:
(XEN)       Apic 0x00, Pin 16: vector=176, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 17 Vec168:
(XEN)       Apic 0x00, Pin 17: vector=168, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:8
(XEN)     IRQ 18 Vec145:
(XEN)       Apic 0x00, Pin 18: vector=145, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 19 Vec169:
(XEN)       Apic 0x00, Pin 19: vector=169, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 21 Vec161:
(XEN)       Apic 0x00, Pin 21: vector=161, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 22 Vec147:
(XEN)       Apic 0x00, Pin 22: vector=147, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:16
(XEN)     IRQ 23 Vec177:
(XEN)       Apic 0x00, Pin 23: vector=177, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 25 Vec162:
(XEN)       Apic 0x01, Pin  1: vector=162, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 26 Vec 74:
(XEN)       Apic 0x01, Pin  2: vector=74, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 27 Vec 65:
(XEN)       Apic 0x01, Pin  3: vector=65, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 29 Vec185:
(XEN)       Apic 0x01, Pin  5: vector=185, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 39 Vec153:
(XEN)       Apic 0x01, Pin 15: vector=153, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 43 Vec 51:
(XEN)       Apic 0x01, Pin 19: vector=51, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 44 Vec 67:
(XEN)       Apic 0x01, Pin 20: vector=67, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 45 Vec 83:
(XEN)       Apic 0x01, Pin 21: vector=83, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN)     IRQ 46 Vec 99:
(XEN)       Apic 0x01, Pin 22: vector=99, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:255
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #1 registers: 24.
(XEN) number of IO-APIC #3 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #1......
(XEN) .... register #00: 04000000
(XEN) .......    : physical APIC id: 04
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00170020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0020
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 001 01  0    0    0   0   0    1    1    30
(XEN)  02 0FF 0F  0    0    0   0   0    1    1    F0
(XEN)  03 0FF 0F  1    0    0   0   0    1    1    38
(XEN)  04 0FF 0F  1    0    0   0   0    1    1    40
(XEN)  05 001 01  0    0    0   0   0    1    1    48
(XEN)  06 001 01  0    0    0   0   0    1    1    50
(XEN)  07 001 01  0    0    0   0   0    1    1    58
(XEN)  08 001 01  0    0    0   0   0    1    1    60
(XEN)  09 001 01  0    1    0   0   0    1    1    68
(XEN)  0a 001 01  0    0    0   0   0    1    1    70
(XEN)  0b 001 01  0    0    0   0   0    1    1    78
(XEN)  0c 001 01  0    0    0   0   0    1    1    88
(XEN)  0d 0FF 0F  1    0    0   0   0    1    1    90
(XEN)  0e 001 01  0    0    0   0   0    1    1    98
(XEN)  0f 001 01  0    0    0   0   0    1    1    A0
(XEN)  10 001 01  0    1    0   1   0    1    1    B0
(XEN)  11 008 08  0    1    0   1   0    1    1    A8
(XEN)  12 001 01  0    1    0   1   0    1    1    91
(XEN)  13 001 01  0    1    0   1   0    1    1    A9
(XEN)  14 04B 0B  1    0    0   0   0    0    2    05
(XEN)  15 001 01  0    1    0   1   0    1    1    A1
(XEN)  16 010 00  0    1    0   1   0    1    1    93
(XEN)  17 001 01  0    1    0   1   0    1    1    B1
(XEN) IO APIC #3......
(XEN) .... register #00: 06000000
(XEN) .......    : physical APIC id: 06
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00170020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 0FF 0F  1    1    0   1   0    1    1    A2
(XEN)  02 0FF 0F  1    1    0   1   0    1    1    4A
(XEN)  03 0FF 0F  1    1    0   1   0    1    1    41
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 0FF 0F  1    1    0   1   0    1    1    B9
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 001 01  0    1    0   1   0    1    1    99
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 0FF 0F  1    1    0   1   0    1    1    33
(XEN)  14 0FF 0F  1    1    0   1   0    1    1    43
(XEN)  15 0FF 0F  1    1    0   1   0    1    1    53
(XEN)  16 0FF 0F  1    1    0   1   0    1    1    63
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ48 -> 0:1
(XEN) IRQ56 -> 0:3
(XEN) IRQ64 -> 0:4
(XEN) IRQ72 -> 0:5
(XEN) IRQ80 -> 0:6
(XEN) IRQ88 -> 0:7
(XEN) IRQ96 -> 0:8
(XEN) IRQ104 -> 0:9
(XEN) IRQ112 -> 0:10
(XEN) IRQ120 -> 0:11
(XEN) IRQ136 -> 0:12
(XEN) IRQ144 -> 0:13
(XEN) IRQ152 -> 0:14
(XEN) IRQ160 -> 0:15
(XEN) IRQ176 -> 0:16
(XEN) IRQ168 -> 0:17
(XEN) IRQ145 -> 0:18
(XEN) IRQ169 -> 0:19
(XEN) IRQ161 -> 0:21
(XEN) IRQ147 -> 0:22
(XEN) IRQ177 -> 0:23
(XEN) IRQ162 -> 1:1
(XEN) IRQ74 -> 1:2
(XEN) IRQ65 -> 1:3
(XEN) IRQ185 -> 1:5
(XEN) IRQ153 -> 1:15
(XEN) IRQ51 -> 1:19
(XEN) IRQ67 -> 1:20
(XEN) IRQ83 -> 1:21
(XEN) IRQ99 -> 1:22
(XEN) .................................... done.

[-- Attachment #5: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Mark Adams]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

Apols - Also see plain dmesg attached. This one from updated machine
(4.0.1) still showing the msi issues.

On Thu, Nov 11, 2010 at 07:22:56PM +0000, Mark Adams wrote:
> See attached
> 
> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> > > I've just noticed this at the end of xm dmesg
> > > 
> > > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > 
> > > Something else trying to use the device being exported? (the nics are
> > > 02:00.0 and 03:00.0)
> > 
> > Hmm, looks like it, but it should not have happend. Can you attach
> > the output of 'xm dmesg' and also do the 'xm debug-keys ..' that I
> > asked for in the previous e-mail?
> > 
> > Jan, the fixes for the MSI you did, they weren't for 4.0.1 right? Just
> > for unstable?
> > 
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users
> 


[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 5388 bytes --]

(XEN) Xen version 4.0.1 (Debian 4.0.1-1) (waldi@debian.org) (gcc version 4.4.5 20100824 (prerelease) (Debian 4.4.4-11) ) Fri Sep  3 15:38:12 UTC 2010
(XEN) Bootloader: GRUB 1.98+20100804-7
(XEN) Command line: dom0_mem=2048M
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 2 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009e800 (usable)
(XEN)  000000000009e800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000bf770000 (usable)
(XEN)  00000000bf770000 - 00000000bf77e000 (ACPI data)
(XEN)  00000000bf77e000 - 00000000bf7d0000 (ACPI NVS)
(XEN)  00000000bf7d0000 - 00000000bf7e0000 (reserved)
(XEN)  00000000bf7ec000 - 00000000c0000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ffa00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000640000000 (usable)
(XEN) ACPI: RSDP 000FA7E0, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT BF770100, 006C (r1 022410 XSDT1354 20100224 MSFT       97)
(XEN) ACPI: FACP BF770290, 00F4 (r4 022410 FACP1354 20100224 MSFT       97)
(XEN) ACPI: DSDT BF770540, 627D (r2  7010_ 7010_001        1 INTL 20051117)
(XEN) ACPI: FACS BF77E000, 0040
(XEN) ACPI: APIC BF770390, 011E (r2 022410 APIC1354 20100224 MSFT       97)
(XEN) ACPI: MCFG BF7704B0, 003C (r1 022410 OEMMCFG  20100224 MSFT       97)
(XEN) ACPI: SPMI BF7704F0, 0041 (r5 022410 OEMSPMI  20100224 MSFT       97)
(XEN) ACPI: OEMB BF77E040, 0082 (r1 022410 OEMB1354 20100224 MSFT       97)
(XEN) ACPI: SRAT BF77A540, 0150 (r2 022410 OEMSRAT         1 INTL        1)
(XEN) ACPI: HPET BF77A690, 0038 (r1 022410 OEMHPET  20100224 MSFT       97)
(XEN) ACPI: DMAR BF77E0D0, 0128 (r1    AMI  OEMDMAR        1 MSFT       97)
(XEN) ACPI: SSDT BF780410, 0363 (r1 DpgPmm    CpuPm       12 INTL 20051117)
(XEN) System RAM: 24567MB (25156664kB)
(XEN) Domain heap initialised DMA width 31 bits
(XEN) Processor #0 6:12 APIC version 21
(XEN) Processor #2 6:12 APIC version 21
(XEN) Processor #18 6:12 APIC version 21
(XEN) Processor #20 6:12 APIC version 21
(XEN) Processor #32 6:12 APIC version 21
(XEN) Processor #34 6:12 APIC version 21
(XEN) Processor #50 6:12 APIC version 21
(XEN) Processor #52 6:12 APIC version 21
(XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 3, version 32, address 0xfec8a000, GSI 24-47
(XEN) Enabling APIC mode:  Flat.  Using 2 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2400.140 MHz processor.
(XEN) Initing memory sharing.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) EPT supports 2MB super page.
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Intel VT-d Snoop Control not supported.
(XEN) Intel VT-d DMA Passthrough not supported.
(XEN) Intel VT-d Queued Invalidation supported.
(XEN) Intel VT-d Interrupt Remapping supported.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) Total of 8 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) TSC is reliable, synchronization unnecessary
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) Brought up 8 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x16b5000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000238000000->000000023c000000 (507904 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff816b5000
(XEN)  Init. ramdisk: ffffffff816b5000->ffffffff83211200
(XEN)  Phys-Mach map: ffffffff83212000->ffffffff83612000
(XEN)  Start info:    ffffffff83612000->ffffffff836124b4
(XEN)  Page tables:   ffffffff83613000->ffffffff83632000
(XEN)  Boot stack:    ffffffff83632000->ffffffff83633000
(XEN)  TOTAL:         ffffffff80000000->ffffffff83800000
(XEN)  ENTRY ADDRESS: ffffffff81506200
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Scrubbing Free RAM: .............................................................................................................................................................................................................................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 176kB init memory.
(XEN) save.c:72:d0 Domain 1 expects freq 2400MHz but host's freq is 2400MHz: trap and emulate rdtsc
(XEN) save.c:72:d0 Domain 2 expects freq 2400MHz but host's freq is 2400MHz: trap and emulate rdtsc
(XEN) msi.c:715: MSI is already in use on device 02:00.0
(XEN) msi.c:715: MSI is already in use on device 02:00.0
(XEN) msi.c:715: MSI is already in use on device 02:00.0

[-- Attachment #3: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Mark Adams]

On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> > I've just noticed this at the end of xm dmesg
> > 
> > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > 
> > Something else trying to use the device being exported? (the nics are
> > 02:00.0 and 03:00.0)
> 
> Hmm, looks like it, but it should not have happend. Can you attach
> the output of 'xm dmesg' and also do the 'xm debug-keys ..' that I
> asked for in the previous e-mail?
> 
> Jan, the fixes for the MSI you did, they weren't for 4.0.1 right? Just
> for unstable?
> 

Any further idea's on this? Is it a xen bug if the hidden device is being
accessed in dom0? or is there an overlap somewhere? (not sure how this
would work)..

Regards,
Mark

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Konrad Rzeszutek Wilk]

On Fri, Nov 12, 2010 at 05:10:58PM +0000, Mark Adams wrote:
> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> > > I've just noticed this at the end of xm dmesg
> > > 
> > > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > 
> > > Something else trying to use the device being exported? (the nics are
> > > 02:00.0 and 03:00.0)
> > 
> > Hmm, looks like it, but it should not have happend. Can you attach
> > the output of 'xm dmesg' and also do the 'xm debug-keys ..' that I
> > asked for in the previous e-mail?
> > 
> > Jan, the fixes for the MSI you did, they weren't for 4.0.1 right? Just
> > for unstable?
> > 
> 
> Any further idea's on this? Is it a xen bug if the hidden device is being
> accessed in dom0? or is there an overlap somewhere? (not sure how this
> would work)..

I was going to look in the source today to get an idea but never got to it...

You might, as I mentioned in earlier emails, try to setup a serial console
or netconsole and log the Linux kernel output when it hangs/fails.

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Mark Adams]

On 12 Nov 2010, at 22:22, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> On Fri, Nov 12, 2010 at 05:10:58PM +0000, Mark Adams wrote:
>> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
>>>> I've just noticed this at the end of xm dmesg
>>>> 
>>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
>>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
>>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
>>>> 
>>>> Something else trying to use the device being exported? (the nics are
>>>> 02:00.0 and 03:00.0)
>>> 
>>> Hmm, looks like it, but it should not have happend. Can you attach
>>> the output of 'xm dmesg' and also do the 'xm debug-keys ..' that I
>>> asked for in the previous e-mail?
>>> 
>>> Jan, the fixes for the MSI you did, they weren't for 4.0.1 right? Just
>>> for unstable?
>>> 
>> 
>> Any further idea's on this? Is it a xen bug if the hidden device is being
>> accessed in dom0? or is there an overlap somewhere? (not sure how this
>> would work)..
> 
> I was going to look in the source today to get an idea but never got to it...
> 
> You might, as I mentioned in earlier emails, try to setup a serial console
> or netconsole and log the Linux kernel output when it hangs/fails.
> 

can't do this unfortunately, the server
Is in use so not able to just let it hang again... The passthrough is not in use now until I think there is some possible solution to get rid of the MSI conflict (when it won't hang anymore!)

> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Mark Adams]

On Sun, Nov 14, 2010 at 05:15:02PM +0000, Mark Adams wrote:
> 
> On 12 Nov 2010, at 22:22, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> > On Fri, Nov 12, 2010 at 05:10:58PM +0000, Mark Adams wrote:
> >> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> >>>> I've just noticed this at the end of xm dmesg
> >>>> 
> >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> >>>> 
> >>>> Something else trying to use the device being exported? (the nics are
> >>>> 02:00.0 and 03:00.0)
> >>> 
> >>> Hmm, looks like it, but it should not have happend. Can you attach
> >>> the output of 'xm dmesg' and also do the 'xm debug-keys ..' that I
> >>> asked for in the previous e-mail?
> >>> 
> >>> Jan, the fixes for the MSI you did, they weren't for 4.0.1 right? Just
> >>> for unstable?
> >>> 
> >> 
> >> Any further idea's on this? Is it a xen bug if the hidden device is being
> >> accessed in dom0? or is there an overlap somewhere? (not sure how this
> >> would work)..
> > 
> > I was going to look in the source today to get an idea but never got to it...
> > 
> > You might, as I mentioned in earlier emails, try to setup a serial console
> > or netconsole and log the Linux kernel output when it hangs/fails.
> > 
> 
> can't do this unfortunately, the server
> Is in use so not able to just let it hang again... The passthrough is
> not in use now until I think there is some possible solution to get
> rid of the MSI conflict (when it won't hang anymore!)

Is there anything else I can do to help with resolution on this issue? I
see another user is also having a similar problem with (quiet possibly)
PCI passthrough causing their RAID array to go offline.

Did my logs show anything useful? You mentioned some fix for MSI earlier
has this been corrected in a newer version of Xen?

Regards,
Mark

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Konrad Rzeszutek Wilk]

On Sun, Nov 14, 2010 at 05:15:02PM +0000, Mark Adams wrote:
> 
> On 12 Nov 2010, at 22:22, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> > On Fri, Nov 12, 2010 at 05:10:58PM +0000, Mark Adams wrote:
> >> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> >>>> I've just noticed this at the end of xm dmesg
> >>>> 
> >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0

Looking briefly at the code it means that somebody enabled the MSI
already on the device and did not disable them. But I wonder how
you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
or pciback.hide (for older kernels) to "hide" the devices away from the
Linux Dom0 kernel?


> > I was going to look in the source today to get an idea but never got to it...
> > 
> > You might, as I mentioned in earlier emails, try to setup a serial console
> > or netconsole and log the Linux kernel output when it hangs/fails.
> > 
> 
> can't do this unfortunately, the server
> Is in use so not able to just let it hang again... The passthrough is not in use now until I think there is some possible solution to get rid of the MSI conflict (when it won't hang anymore!)

Didn't you say that you had two servers and saw this problem on another
box too?

Without more details on the Xen hypervisor line or the kernel line when
the failure occurs I sadly can't help you.

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Mark Adams]

On Mon, Nov 15, 2010 at 12:15:44PM -0500, Konrad Rzeszutek Wilk wrote:
> On Sun, Nov 14, 2010 at 05:15:02PM +0000, Mark Adams wrote:
> > 
> > On 12 Nov 2010, at 22:22, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > 
> > > On Fri, Nov 12, 2010 at 05:10:58PM +0000, Mark Adams wrote:
> > >> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> > >>>> I've just noticed this at the end of xm dmesg
> > >>>> 
> > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> 
> Looking briefly at the code it means that somebody enabled the MSI
> already on the device and did not disable them. But I wonder how
> you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> or pciback.hide (for older kernels) to "hide" the devices away from the
> Linux Dom0 kernel?

using xen-pciback.hide as its a pvops kernel (debian squeeze
2.6.32-5-27)

> 
> 
> > > I was going to look in the source today to get an idea but never got to it...
> > > 
> > > You might, as I mentioned in earlier emails, try to setup a serial console
> > > or netconsole and log the Linux kernel output when it hangs/fails.
> > > 
> > 
> > can't do this unfortunately, the server
> > Is in use so not able to just let it hang again... The passthrough is not in use now until I think there is some possible solution to get rid of the MSI conflict (when it won't hang anymore!)
> 
> Didn't you say that you had two servers and saw this problem on another
> box too?
> 
> Without more details on the Xen hypervisor line or the kernel line when
> the failure occurs I sadly can't help you.

Yes this occurs on both servers that I've tried it on. Doesn't the MSI
log above indicate that there is a conflict - which is what ends up
causing the device to go offline? Is there no other way to identify the
conflict?

Regards,
Mark

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Konrad Rzeszutek Wilk]

On Mon, Nov 15, 2010 at 05:23:09PM +0000, Mark Adams wrote:
> On Mon, Nov 15, 2010 at 12:15:44PM -0500, Konrad Rzeszutek Wilk wrote:
> > On Sun, Nov 14, 2010 at 05:15:02PM +0000, Mark Adams wrote:
> > > 
> > > On 12 Nov 2010, at 22:22, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > > 
> > > > On Fri, Nov 12, 2010 at 05:10:58PM +0000, Mark Adams wrote:
> > > >> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> > > >>>> I've just noticed this at the end of xm dmesg
> > > >>>> 
> > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > 
> > Looking briefly at the code it means that somebody enabled the MSI
> > already on the device and did not disable them. But I wonder how
> > you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> > or pciback.hide (for older kernels) to "hide" the devices away from the
> > Linux Dom0 kernel?
> 
> using xen-pciback.hide as its a pvops kernel (debian squeeze
> 2.6.32-5-27)

Ok. Then it might be worth looking in when this happens. I think
there is an argument on the Xen hyperisor line to include the time-stamp, but
I don't remember it :-(

> > Didn't you say that you had two servers and saw this problem on another
> > box too?
> > 
> > Without more details on the Xen hypervisor line or the kernel line when
> > the failure occurs I sadly can't help you.
> 
> Yes this occurs on both servers that I've tried it on. Doesn't the MSI
> log above indicate that there is a conflict - which is what ends up
> causing the device to go offline? Is there no other way to identify the

Could be, but it is unclear - it depends on when the message pops out.

But that does not help with finding out why your RAID controller goes offline.

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Mark Adams]

On Mon, Nov 15, 2010 at 12:44:13PM -0500, Konrad Rzeszutek Wilk wrote:
> On Mon, Nov 15, 2010 at 05:23:09PM +0000, Mark Adams wrote:
> > On Mon, Nov 15, 2010 at 12:15:44PM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Sun, Nov 14, 2010 at 05:15:02PM +0000, Mark Adams wrote:
> > > > 
> > > > On 12 Nov 2010, at 22:22, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > > > 
> > > > > On Fri, Nov 12, 2010 at 05:10:58PM +0000, Mark Adams wrote:
> > > > >> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> > > > >>>> I've just noticed this at the end of xm dmesg
> > > > >>>> 
> > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > 
> > > Looking briefly at the code it means that somebody enabled the MSI
> > > already on the device and did not disable them. But I wonder how
> > > you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> > > or pciback.hide (for older kernels) to "hide" the devices away from the
> > > Linux Dom0 kernel?
> > 
> > using xen-pciback.hide as its a pvops kernel (debian squeeze
> > 2.6.32-5-27)
> 
> Ok. Then it might be worth looking in when this happens. I think
> there is an argument on the Xen hyperisor line to include the time-stamp, but
> I don't remember it :-(
> 
> > > Didn't you say that you had two servers and saw this problem on another
> > > box too?
> > > 
> > > Without more details on the Xen hypervisor line or the kernel line when
> > > the failure occurs I sadly can't help you.
> > 
> > Yes this occurs on both servers that I've tried it on. Doesn't the MSI
> > log above indicate that there is a conflict - which is what ends up
> > causing the device to go offline? Is there no other way to identify the
> 
> Could be, but it is unclear - it depends on when the message pops out.

The message appears immediately on boot.

> 
> But that does not help with finding out why your RAID controller goes offline.

Maybe the other user having a similar issue can help with logs if it is
still happening to him. I'll ask on that thread...

Regards,
Mark

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Mark Adams]

> > > > > >>>> 
> > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > 
> > > > Looking briefly at the code it means that somebody enabled the MSI
> > > > already on the device and did not disable them. But I wonder how
> > > > you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> > > > or pciback.hide (for older kernels) to "hide" the devices away from the
> > > > Linux Dom0 kernel?
> > > 
> > > using xen-pciback.hide as its a pvops kernel (debian squeeze
> > > 2.6.32-5-27)
> > 
> > Ok. Then it might be worth looking in when this happens. I think
> > there is an argument on the Xen hyperisor line to include the time-stamp, but
> > I don't remember it :-(
> > 

I've got a test setup in place now, and am trying to reproduce this.
I've not connected up serial as yet, but can see the following logs in
the qemu-dm log file when I get the "MSI is already in use" errors
above. Note also that this error -always- shows for the first specified
device in the pci= field, and not the 2nd.

pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation
pci_intx: intx=1
pt_msix_update_one: Update msix entry 0 with pirq 4d gvec 59
pt_msix_update_one: Update msix entry 1 with pirq 4c gvec 61
pt_msix_update_one: Update msix entry 2 with pirq 4b gvec 69
pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation
pci_intx: intx=2

I have also seen the following log just once, not sure if it's related:

(XEN) domctl.c:811:d0 XEN_DOMCTL_test_assign_device: 2:0.0 already assigned, or non-existent

Does this help at all with debugging my issues?

Regards,
Mark

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Konrad Rzeszutek Wilk]

On Wed, Nov 24, 2010 at 05:59:26PM +0000, Mark Adams wrote:
> > > > > > >>>> 
> > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > 
> > > > > Looking briefly at the code it means that somebody enabled the MSI
> > > > > already on the device and did not disable them. But I wonder how
> > > > > you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> > > > > or pciback.hide (for older kernels) to "hide" the devices away from the
> > > > > Linux Dom0 kernel?
> > > > 
> > > > using xen-pciback.hide as its a pvops kernel (debian squeeze
> > > > 2.6.32-5-27)
> > > 
> > > Ok. Then it might be worth looking in when this happens. I think
> > > there is an argument on the Xen hyperisor line to include the time-stamp, but
> > > I don't remember it :-(
> > > 
> 
> I've got a test setup in place now, and am trying to reproduce this.
> I've not connected up serial as yet, but can see the following logs in
> the qemu-dm log file when I get the "MSI is already in use" errors
> above. Note also that this error -always- shows for the first specified
> device in the pci= field, and not the 2nd.
> 
> pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation
> pci_intx: intx=1
> pt_msix_update_one: Update msix entry 0 with pirq 4d gvec 59
> pt_msix_update_one: Update msix entry 1 with pirq 4c gvec 61
> pt_msix_update_one: Update msix entry 2 with pirq 4b gvec 69
> pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation
> pci_intx: intx=2
> 
> I have also seen the following log just once, not sure if it's related:
> 
> (XEN) domctl.c:811:d0 XEN_DOMCTL_test_assign_device: 2:0.0 already assigned, or non-existent
> 
> Does this help at all with debugging my issues?

Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
machine is toast. I mentioned in the previous email the key sequences - look on Google
on how to pass in SysRQ if you are using a serial concentrator.

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Mark Adams]

On Wed, Nov 24, 2010 at 03:28:43PM -0500, Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 24, 2010 at 05:59:26PM +0000, Mark Adams wrote:
> > > > > > > >>>> 
> > > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > 
> > > > > > Looking briefly at the code it means that somebody enabled the MSI
> > > > > > already on the device and did not disable them. But I wonder how
> > > > > > you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> > > > > > or pciback.hide (for older kernels) to "hide" the devices away from the
> > > > > > Linux Dom0 kernel?
> > 
> > I've got a test setup in place now, and am trying to reproduce this.
> > I've not connected up serial as yet, but can see the following logs in
> > the qemu-dm log file when I get the "MSI is already in use" errors
> > above. Note also that this error -always- shows for the first specified
> > device in the pci= field, and not the 2nd.
> > 

In my new test setup, I have seen some strange behaviour. 1 of the HVM's
(with identical config in dom0 and domU) suddenly would not allow the
igb driver to be loaded in domU, even though the device was visible in
lspci. Shutting the machine down, removing the power cord, waiting 5
seconds then plugging it in again corrected that issue - Is this
possibly a motherboard bug? I have also disabled the SR-IOV
functionality in the BIOS incase this is causing any issues.

In addition, to try to correct the MSI issue noted above, I have changed
my pci= line to the following:

pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]

This has stopped the "already in use on device" log, and the devices
appear to show correctly in the domU. Is it safe to disable
msitranslate? as I understand it, its for allowing multifunction devices
to be seen as such in domU. Is that correct?

I haven't been able to reproduce the dropped raid issue yet, but I am
awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
this due to their very high interrupt usage (2000 per second).

In the mean time, I can see the following in the qemu-dm logs now with
the msitranslate=0 enabled. Is it anything to worry about?

pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.

> 
> Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
> machine is toast. I mentioned in the previous email the key sequences - look on Google
> on how to pass in SysRQ if you are using a serial concentrator.

I will do this when I can get the machine to crash.

Best Regards,
Mark

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Mark Adams]

On Fri, Nov 26, 2010 at 11:15:20AM +0000, Mark Adams wrote:
> On Wed, Nov 24, 2010 at 03:28:43PM -0500, Konrad Rzeszutek Wilk wrote:
> > On Wed, Nov 24, 2010 at 05:59:26PM +0000, Mark Adams wrote:
> > > > > > > > >>>> 
> > > > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > > > > 
> > > > > > > Looking briefly at the code it means that somebody enabled the MSI
> > > > > > > already on the device and did not disable them. But I wonder how
> > > > > > > you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> > > > > > > or pciback.hide (for older kernels) to "hide" the devices away from the
> > > > > > > Linux Dom0 kernel?
> > > 
> > > I've got a test setup in place now, and am trying to reproduce this.
> > > I've not connected up serial as yet, but can see the following logs in
> > > the qemu-dm log file when I get the "MSI is already in use" errors
> > > above. Note also that this error -always- shows for the first specified
> > > device in the pci= field, and not the 2nd.
> > > 
> 
> In my new test setup, I have seen some strange behaviour. 1 of the HVM's
> (with identical config in dom0 and domU) suddenly would not allow the
> igb driver to be loaded in domU, even though the device was visible in
> lspci. Shutting the machine down, removing the power cord, waiting 5
> seconds then plugging it in again corrected that issue - Is this
> possibly a motherboard bug? I have also disabled the SR-IOV
> functionality in the BIOS incase this is causing any issues.
> 
> In addition, to try to correct the MSI issue noted above, I have changed
> my pci= line to the following:
> 
> pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]

Apologies for replying to my own post, but I'm having an issue with this
setup in that I can not get a link on the 2nd interface that I'm passing
through. I've tried with msitranslate both on and off, with no success.

The device shows in the domU as an interface correctly, but even with
the link up (link led's show on the interface) domU always shows the
eth2 interface as not ready.

[    7.001784] ADDRCONF(NETDEV_UP): eth1: link is not ready
[    7.047903] ADDRCONF(NETDEV_UP): eth2: link is not ready
[   10.108995] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   10.109653] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   16.468102] eth0: no IPv6 routers present
[   20.404092] eth1: no IPv6 routers present

I've tried using the ET dual port card (igb) aswell as the onboard
interfaces (e1000e) with exactly the same result. 

The eth0 interface is a xen bridge device, and if I remove this, both
passthrough interfaces work correctly.

Can you not have a bridge and pci-passthrough operational? is there a
limit of 2 NIC's in a HVM domU? (this doesn't sound right...)

Regards,
Mark

HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Konrad Rzeszutek Wilk]

> 
> In my new test setup, I have seen some strange behaviour. 1 of the HVM's
> (with identical config in dom0 and domU) suddenly would not allow the
> igb driver to be loaded in domU, even though the device was visible in

Let's create a new thread for this other issue.

> lspci. Shutting the machine down, removing the power cord, waiting 5
> seconds then plugging it in again corrected that issue - Is this
> possibly a motherboard bug? I have also disabled the SR-IOV
> functionality in the BIOS incase this is causing any issues.
> 
> In addition, to try to correct the MSI issue noted above, I have changed
> my pci= line to the following:
> 
> pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]

With the msi_translate=1 turned on the DomU HVM guests did work, right?

> 
> This has stopped the "already in use on device" log, and the devices
> appear to show correctly in the domU. Is it safe to disable
> msitranslate? as I understand it, its for allowing multifunction devices
> to be seen as such in domU. Is that correct?
> 
> I haven't been able to reproduce the dropped raid issue yet, but I am
> awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
> this due to their very high interrupt usage (2000 per second).

OK.
> 
> In the mean time, I can see the following in the qemu-dm logs now with
> the msitranslate=0 enabled. Is it anything to worry about?

Well, the  "Error" ones are pretty bad, thought I am having a hard time
understanding what it means. Lets copy some of the QEMU folks on this.

> pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
> pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
> pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
> pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
> pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
> pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
> pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
> pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
> pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
> pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
> pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
> pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> 
> > 
> > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
> > machine is toast. I mentioned in the previous email the key sequences - look on Google
> > on how to pass in SysRQ if you are using a serial concentrator.
> 
> I will do this when I can get the machine to crash.
> 
> Best Regards,
> Mark
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Mark Adams]

Hi - Apologies to top post this, but after alot of testing, I believe
there must be an issue with IRQ's going missing between domU and dom0.
Unfortunately I have no data to prove this!

With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel
grub config, all 3 NIC's appear OK in the domU however I still had
issues with the red-fone ISDN box. The interrupts were showing correctly
(2000/s) in the domU but communication to the device via the NIC was
still being interrupted (as shown in the asterisk console)Note that to
get the igb driver to allow this many interrupts, the
InterruptThrottleRate was set to 0. The same config (red-fone box,
asterisk etc) works fine with a physical server.

There is also the additional issue that I could not get the passthrough
NIC's to show correctly when I also had a bridge setup.

Throughout my testing however, I could not get the machine to crash.

Not sure where to go with this one. For now we are keeping our VoIP
servers physical when ISDN connections are required.

Regards,
Mark

On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:
> > 
> > In my new test setup, I have seen some strange behaviour. 1 of the HVM's
> > (with identical config in dom0 and domU) suddenly would not allow the
> > igb driver to be loaded in domU, even though the device was visible in
> 
> Let's create a new thread for this other issue.
> 
> > lspci. Shutting the machine down, removing the power cord, waiting 5
> > seconds then plugging it in again corrected that issue - Is this
> > possibly a motherboard bug? I have also disabled the SR-IOV
> > functionality in the BIOS incase this is causing any issues.
> > 
> > In addition, to try to correct the MSI issue noted above, I have changed
> > my pci= line to the following:
> > 
> > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]
> 
> With the msi_translate=1 turned on the DomU HVM guests did work, right?
> 
> > 
> > This has stopped the "already in use on device" log, and the devices
> > appear to show correctly in the domU. Is it safe to disable
> > msitranslate? as I understand it, its for allowing multifunction devices
> > to be seen as such in domU. Is that correct?
> > 
> > I haven't been able to reproduce the dropped raid issue yet, but I am
> > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
> > this due to their very high interrupt usage (2000 per second).
> 
> OK.
> > 
> > In the mean time, I can see the following in the qemu-dm logs now with
> > the msitranslate=0 enabled. Is it anything to worry about?
> 
> Well, the  "Error" ones are pretty bad, thought I am having a hard time
> understanding what it means. Lets copy some of the QEMU folks on this.
> 
> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
> > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
> > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
> > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
> > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
> > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
> > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
> > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
> > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
> > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> > 
> > > 
> > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
> > > machine is toast. I mentioned in the previous email the key sequences - look on Google
> > > on how to pass in SysRQ if you are using a serial concentrator.
> > 
> > I will do this when I can get the machine to crash.
> > 
> > Best Regards,
> > Mark
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Sander Eikelenboom]

Hello Mark,

Just a recap:
     you pass through:
     - 3 physical nics/IGB
     - 1 ISDN pci ISDN box
     - all using msi/msi-x interrupts ?

Have you tried using a PV domU instead of a HVM domU ?
Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?


--
Sander



Wednesday, December 8, 2010, 1:58:55 PM, you wrote:

> Hi - Apologies to top post this, but after alot of testing, I believe
> there must be an issue with IRQ's going missing between domU and dom0.
> Unfortunately I have no data to prove this!

> With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel
> grub config, all 3 NIC's appear OK in the domU however I still had
> issues with the red-fone ISDN box. The interrupts were showing correctly
> (2000/s) in the domU but communication to the device via the NIC was
> still being interrupted (as shown in the asterisk console)Note that to
> get the igb driver to allow this many interrupts, the
> InterruptThrottleRate was set to 0. The same config (red-fone box,
> asterisk etc) works fine with a physical server.

> There is also the additional issue that I could not get the passthrough
> NIC's to show correctly when I also had a bridge setup.

> Throughout my testing however, I could not get the machine to crash.

> Not sure where to go with this one. For now we are keeping our VoIP
> servers physical when ISDN connections are required.

> Regards,
> Mark

> On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:
>> > 
>> > In my new test setup, I have seen some strange behaviour. 1 of the HVM's
>> > (with identical config in dom0 and domU) suddenly would not allow the
>> > igb driver to be loaded in domU, even though the device was visible in
>> 
>> Let's create a new thread for this other issue.
>> 
>> > lspci. Shutting the machine down, removing the power cord, waiting 5
>> > seconds then plugging it in again corrected that issue - Is this
>> > possibly a motherboard bug? I have also disabled the SR-IOV
>> > functionality in the BIOS incase this is causing any issues.
>> > 
>> > In addition, to try to correct the MSI issue noted above, I have changed
>> > my pci= line to the following:
>> > 
>> > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]
>> 
>> With the msi_translate=1 turned on the DomU HVM guests did work, right?
>> 
>> > 
>> > This has stopped the "already in use on device" log, and the devices
>> > appear to show correctly in the domU. Is it safe to disable
>> > msitranslate? as I understand it, its for allowing multifunction devices
>> > to be seen as such in domU. Is that correct?
>> > 
>> > I haven't been able to reproduce the dropped raid issue yet, but I am
>> > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
>> > this due to their very high interrupt usage (2000 per second).
>> 
>> OK.
>> > 
>> > In the mean time, I can see the following in the qemu-dm logs now with
>> > the msitranslate=0 enabled. Is it anything to worry about?
>> 
>> Well, the  "Error" ones are pretty bad, thought I am having a hard time
>> understanding what it means. Lets copy some of the QEMU folks on this.
>> 
>> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
>> > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
>> > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
>> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
>> > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
>> > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
>> > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
>> > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
>> > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
>> > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
>> > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
>> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
>> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
>> > 
>> > > 
>> > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
>> > > machine is toast. I mentioned in the previous email the key sequences - look on Google
>> > > on how to pass in SysRQ if you are using a serial concentrator.
>> > 
>> > I will do this when I can get the machine to crash.
>> > 
>> > Best Regards,
>> > Mark
>> > 
>> > _______________________________________________
>> > Xen-devel mailing list
>> > Xen-devel@lists.xensource.com
>> > http://lists.xensource.com/xen-devel





-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Mark Adams]

On Wed, Dec 08, 2010 at 02:37:15PM +0100, Sander Eikelenboom wrote:
> Hello Mark,

Hi

> 
> Just a recap:
>      you pass through:
>      - 3 physical nics/IGB
>      - 1 ISDN pci ISDN box

The redfone box runs on 1 of the nics - its not seperate. It converts
ISDN to TDMoE see here.. http://www.red-fone.com/

>      - all using msi/msi-x interrupts ?

I tried using msi/msi-x interrupts, but it caused the raid card to drop
off (after some use) and provided seemingly even worse performance than
pegging everything back to legacy.

> 
> Have you tried using a PV domU instead of a HVM domU ?

I initially tried PV but had issues with the igb NIC's. There was
another thread somewhere about my issues with that.


> Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?
> 
> 
> --
> Sander
> 
> 
> 
> Wednesday, December 8, 2010, 1:58:55 PM, you wrote:
> 
> > Hi - Apologies to top post this, but after alot of testing, I believe
> > there must be an issue with IRQ's going missing between domU and dom0.
> > Unfortunately I have no data to prove this!
> 
> > With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel
> > grub config, all 3 NIC's appear OK in the domU however I still had
> > issues with the red-fone ISDN box. The interrupts were showing correctly
> > (2000/s) in the domU but communication to the device via the NIC was
> > still being interrupted (as shown in the asterisk console)Note that to
> > get the igb driver to allow this many interrupts, the
> > InterruptThrottleRate was set to 0. The same config (red-fone box,
> > asterisk etc) works fine with a physical server.
> 
> > There is also the additional issue that I could not get the passthrough
> > NIC's to show correctly when I also had a bridge setup.
> 
> > Throughout my testing however, I could not get the machine to crash.
> 
> > Not sure where to go with this one. For now we are keeping our VoIP
> > servers physical when ISDN connections are required.
> 
> > Regards,
> > Mark
> 
> > On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:
> >> > 
> >> > In my new test setup, I have seen some strange behaviour. 1 of the HVM's
> >> > (with identical config in dom0 and domU) suddenly would not allow the
> >> > igb driver to be loaded in domU, even though the device was visible in
> >> 
> >> Let's create a new thread for this other issue.
> >> 
> >> > lspci. Shutting the machine down, removing the power cord, waiting 5
> >> > seconds then plugging it in again corrected that issue - Is this
> >> > possibly a motherboard bug? I have also disabled the SR-IOV
> >> > functionality in the BIOS incase this is causing any issues.
> >> > 
> >> > In addition, to try to correct the MSI issue noted above, I have changed
> >> > my pci= line to the following:
> >> > 
> >> > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]
> >> 
> >> With the msi_translate=1 turned on the DomU HVM guests did work, right?
> >> 
> >> > 
> >> > This has stopped the "already in use on device" log, and the devices
> >> > appear to show correctly in the domU. Is it safe to disable
> >> > msitranslate? as I understand it, its for allowing multifunction devices
> >> > to be seen as such in domU. Is that correct?
> >> > 
> >> > I haven't been able to reproduce the dropped raid issue yet, but I am
> >> > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
> >> > this due to their very high interrupt usage (2000 per second).
> >> 
> >> OK.
> >> > 
> >> > In the mean time, I can see the following in the qemu-dm logs now with
> >> > the msitranslate=0 enabled. Is it anything to worry about?
> >> 
> >> Well, the  "Error" ones are pretty bad, thought I am having a hard time
> >> understanding what it means. Lets copy some of the QEMU folks on this.
> >> 
> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
> >> > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
> >> > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
> >> > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
> >> > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
> >> > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
> >> > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
> >> > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
> >> > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
> >> > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> >> > 
> >> > > 
> >> > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
> >> > > machine is toast. I mentioned in the previous email the key sequences - look on Google
> >> > > on how to pass in SysRQ if you are using a serial concentrator.
> >> > 
> >> > I will do this when I can get the machine to crash.
> >> > 
> >> > Best Regards,
> >> > Mark
> >> > 
> >> > _______________________________________________
> >> > Xen-devel mailing list
> >> > Xen-devel@lists.xensource.com
> >> > http://lists.xensource.com/xen-devel
> 
> 
> 
> 
> 
> -- 
> Best regards,
>  Sander                            mailto:linux@eikelenboom.it
> 

Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Sander Eikelenboom]

Wednesday, December 8, 2010, 2:48:48 PM, you wrote:

> On Wed, Dec 08, 2010 at 02:37:15PM +0100, Sander Eikelenboom wrote:
>> Hello Mark,

> Hi

>> 
>> Just a recap:
>>      you pass through:
>>      - 3 physical nics/IGB
>>      - 1 ISDN pci ISDN box

> The redfone box runs on 1 of the nics - its not seperate. It converts
> ISDN to TDMoE see here.. http://www.red-fone.com/

So the problem is probably with the igb's.
Searching showed http://forums.virtualbox.org/viewtopic.php?f=7&t=32171 , perhaps worth a try ?

Have you tried with just 1 IGB, and/or another simple 1gb NIC (non intel) to see if it's due to any of the special offload features ?


>>      - all using msi/msi-x interrupts ?

> I tried using msi/msi-x interrupts, but it caused the raid card to drop
> off (after some use) and provided seemingly even worse performance than
> pegging everything back to legacy.

>> 
>> Have you tried using a PV domU instead of a HVM domU ?

> I initially tried PV but had issues with the igb NIC's. There was
> another thread somewhere about my issues with that.


>> Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?
>> 
>> 
>> --
>> Sander
>> 
>> 
>> 
>> Wednesday, December 8, 2010, 1:58:55 PM, you wrote:
>> 
>> > Hi - Apologies to top post this, but after alot of testing, I believe
>> > there must be an issue with IRQ's going missing between domU and dom0.
>> > Unfortunately I have no data to prove this!
>> 
>> > With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel
>> > grub config, all 3 NIC's appear OK in the domU however I still had
>> > issues with the red-fone ISDN box. The interrupts were showing correctly
>> > (2000/s) in the domU but communication to the device via the NIC was
>> > still being interrupted (as shown in the asterisk console)Note that to
>> > get the igb driver to allow this many interrupts, the
>> > InterruptThrottleRate was set to 0. The same config (red-fone box,
>> > asterisk etc) works fine with a physical server.
>> 
>> > There is also the additional issue that I could not get the passthrough
>> > NIC's to show correctly when I also had a bridge setup.
>> 
>> > Throughout my testing however, I could not get the machine to crash.
>> 
>> > Not sure where to go with this one. For now we are keeping our VoIP
>> > servers physical when ISDN connections are required.
>> 
>> > Regards,
>> > Mark
>> 
>> > On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:
>> >> > 
>> >> > In my new test setup, I have seen some strange behaviour. 1 of the HVM's
>> >> > (with identical config in dom0 and domU) suddenly would not allow the
>> >> > igb driver to be loaded in domU, even though the device was visible in
>> >> 
>> >> Let's create a new thread for this other issue.
>> >> 
>> >> > lspci. Shutting the machine down, removing the power cord, waiting 5
>> >> > seconds then plugging it in again corrected that issue - Is this
>> >> > possibly a motherboard bug? I have also disabled the SR-IOV
>> >> > functionality in the BIOS incase this is causing any issues.
>> >> > 
>> >> > In addition, to try to correct the MSI issue noted above, I have changed
>> >> > my pci= line to the following:
>> >> > 
>> >> > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]
>> >> 
>> >> With the msi_translate=1 turned on the DomU HVM guests did work, right?
>> >> 
>> >> > 
>> >> > This has stopped the "already in use on device" log, and the devices
>> >> > appear to show correctly in the domU. Is it safe to disable
>> >> > msitranslate? as I understand it, its for allowing multifunction devices
>> >> > to be seen as such in domU. Is that correct?
>> >> > 
>> >> > I haven't been able to reproduce the dropped raid issue yet, but I am
>> >> > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
>> >> > this due to their very high interrupt usage (2000 per second).
>> >> 
>> >> OK.
>> >> > 
>> >> > In the mean time, I can see the following in the qemu-dm logs now with
>> >> > the msitranslate=0 enabled. Is it anything to worry about?
>> >> 
>> >> Well, the  "Error" ones are pretty bad, thought I am having a hard time
>> >> understanding what it means. Lets copy some of the QEMU folks on this.
>> >> 
>> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
>> >> > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
>> >> > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
>> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
>> >> > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
>> >> > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
>> >> > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
>> >> > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
>> >> > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
>> >> > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
>> >> > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
>> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
>> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
>> >> > 
>> >> > > 
>> >> > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
>> >> > > machine is toast. I mentioned in the previous email the key sequences - look on Google
>> >> > > on how to pass in SysRQ if you are using a serial concentrator.
>> >> > 
>> >> > I will do this when I can get the machine to crash.
>> >> > 
>> >> > Best Regards,
>> >> > Mark
>> >> > 
>> >> > _______________________________________________
>> >> > Xen-devel mailing list
>> >> > Xen-devel@lists.xensource.com
>> >> > http://lists.xensource.com/xen-devel
>> 
>> 
>> 
>> 
>> 
>> -- 
>> Best regards,
>>  Sander                            mailto:linux@eikelenboom.it
>> 



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: [Xen-users] Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Mark Adams]

On Wed, Dec 08, 2010 at 03:05:50PM +0100, Sander Eikelenboom wrote:
> 
> Wednesday, December 8, 2010, 2:48:48 PM, you wrote:
> 
> > On Wed, Dec 08, 2010 at 02:37:15PM +0100, Sander Eikelenboom wrote:
> >> Hello Mark,
> 
> > Hi
> 
> >> 
> >> Just a recap:
> >>      you pass through:
> >>      - 3 physical nics/IGB
> >>      - 1 ISDN pci ISDN box
> 
> > The redfone box runs on 1 of the nics - its not seperate. It converts
> > ISDN to TDMoE see here.. http://www.red-fone.com/
> 
> So the problem is probably with the igb's.
> Searching showed http://forums.virtualbox.org/viewtopic.php?f=7&t=32171 , perhaps worth a try ?

Tried this - doesn't help.

> 
> Have you tried with just 1 IGB, and/or another simple 1gb NIC (non intel) to see if it's due to any of the special offload features ?

Haven't got any other NIC's to try unfortunately. Even if it did work
with 1, it would be no use to me as I need 3.


> 
> 
> >>      - all using msi/msi-x interrupts ?
> 
> > I tried using msi/msi-x interrupts, but it caused the raid card to drop
> > off (after some use) and provided seemingly even worse performance than
> > pegging everything back to legacy.
> 
> >> 
> >> Have you tried using a PV domU instead of a HVM domU ?
> 
> > I initially tried PV but had issues with the igb NIC's. There was
> > another thread somewhere about my issues with that.
> 
> 
> >> Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?
> >> 
> >> 
> >> --
> >> Sander
> >> 
> >> 
> >> 
> >> Wednesday, December 8, 2010, 1:58:55 PM, you wrote:
> >> 
> >> > Hi - Apologies to top post this, but after alot of testing, I believe
> >> > there must be an issue with IRQ's going missing between domU and dom0.
> >> > Unfortunately I have no data to prove this!
> >> 
> >> > With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel
> >> > grub config, all 3 NIC's appear OK in the domU however I still had
> >> > issues with the red-fone ISDN box. The interrupts were showing correctly
> >> > (2000/s) in the domU but communication to the device via the NIC was
> >> > still being interrupted (as shown in the asterisk console)Note that to
> >> > get the igb driver to allow this many interrupts, the
> >> > InterruptThrottleRate was set to 0. The same config (red-fone box,
> >> > asterisk etc) works fine with a physical server.
> >> 
> >> > There is also the additional issue that I could not get the passthrough
> >> > NIC's to show correctly when I also had a bridge setup.
> >> 
> >> > Throughout my testing however, I could not get the machine to crash.
> >> 
> >> > Not sure where to go with this one. For now we are keeping our VoIP
> >> > servers physical when ISDN connections are required.
> >> 
> >> > Regards,
> >> > Mark
> >> 
> >> > On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:
> >> >> > 
> >> >> > In my new test setup, I have seen some strange behaviour. 1 of the HVM's
> >> >> > (with identical config in dom0 and domU) suddenly would not allow the
> >> >> > igb driver to be loaded in domU, even though the device was visible in
> >> >> 
> >> >> Let's create a new thread for this other issue.
> >> >> 
> >> >> > lspci. Shutting the machine down, removing the power cord, waiting 5
> >> >> > seconds then plugging it in again corrected that issue - Is this
> >> >> > possibly a motherboard bug? I have also disabled the SR-IOV
> >> >> > functionality in the BIOS incase this is causing any issues.
> >> >> > 
> >> >> > In addition, to try to correct the MSI issue noted above, I have changed
> >> >> > my pci= line to the following:
> >> >> > 
> >> >> > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]
> >> >> 
> >> >> With the msi_translate=1 turned on the DomU HVM guests did work, right?
> >> >> 
> >> >> > 
> >> >> > This has stopped the "already in use on device" log, and the devices
> >> >> > appear to show correctly in the domU. Is it safe to disable
> >> >> > msitranslate? as I understand it, its for allowing multifunction devices
> >> >> > to be seen as such in domU. Is that correct?
> >> >> > 
> >> >> > I haven't been able to reproduce the dropped raid issue yet, but I am
> >> >> > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
> >> >> > this due to their very high interrupt usage (2000 per second).
> >> >> 
> >> >> OK.
> >> >> > 
> >> >> > In the mean time, I can see the following in the qemu-dm logs now with
> >> >> > the msitranslate=0 enabled. Is it anything to worry about?
> >> >> 
> >> >> Well, the  "Error" ones are pretty bad, thought I am having a hard time
> >> >> understanding what it means. Lets copy some of the QEMU folks on this.
> >> >> 
> >> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
> >> >> > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
> >> >> > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
> >> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
> >> >> > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
> >> >> > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
> >> >> > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
> >> >> > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
> >> >> > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
> >> >> > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
> >> >> > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> >> >> > 
> >> >> > > 
> >> >> > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
> >> >> > > machine is toast. I mentioned in the previous email the key sequences - look on Google
> >> >> > > on how to pass in SysRQ if you are using a serial concentrator.
> >> >> > 
> >> >> > I will do this when I can get the machine to crash.
> >> >> > 
> >> >> > Best Regards,
> >> >> > Mark
> >> >> > 
> >> >> > _______________________________________________
> >> >> > Xen-devel mailing list
> >> >> > Xen-devel@lists.xensource.com
> >> >> > http://lists.xensource.com/xen-devel
> >> 
> >> 
> >> 
> >> 
> >> 
> >> -- 
> >> Best regards,
> >>  Sander                            mailto:linux@eikelenboom.it
> >> 
> 
> 
> 
> -- 
> Best regards,
>  Sander                            mailto:linux@eikelenboom.it
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

Re: [Xen-users] Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Sander Eikelenboom]

Wednesday, December 8, 2010, 4:48:57 PM, you wrote:

> On Wed, Dec 08, 2010 at 03:05:50PM +0100, Sander Eikelenboom wrote:
>> 
>> Wednesday, December 8, 2010, 2:48:48 PM, you wrote:
>> 
>> > On Wed, Dec 08, 2010 at 02:37:15PM +0100, Sander Eikelenboom wrote:
>> >> Hello Mark,
>> 
>> > Hi
>> 
>> >> 
>> >> Just a recap:
>> >>      you pass through:
>> >>      - 3 physical nics/IGB
>> >>      - 1 ISDN pci ISDN box
>> 
>> > The redfone box runs on 1 of the nics - its not seperate. It converts
>> > ISDN to TDMoE see here.. http://www.red-fone.com/
>> 
>> So the problem is probably with the igb's.
>> Searching showed http://forums.virtualbox.org/viewtopic.php?f=7&t=32171 , perhaps worth a try ?

> Tried this - doesn't help.

>> 
>> Have you tried with just 1 IGB, and/or another simple 1gb NIC (non intel) to see if it's due to any of the special offload features ?

> Haven't got any other NIC's to try unfortunately. Even if it did work
> with 1, it would be no use to me as I need 3.

I understand, but simplifying the setup and trying to isolate the problem, could clarify things.

I also read you previous thread, and i saw you hide the 02:00.0 and 03:00.0 with xen-pciback (e1000e driver) there, but now you seem to be passing through 08:00.0 and 08:00.1 (igb) ?
So i assume you have already tried 2 different NIC's

http://download.intel.com/design/network/specupdt/82574.pdf though shows some errata regarding msi-x interrupts and timing issues and workarounds on the 82574 (02:00.0 and 03:00.0) nics.

--
Sander


>> 
>> 
>> >>      - all using msi/msi-x interrupts ?
>> 
>> > I tried using msi/msi-x interrupts, but it caused the raid card to drop
>> > off (after some use) and provided seemingly even worse performance than
>> > pegging everything back to legacy.
>> 
>> >> 
>> >> Have you tried using a PV domU instead of a HVM domU ?
>> 
>> > I initially tried PV but had issues with the igb NIC's. There was
>> > another thread somewhere about my issues with that.
>> 
>> 
>> >> Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?
>> >> 
>> >> 
>> >> --
>> >> Sander
>> >> 
>> >> 
>> >> 
>> >> Wednesday, December 8, 2010, 1:58:55 PM, you wrote:
>> >> 
>> >> > Hi - Apologies to top post this, but after alot of testing, I believe
>> >> > there must be an issue with IRQ's going missing between domU and dom0.
>> >> > Unfortunately I have no data to prove this!
>> >> 
>> >> > With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel
>> >> > grub config, all 3 NIC's appear OK in the domU however I still had
>> >> > issues with the red-fone ISDN box. The interrupts were showing correctly
>> >> > (2000/s) in the domU but communication to the device via the NIC was
>> >> > still being interrupted (as shown in the asterisk console)Note that to
>> >> > get the igb driver to allow this many interrupts, the
>> >> > InterruptThrottleRate was set to 0. The same config (red-fone box,
>> >> > asterisk etc) works fine with a physical server.
>> >> 
>> >> > There is also the additional issue that I could not get the passthrough
>> >> > NIC's to show correctly when I also had a bridge setup.
>> >> 
>> >> > Throughout my testing however, I could not get the machine to crash.
>> >> 
>> >> > Not sure where to go with this one. For now we are keeping our VoIP
>> >> > servers physical when ISDN connections are required.
>> >> 
>> >> > Regards,
>> >> > Mark
>> >> 
>> >> > On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:
>> >> >> > 
>> >> >> > In my new test setup, I have seen some strange behaviour. 1 of the HVM's
>> >> >> > (with identical config in dom0 and domU) suddenly would not allow the
>> >> >> > igb driver to be loaded in domU, even though the device was visible in
>> >> >> 
>> >> >> Let's create a new thread for this other issue.
>> >> >> 
>> >> >> > lspci. Shutting the machine down, removing the power cord, waiting 5
>> >> >> > seconds then plugging it in again corrected that issue - Is this
>> >> >> > possibly a motherboard bug? I have also disabled the SR-IOV
>> >> >> > functionality in the BIOS incase this is causing any issues.
>> >> >> > 
>> >> >> > In addition, to try to correct the MSI issue noted above, I have changed
>> >> >> > my pci= line to the following:
>> >> >> > 
>> >> >> > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]
>> >> >> 
>> >> >> With the msi_translate=1 turned on the DomU HVM guests did work, right?
>> >> >> 
>> >> >> > 
>> >> >> > This has stopped the "already in use on device" log, and the devices
>> >> >> > appear to show correctly in the domU. Is it safe to disable
>> >> >> > msitranslate? as I understand it, its for allowing multifunction devices
>> >> >> > to be seen as such in domU. Is that correct?
>> >> >> > 
>> >> >> > I haven't been able to reproduce the dropped raid issue yet, but I am
>> >> >> > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
>> >> >> > this due to their very high interrupt usage (2000 per second).
>> >> >> 
>> >> >> OK.
>> >> >> > 
>> >> >> > In the mean time, I can see the following in the qemu-dm logs now with
>> >> >> > the msitranslate=0 enabled. Is it anything to worry about?
>> >> >> 
>> >> >> Well, the  "Error" ones are pretty bad, thought I am having a hard time
>> >> >> understanding what it means. Lets copy some of the QEMU folks on this.
>> >> >> 
>> >> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
>> >> >> > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
>> >> >> > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
>> >> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
>> >> >> > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
>> >> >> > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
>> >> >> > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
>> >> >> > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
>> >> >> > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
>> >> >> > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
>> >> >> > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
>> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
>> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
>> >> >> > 
>> >> >> > > 
>> >> >> > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
>> >> >> > > machine is toast. I mentioned in the previous email the key sequences - look on Google
>> >> >> > > on how to pass in SysRQ if you are using a serial concentrator.
>> >> >> > 
>> >> >> > I will do this when I can get the machine to crash.
>> >> >> > 
>> >> >> > Best Regards,
>> >> >> > Mark
>> >> >> > 
>> >> >> > _______________________________________________
>> >> >> > Xen-devel mailing list
>> >> >> > Xen-devel@lists.xensource.com
>> >> >> > http://lists.xensource.com/xen-devel
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> -- 
>> >> Best regards,
>> >>  Sander                            mailto:linux@eikelenboom.it
>> >> 
>> 
>> 
>> 
>> -- 
>> Best regards,
>>  Sander                            mailto:linux@eikelenboom.it
>> 
>> 
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users




-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: [Xen-users] Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Mark Adams]

On Wed, Dec 08, 2010 at 05:44:39PM +0100, Sander Eikelenboom wrote:
> Wednesday, December 8, 2010, 4:48:57 PM, you wrote:
> 
> > On Wed, Dec 08, 2010 at 03:05:50PM +0100, Sander Eikelenboom wrote:
> >> 
> >> Wednesday, December 8, 2010, 2:48:48 PM, you wrote:
> >> 
> >> > On Wed, Dec 08, 2010 at 02:37:15PM +0100, Sander Eikelenboom wrote:
> >> >> Hello Mark,
> >> 
> >> > Hi
> >> 
> >> >> 
> >> >> Just a recap:
> >> >>      you pass through:
> >> >>      - 3 physical nics/IGB
> >> >>      - 1 ISDN pci ISDN box
> >> 
> >> > The redfone box runs on 1 of the nics - its not seperate. It converts
> >> > ISDN to TDMoE see here.. http://www.red-fone.com/
> >> 
> >> So the problem is probably with the igb's.
> >> Searching showed http://forums.virtualbox.org/viewtopic.php?f=7&t=32171 , perhaps worth a try ?
> 
> > Tried this - doesn't help.
> 
> >> 
> >> Have you tried with just 1 IGB, and/or another simple 1gb NIC (non intel) to see if it's due to any of the special offload features ?
> 
> > Haven't got any other NIC's to try unfortunately. Even if it did work
> > with 1, it would be no use to me as I need 3.
> 
> I understand, but simplifying the setup and trying to isolate the problem, could clarify things.
> 
> I also read you previous thread, and i saw you hide the 02:00.0 and 03:00.0 with xen-pciback (e1000e driver) there, but now you seem to be passing through 08:00.0 and 08:00.1 (igb) ?
> So i assume you have already tried 2 different NIC's
> 
> http://download.intel.com/design/network/specupdt/82574.pdf though shows some errata regarding msi-x interrupts and timing issues and workarounds on the 82574 (02:00.0 and 03:00.0) nics.
> 

I was initially using the onboard NICs (e1000e) when I had the crashing
problem. To try to get around this, I disabled all the msi based stuff I
could find - which seemed to correct the crashing issue. In order to do
this I needed 3 NIC's because bridging would not work at the same time
as passthrough (would not show all devices being passed through?) hence
starting to use the igb based NIC card thats also in the machine.

Unfortunately the servers I've been testing on need to go in to
production now, so can't test any further (hence sticking the voip stuff
on to a physical box). Xen works really well for me when I don't use
pci-passthrough!

Regards,
Mark

> --
> Sander
> 
> 
> >> 
> >> 
> >> >>      - all using msi/msi-x interrupts ?
> >> 
> >> > I tried using msi/msi-x interrupts, but it caused the raid card to drop
> >> > off (after some use) and provided seemingly even worse performance than
> >> > pegging everything back to legacy.
> >> 
> >> >> 
> >> >> Have you tried using a PV domU instead of a HVM domU ?
> >> 
> >> > I initially tried PV but had issues with the igb NIC's. There was
> >> > another thread somewhere about my issues with that.
> >> 
> >> 
> >> >> Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?
> >> >> 
> >> >> 
> >> >> --
> >> >> Sander
> >> >> 
> >> >> 
> >> >> 
> >> >> Wednesday, December 8, 2010, 1:58:55 PM, you wrote:
> >> >> 
> >> >> > Hi - Apologies to top post this, but after alot of testing, I believe
> >> >> > there must be an issue with IRQ's going missing between domU and dom0.
> >> >> > Unfortunately I have no data to prove this!
> >> >> 
> >> >> > With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel
> >> >> > grub config, all 3 NIC's appear OK in the domU however I still had
> >> >> > issues with the red-fone ISDN box. The interrupts were showing correctly
> >> >> > (2000/s) in the domU but communication to the device via the NIC was
> >> >> > still being interrupted (as shown in the asterisk console)Note that to
> >> >> > get the igb driver to allow this many interrupts, the
> >> >> > InterruptThrottleRate was set to 0. The same config (red-fone box,
> >> >> > asterisk etc) works fine with a physical server.
> >> >> 
> >> >> > There is also the additional issue that I could not get the passthrough
> >> >> > NIC's to show correctly when I also had a bridge setup.
> >> >> 
> >> >> > Throughout my testing however, I could not get the machine to crash.
> >> >> 
> >> >> > Not sure where to go with this one. For now we are keeping our VoIP
> >> >> > servers physical when ISDN connections are required.
> >> >> 
> >> >> > Regards,
> >> >> > Mark
> >> >> 
> >> >> > On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:
> >> >> >> > 
> >> >> >> > In my new test setup, I have seen some strange behaviour. 1 of the HVM's
> >> >> >> > (with identical config in dom0 and domU) suddenly would not allow the
> >> >> >> > igb driver to be loaded in domU, even though the device was visible in
> >> >> >> 
> >> >> >> Let's create a new thread for this other issue.
> >> >> >> 
> >> >> >> > lspci. Shutting the machine down, removing the power cord, waiting 5
> >> >> >> > seconds then plugging it in again corrected that issue - Is this
> >> >> >> > possibly a motherboard bug? I have also disabled the SR-IOV
> >> >> >> > functionality in the BIOS incase this is causing any issues.
> >> >> >> > 
> >> >> >> > In addition, to try to correct the MSI issue noted above, I have changed
> >> >> >> > my pci= line to the following:
> >> >> >> > 
> >> >> >> > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]
> >> >> >> 
> >> >> >> With the msi_translate=1 turned on the DomU HVM guests did work, right?
> >> >> >> 
> >> >> >> > 
> >> >> >> > This has stopped the "already in use on device" log, and the devices
> >> >> >> > appear to show correctly in the domU. Is it safe to disable
> >> >> >> > msitranslate? as I understand it, its for allowing multifunction devices
> >> >> >> > to be seen as such in domU. Is that correct?
> >> >> >> > 
> >> >> >> > I haven't been able to reproduce the dropped raid issue yet, but I am
> >> >> >> > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
> >> >> >> > this due to their very high interrupt usage (2000 per second).
> >> >> >> 
> >> >> >> OK.
> >> >> >> > 
> >> >> >> > In the mean time, I can see the following in the qemu-dm logs now with
> >> >> >> > the msitranslate=0 enabled. Is it anything to worry about?
> >> >> >> 
> >> >> >> Well, the  "Error" ones are pretty bad, thought I am having a hard time
> >> >> >> understanding what it means. Lets copy some of the QEMU folks on this.
> >> >> >> 
> >> >> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
> >> >> >> > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
> >> >> >> > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
> >> >> >> > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
> >> >> >> > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
> >> >> >> > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
> >> >> >> > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
> >> >> >> > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
> >> >> >> > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
> >> >> >> > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
> >> >> >> > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> >> >> >> > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> >> >> >> > 
> >> >> >> > > 
> >> >> >> > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
> >> >> >> > > machine is toast. I mentioned in the previous email the key sequences - look on Google
> >> >> >> > > on how to pass in SysRQ if you are using a serial concentrator.
> >> >> >> > 
> >> >> >> > I will do this when I can get the machine to crash.
> >> >> >> > 
> >> >> >> > Best Regards,
> >> >> >> > Mark
> >> >> >> > 
> >> >> >> > _______________________________________________
> >> >> >> > Xen-devel mailing list
> >> >> >> > Xen-devel@lists.xensource.com
> >> >> >> > http://lists.xensource.com/xen-devel
> >> >> 
> >> >> 
> >> >> 
> >> >> 
> >> >> 
> >> >> -- 
> >> >> Best regards,
> >> >>  Sander                            mailto:linux@eikelenboom.it
> >> >> 
> >> 
> >> 
> >> 
> >> -- 
> >> Best regards,
> >>  Sander                            mailto:linux@eikelenboom.it
> >> 
> >> 
> >> _______________________________________________
> >> Xen-users mailing list
> >> Xen-users@lists.xensource.com
> >> http://lists.xensource.com/xen-users
> 
> 
> 
> 
> -- 
> Best regards,
>  Sander                            mailto:linux@eikelenboom.it
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

Re: [Xen-devel] Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Konrad Rzeszutek Wilk]

> > Just a recap:
> >      you pass through:
> >      - 3 physical nics/IGB
> >      - 1 ISDN pci ISDN box
> 
> The redfone box runs on 1 of the nics - its not seperate. It converts
> ISDN to TDMoE see here.. http://www.red-fone.com/
> 
> >      - all using msi/msi-x interrupts ?
> 
> I tried using msi/msi-x interrupts, but it caused the raid card to drop
> off (after some use) and provided seemingly even worse performance than
> pegging everything back to legacy.

Were you able to get a serial log and hit all of the differetn debug options
when the the RAID card died?
> 
> > 
> > Have you tried using a PV domU instead of a HVM domU ?
> 
> I initially tried PV but had issues with the igb NIC's. There was
> another thread somewhere about my issues with that.

Hmm, the only other thread I see from you is about the RAID.
> 
> 
> > Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?

Mr. Sander idea of isolating one piece by piece is the right way.

There is a bunch of warnings in the QEMU output - some of them quite .. troubling.

You seem to have issues with gntdev (as in, not found), but if you are using OpenSuSE
then that would work - I think. When you tested xen-unstable did you use the
OpenSUSE kernel or the PV-OPS one?

Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Sander Eikelenboom]

It seems Jeremy has had some troubles with the Intel 82754L as well.

Don't know if he has got those resolved, or has to use workarounds ?

http://www.gossamer-threads.com/lists/linux/kernel/1166308

What kernel are you using in your domU ? Also the 2.6.32 debian ?
Perhaps it's also worth a try to test a newer intel driver/kernel ?

--
Sander


Wednesday, December 8, 2010, 6:01:25 PM, you wrote:

>> > Just a recap:
>> >      you pass through:
>> >      - 3 physical nics/IGB
>> >      - 1 ISDN pci ISDN box
>> 
>> The redfone box runs on 1 of the nics - its not seperate. It converts
>> ISDN to TDMoE see here.. http://www.red-fone.com/
>> 
>> >      - all using msi/msi-x interrupts ?
>> 
>> I tried using msi/msi-x interrupts, but it caused the raid card to drop
>> off (after some use) and provided seemingly even worse performance than
>> pegging everything back to legacy.

> Were you able to get a serial log and hit all of the differetn debug options
> when the the RAID card died?
>> 
>> > 
>> > Have you tried using a PV domU instead of a HVM domU ?
>> 
>> I initially tried PV but had issues with the igb NIC's. There was
>> another thread somewhere about my issues with that.

> Hmm, the only other thread I see from you is about the RAID.
>> 
>> 
>> > Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?

> Mr. Sander idea of isolating one piece by piece is the right way.

> There is a bunch of warnings in the QEMU output - some of them quite .. troubling.

> You seem to have issues with gntdev (as in, not found), but if you are using OpenSuSE
> then that would work - I think. When you tested xen-unstable did you use the
> OpenSUSE kernel or the PV-OPS one?



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Jeremy Fitzhardinge]

On 12/08/2010 09:15 AM, Sander Eikelenboom wrote:
> It seems Jeremy has had some troubles with the Intel 82754L as well.
>
> Don't know if he has got those resolved, or has to use workarounds ?

Its basically a bug in that particular chip, I think.  The workaround is
to disable ASPM for it; I'm not sure if the upstream driver does that
yet, but the workaround used to be to completely disable ASPM
(pcie_aspm=off on the kernel command line, and in the BIOS).

    J

Re: [Xen-devel] Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Sander Eikelenboom]

And another thread with problems and strange irq reports with regard to the 82754
http://www.linuxquestions.org/questions/linux-hardware-18/intel-82574l-gigabit-network-card-issues-and-resolution-831364/

(please Mr. Konrad, no "Mr." that makes me feel really really old ;-) )

Wednesday, December 8, 2010, 6:01:25 PM, you wrote:

>> > Just a recap:
>> >      you pass through:
>> >      - 3 physical nics/IGB
>> >      - 1 ISDN pci ISDN box
>> 
>> The redfone box runs on 1 of the nics - its not seperate. It converts
>> ISDN to TDMoE see here.. http://www.red-fone.com/
>> 
>> >      - all using msi/msi-x interrupts ?
>> 
>> I tried using msi/msi-x interrupts, but it caused the raid card to drop
>> off (after some use) and provided seemingly even worse performance than
>> pegging everything back to legacy.

> Were you able to get a serial log and hit all of the differetn debug options
> when the the RAID card died?
>> 
>> > 
>> > Have you tried using a PV domU instead of a HVM domU ?
>> 
>> I initially tried PV but had issues with the igb NIC's. There was
>> another thread somewhere about my issues with that.

> Hmm, the only other thread I see from you is about the RAID.
>> 
>> 
>> > Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?

> Mr. Sander idea of isolating one piece by piece is the right way.

> There is a bunch of warnings in the QEMU output - some of them quite .. troubling.

> You seem to have issues with gntdev (as in, not found), but if you are using OpenSuSE
> then that would work - I think. When you tested xen-unstable did you use the
> OpenSUSE kernel or the PV-OPS one?



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: [Xen-devel] Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Konrad Rzeszutek Wilk]

On Wed, Dec 08, 2010 at 06:18:06PM +0100, Sander Eikelenboom wrote:
> And another thread with problems and strange irq reports with regard to the 82754
> http://www.linuxquestions.org/questions/linux-hardware-18/intel-82574l-gigabit-network-card-issues-and-resolution-831364/

Ugh, that device sure looks to have some faults.

So I think I've confused myself. There was another person who tried a similar
pass-through to an HVM guest of a sound card. While it worked it did not seem
to work that well and spitted out lots of warnings. But those are quite
different from what Mark had.

Mark, right now we are all busy trying to get patches ready for 2.6.38 so
hence the reason for not being so fast at responding to you or trying to
reproduce this on our machines.

The RAID is troubling, but the neat thing about it is that it hangs your
machine so if you hit all of those debug options via the serial console,
it can help us narrow down on where the problem is. The other issues you have
- well, there are many posibilities (and it might be very well the same issue
you are hitting with the RAID card) and narrowing it down to the exact cause
(say - it might be what Sander suggested - one of the NICs is just funky or perhaps
needs a firmware update) can help here.

The latest kernel is 2.6.32.26 (I think?) and the latest xen-unstable.hg has
some fixes to the MSI ownership and some IRQ migration issues fixed.

> 
> (please Mr. Konrad, no "Mr." that makes me feel really really old ;-) )

Sure thing :-)

Re: [Xen-devel] Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Mark Adams]

On Wed, Dec 08, 2010 at 12:01:25PM -0500, Konrad Rzeszutek Wilk wrote:
> > > Just a recap:
> > >      you pass through:
> > >      - 3 physical nics/IGB
> > >      - 1 ISDN pci ISDN box
> > 
> > The redfone box runs on 1 of the nics - its not seperate. It converts
> > ISDN to TDMoE see here.. http://www.red-fone.com/
> > 
> > >      - all using msi/msi-x interrupts ?
> > 
> > I tried using msi/msi-x interrupts, but it caused the raid card to drop
> > off (after some use) and provided seemingly even worse performance than
> > pegging everything back to legacy.
> 
> Were you able to get a serial log and hit all of the differetn debug options
> when the the RAID card died?
> > 

I didn't get it to crash - I have spent my limited time with the
hardware trying to get the setup to work effectively on the current
debian xen packages (hence disabling all the MSI stuff which seemed to
be the problem).

I understand that this isn't helpful in terms of getting whatever
bugs may be there fixed for the future however!

> > > 
> > > Have you tried using a PV domU instead of a HVM domU ?
> > 
> > I initially tried PV but had issues with the igb NIC's. There was
> > another thread somewhere about my issues with that.
> 
> Hmm, the only other thread I see from you is about the RAID.
> > 
> > 
> > > Have you tried passing through only the ISDN box, and let the network run with the xen backend/frontend to rule out the IGB/network stuff ?
> 
> Mr. Sander idea of isolating one piece by piece is the right way.
> 
> There is a bunch of warnings in the QEMU output - some of them quite .. troubling.
> 
> You seem to have issues with gntdev (as in, not found), but if you are using OpenSuSE
> then that would work - I think. When you tested xen-unstable did you use the
> OpenSUSE kernel or the PV-OPS one?

I'm running the Debian xen packages in squeeze (4.0.1-1 and 2.6.32-28) -
haven't stepped out of the packages at all.

Regards,
Mark

Re: Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.

[Dietmar Hahn]

[-- Attachment #1.1: Type: text/plain, Size: 6279 bytes --]

Hi,

Am 08.12.2010 schrieb "Mark Adams <mark@campbell-lange.net>":
> Hi - Apologies to top post this, but after alot of testing, I believe
> there must be an issue with IRQ's going missing between domU and dom0.
> Unfortunately I have no data to prove this!
> 
> With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel
> grub config, all 3 NIC's appear OK in the domU however I still had
> issues with the red-fone ISDN box. The interrupts were showing correctly
> (2000/s) in the domU but communication to the device via the NIC was
> still being interrupted (as shown in the asterisk console)Note that to
> get the igb driver to allow this many interrupts, the
> InterruptThrottleRate was set to 0. The same config (red-fone box,
> asterisk etc) works fine with a physical server.
> 
> There is also the additional issue that I could not get the passthrough
> NIC's to show correctly when I also had a bridge setup.
> 
> Throughout my testing however, I could not get the machine to crash.
> 
> Not sure where to go with this one. For now we are keeping our VoIP
> servers physical when ISDN connections are required.

Today I did some tests with xen-unstable and found these problems too.
I tried to passthrough 2 pci cards and got some error messages on the xen
xonsole and in the qemu logs.
With msitranslate=0 and pci=nomsi I got the soundcard working in a domU linux
but it doesn't help on windows.
I attached the logs from the xen serial console and the qemu logs.
Thanks!

Dietmar.

> 
> Regards,
> Mark
> 
> On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:
> > > 
> > > In my new test setup, I have seen some strange behaviour. 1 of the HVM's
> > > (with identical config in dom0 and domU) suddenly would not allow the
> > > igb driver to be loaded in domU, even though the device was visible in
> > 
> > Let's create a new thread for this other issue.
> > 
> > > lspci. Shutting the machine down, removing the power cord, waiting 5
> > > seconds then plugging it in again corrected that issue - Is this
> > > possibly a motherboard bug? I have also disabled the SR-IOV
> > > functionality in the BIOS incase this is causing any issues.
> > > 
> > > In addition, to try to correct the MSI issue noted above, I have changed
> > > my pci= line to the following:
> > > 
> > > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]
> > 
> > With the msi_translate=1 turned on the DomU HVM guests did work, right?
> > 
> > > 
> > > This has stopped the "already in use on device" log, and the devices
> > > appear to show correctly in the domU. Is it safe to disable
> > > msitranslate? as I understand it, its for allowing multifunction devices
> > > to be seen as such in domU. Is that correct?
> > > 
> > > I haven't been able to reproduce the dropped raid issue yet, but I am
> > > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause
> > > this due to their very high interrupt usage (2000 per second).
> > 
> > OK.
> > > 
> > > In the mean time, I can see the following in the qemu-dm logs now with
> > > the msitranslate=0 enabled. Is it anything to worry about?
> > 
> > Well, the  "Error" ones are pretty bad, thought I am having a hard time
> > understanding what it means. Lets copy some of the QEMU folks on this.
> > 
> > > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]
> > > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0
> > > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0
> > > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]
> > > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0
> > > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0
> > > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
> > > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61
> > > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69
> > > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71
> > > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79
> > > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> > > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.
> > > 
> > > > 
> > > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your
> > > > machine is toast. I mentioned in the previous email the key sequences - look on Google
> > > > on how to pass in SysRQ if you are using a serial concentrator.
> > > 
> > > I will do this when I can get the machine to crash.
> > > 
> > > Best Regards,
> > > Mark
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
> 
-- 
Company details: http://ts.fujitsu.com/imprint.html

[-- Attachment #1.2: Type: text/html, Size: 25276 bytes --]

[-- Attachment #2: xen-serial-log.txt --]
[-- Type: text/plain, Size: 7666 bytes --]


(XEN) tmem: all pools thawed for all domains
(XEN) tmem: all pools frozen for all domains
(XEN) tmem: all pools thawed for all domains
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=18 memflags=0 (3 of 4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=18 memflags=0 (0 of 3)
(XEN) [VT-D]iommu.c:1514: d0:PCIe: unmap bdf = 0:1b.0
(XEN) [VT-D]iommu.c:1387: d18:PCIe: map bdf = 0:1b.0
(XEN) [VT-D]io.c:303: d18: bind: m_gsi=0 g_gsi=32 device=4 intx=0
(XEN) [VT-D]iommu.c:1514: d0:PCIe: unmap bdf = 10:0.0
(XEN) HAHN-me_wifi_quirk: id 422c8086
(XEN) [VT-D]iommu.c:1387: d18:PCIe: map bdf = 10:0.0
(XEN) HAHN-me_wifi_quirk: id 422c8086
(XEN) irq.c:1511: dom18: pirq 0 or irq 40 already mapped
(XEN) [VT-D]io.c:303: d18: bind: m_gsi=0 g_gsi=36 device=5 intx=0
(XEN) HVM18: HVM Loader
(XEN) HVM18: Detected Xen v4.1-unstable
(XEN) HVM18: CPU speed is 2394 MHz
(XEN) HVM18: Xenbus rings @0xfeffc000, event channel 2
(XEN) irq.c:258: Dom18 PCI link 0 changed 0 -> 5
(XEN) HVM18: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:258: Dom18 PCI link 1 changed 0 -> 10
(XEN) HVM18: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:258: Dom18 PCI link 2 changed 0 -> 11
(XEN) HVM18: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:258: Dom18 PCI link 3 changed 0 -> 5
(XEN) HVM18: PCI-ISA link 3 routed to IRQ5
(XEN) HVM18: pci dev 01:2 INTD->IRQ5
(XEN) HVM18: pci dev 01:3 INTA->IRQ10
(XEN) HVM18: pci dev 03:0 INTA->IRQ5
(XEN) HVM18: pci dev 04:0 INTA->IRQ5
(XEN) HVM18: pci dev 05:0 INTA->IRQ10
(XEN) HVM18: pci dev 02:0 bar 10 size 02000000: f0000008
(XEN) HVM18: pci dev 03:0 bar 14 size 01000000: f2000008
(XEN) HVM18: pci dev 04:0 bar 10 size 00004000: f3000004
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=f2720 nr_mfns=4
(XEN) HVM18: pci dev 05:0 bar 10 size 00002000: f3004004
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3004 mfn=f2400 nr_mfns=2
(XEN) HVM18: pci dev 02:0 bar 14 size 00001000: f3006000
(XEN) HVM18: pci dev 03:0 bar 10 size 00000100: 0000c001
(XEN) HVM18: pci dev 01:2 bar 20 size 00000020: 0000c101
(XEN) HVM18: pci dev 01:1 bar 20 size 00000010: 0000c121
(XEN) HVM18: Multiprocessor initialisation:
(XEN) HVM18:  - CPU0 ... 36-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(XEN) HVM18: Testing HVM environment:
(XEN) HVM18:  - REP INSB across page boundaries ... passed
(XEN) HVM18:  - GS base MSRs and SWAPGS ... passed
(XEN) HVM18: Passed 2 of 2 tests
(XEN) HVM18: Writing SMBIOS tables ...
(XEN) HVM18: Loading ROMBIOS ...
(XEN) HVM18: 9660 bytes of ROMBIOS high-memory extensions:
(XEN) HVM18:   Relocating to 0xfc000000-0xfc0025bc ... done
(XEN) HVM18: Creating MP tables ...
(XEN) HVM18: Loading Cirrus VGABIOS ...
(XEN) HVM18: Loading ACPI ...
(XEN) HVM18:  - Lo data: 000ea020-000ea04f
(XEN) HVM18:  - Hi data: fc002800-fc01291f
(XEN) HVM18: vm86 TSS at fc012c00
(XEN) HVM18: BIOS map:
(XEN) HVM18:  c0000-c8fff: VGA BIOS
(XEN) HVM18:  eb000-eb158: SMBIOS tables
(XEN) HVM18:  f0000-fffff: Main BIOS
(XEN) HVM18: E820 table:
(XEN) HVM18:  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(XEN) HVM18:  [01]: 00000000:0009e000 - 00000000:0009fc00: RESERVED
(XEN) HVM18:  [02]: 00000000:0009fc00 - 00000000:000a0000: RESERVED
(XEN) HVM18:  HOLE: 00000000:000a0000 - 00000000:000e0000
(XEN) HVM18:  [03]: 00000000:000e0000 - 00000000:00100000: RESERVED
(XEN) HVM18:  [04]: 00000000:00100000 - 00000000:40000000: RAM
(XEN) HVM18:  HOLE: 00000000:40000000 - 00000000:fc000000
(XEN) HVM18:  [05]: 00000000:fc000000 - 00000001:00000000: RESERVED
(XEN) HVM18: Invoking ROMBIOS ...
(XEN) HVM18: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) stdvga.c:147:d18 entering stdvga and caching modes
(XEN) HVM18: VGABios $Id: vgabios.c,v 1.67 2008/01/27 09:44:12 vruppert Exp $
(XEN) HVM18: Bochs BIOS - build: 06/23/99
(XEN) HVM18: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) HVM18: Options: apmbios pcibios eltorito PMM 
(XEN) HVM18: 
(XEN) HVM18: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
(XEN) HVM18: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (8192 MBytes)
(XEN) HVM18: IDE time out
(XEN) HVM18: 
(XEN) HVM18: 
(XEN) HVM18: 
(XEN) HVM18: Press F12 for boot menu.
(XEN) HVM18: 
(XEN) HVM18: Booting from Hard Disk...
(XEN) HVM18: Booting from 0000:7c00
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=81
(XEN) HVM18: int13_harddisk: function 08, unmapped device for ELDL=81
(XEN) HVM18: *** int 15h function AX=00c0, BX=0000 not yet supported!
(XEN) HVM18: *** int 15h function AX=ec00, BX=0002 not yet supported!
(XEN) HVM18: KBD: unsupported int 16h function 03
(XEN) HVM18: *** int 15h function AX=e980, BX=0000 not yet supported!
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=81
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=81
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=82
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=82
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=83
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=83
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=84
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=84
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=85
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=85
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=86
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=86
(XEN) HVM18: int13_harddisk: function 41, unmapped device for ELDL=87
(XEN) HVM18: int13_harddisk: function 02, unmapped device for ELDL=87
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 88
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 88
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 89
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 89
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8a
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8a
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8b
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8b
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8c
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8c
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8d
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8d
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8e
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8e
(XEN) HVM18: int13_harddisk: function 41, ELDL out of range 8f
(XEN) HVM18: int13_harddisk: function 02, ELDL out of range 8f
(XEN) stdvga.c:151:d18 leaving stdvga
(XEN) vlapic.c:699:d18 Local APIC Write to read-only register 0x30
(XEN) vlapic.c:699:d18 Local APIC Write to read-only register 0x20
(XEN) vlapic.c:699:d18 Local APIC Write to read-only register 0x20
(XEN) irq.c:258: Dom18 PCI link 0 changed 5 -> 0
(XEN) irq.c:258: Dom18 PCI link 1 changed 10 -> 0
(XEN) irq.c:258: Dom18 PCI link 2 changed 11 -> 0
(XEN) irq.c:258: Dom18 PCI link 3 changed 5 -> 0
(XEN) [VT-D]io.c:327: d18: unbind: m_gsi=0 g_gsi=36 device=5 intx=0
(XEN) [VT-D]io.c:386: d18 unmap: m_irq=0 device=5 intx=0
(XEN) [VT-D]io.c:303: d18: bind: m_gsi=17 g_gsi=36 device=5 intx=0
(XEN) domctl.c:920:d0 pt_irq_create_bind failed!
(XEN) irq.c:1590: dom18: forcing unbind of pirq 0
(XEN) [VT-D]io.c:327: d18: unbind: m_gsi=0 g_gsi=32 device=4 intx=0
(XEN) irq.c:1856: dom18: pirq 0 not mapped
(XEN) [VT-D]io.c:386: d18 unmap: m_irq=0 device=4 intx=0
(XEN) [VT-D]io.c:303: d18: bind: m_gsi=16 g_gsi=32 device=4 intx=0
(XEN) domctl.c:920:d0 pt_irq_create_bind failed!
(XEN) irq.c:1856: dom18: pirq 0 not mapped


[-- Attachment #3: qemu-dm-OpenSuseHVM.log --]
[-- Type: text/x-log, Size: 4425 bytes --]

domid: 18
Strip off blktap sub-type prefix to /home/VMs/openSuse11.3.img (drv 'aio')
Using file /home/VMs/openSuse11.3.img in read-write mode
Watching /local/domain/0/device-model/18/logdirty/cmd
Watching /local/domain/0/device-model/18/command
Watching /local/domain/18/cpu
char device redirected to /dev/pts/2
qemu_map_cache_init nr_buckets = 10000 size 4194304
shared page at pfn feffd
buffered io page at pfn feffb
Guest uuid = 53580b33-0ad5-8aa1-5736-93a294fd0a6c
Time offset set 0
populating video RAM at ff000000
mapping video RAM from ff000000
Register xen platform.
Done register platform.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
xs_read(/local/domain/0/device-model/18/xen_extended_power_mgmt): read error
Log-dirty: no command yet.
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
dm-command: hot insert pass-through pci dev 
register_real_device: Assigning real physical device 00:1b.0 ...
register_real_device: Enable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init: No such device: setup io multiplexing failed! 0x0:0x1b.0x0
pt_register_regions: IO region registered (size=0x00004000 base_addr=0xf2720004)
pt_msi_setup: pt_msi_setup requested pirq = 0
pt_msi_setup: msi mapped with pirq 0
pci_intx: intx=1
register_real_device: Real physical device 00:1b.0 registered successfuly!
IRQ type = MSI-INTx
vcpu-set: watch node error.
xs_read(/local/domain/18/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/18/log-throttling'
medium change watch on `/local/domain/18/log-throttling' - unknown device, ignored
dm-command: hot insert pass-through pci dev 
register_real_device: Assigning real physical device 10:00.0 ...
register_real_device: Enable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init: No such device: setup io multiplexing failed! 0x10:0x0.0x0
pt_register_regions: IO region registered (size=0x00002000 base_addr=0xf2400004)
pt_msi_setup: pt_msi_setup requested pirq = 0
pt_msi_setup: msi mapped with pirq 0
pci_intx: intx=1
register_real_device: Real physical device 10:00.0 registered successfuly!
IRQ type = MSI-INTx
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
xen be core: xen be core: can't open gnttab device
can't open gnttab device
cirrus vga map change while on lfb mode
pt_iomem_map: e_phys=f3000000 maddr=f2720000 type=0 len=16384 index=0 first_map=1
pt_iomem_map: e_phys=f3004000 maddr=f2400000 type=0 len=8192 index=0 first_map=1
mapping vram to f0000000 - f0400000
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:04.0][Offset:30h][Length:4]
pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:30h][Length:4]
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 0 gvec 49 gflags 0
pt_msi_update: Error: Binding of MSI failed.
pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 28
pt_msgctrl_reg_write: setup msi for dev 28
pt_msi_setup: msi mapped with pirq 37
pt_msi_update: Update msi with pirq 37 gvec 49 gflags 0
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 0 gvec 59 gflags 0
pt_msi_update: Error: Binding of MSI failed.
pt_msi_update: Error: Unmapping of MSI failed.
pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 20
pt_msgctrl_reg_write: setup msi for dev 20
pt_msi_setup: msi mapped with pirq 36
pt_msi_update: Update msi with pirq 36 gvec 59 gflags 0

[-- Attachment #4: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Pasi Kärkkäinen]

On Mon, Nov 15, 2010 at 12:44:13PM -0500, Konrad Rzeszutek Wilk wrote:
> On Mon, Nov 15, 2010 at 05:23:09PM +0000, Mark Adams wrote:
> > On Mon, Nov 15, 2010 at 12:15:44PM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Sun, Nov 14, 2010 at 05:15:02PM +0000, Mark Adams wrote:
> > > > 
> > > > On 12 Nov 2010, at 22:22, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > > > 
> > > > > On Fri, Nov 12, 2010 at 05:10:58PM +0000, Mark Adams wrote:
> > > > >> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> > > > >>>> I've just noticed this at the end of xm dmesg
> > > > >>>> 
> > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > 
> > > Looking briefly at the code it means that somebody enabled the MSI
> > > already on the device and did not disable them. But I wonder how
> > > you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> > > or pciback.hide (for older kernels) to "hide" the devices away from the
> > > Linux Dom0 kernel?
> > 
> > using xen-pciback.hide as its a pvops kernel (debian squeeze
> > 2.6.32-5-27)
> 
> Ok. Then it might be worth looking in when this happens. I think
> there is an argument on the Xen hyperisor line to include the time-stamp, but
> I don't remember it :-(
> 

http://wiki.xen.org/xenwiki/XenHypervisorBootOptions

So I think it's "console_timestamps"


-- Pasi

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Mark Adams]

On Mon, Nov 15, 2010 at 12:44:13PM -0500, Konrad Rzeszutek Wilk wrote:
> On Mon, Nov 15, 2010 at 05:23:09PM +0000, Mark Adams wrote:
> > On Mon, Nov 15, 2010 at 12:15:44PM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Sun, Nov 14, 2010 at 05:15:02PM +0000, Mark Adams wrote:
> > > > 
> > > > On 12 Nov 2010, at 22:22, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > > > 
> > > > > On Fri, Nov 12, 2010 at 05:10:58PM +0000, Mark Adams wrote:
> > > > >> On Thu, Nov 11, 2010 at 02:06:58PM -0500, Konrad Rzeszutek Wilk wrote:
> > > > >>>> I've just noticed this at the end of xm dmesg
> > > > >>>> 
> > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > > >>>> (XEN) msi.c:715: MSI is already in use on device 02:00.0
> > > 
> > > Looking briefly at the code it means that somebody enabled the MSI
> > > already on the device and did not disable them. But I wonder how
> > > you got those in the first place. Did you use xen-pciback.hide (for PVOPS kernels)
> > > or pciback.hide (for older kernels) to "hide" the devices away from the
> > > Linux Dom0 kernel?
> > 
> > using xen-pciback.hide as its a pvops kernel (debian squeeze
> > 2.6.32-5-27)
> 
> Ok. Then it might be worth looking in when this happens. I think
> there is an argument on the Xen hyperisor line to include the time-stamp, but
> I don't remember it :-(
> 
> > > Didn't you say that you had two servers and saw this problem on another
> > > box too?
> > > 
> > > Without more details on the Xen hypervisor line or the kernel line when
> > > the failure occurs I sadly can't help you.
> > 
> > Yes this occurs on both servers that I've tried it on. Doesn't the MSI
> > log above indicate that there is a conflict - which is what ends up
> > causing the device to go offline? Is there no other way to identify the
> 
> Could be, but it is unclear - it depends on when the message pops out.
> 
> But that does not help with finding out why your RAID controller goes offline.

Stephan Austermuhle advises that nothing is logged via remote syslog
when this hang occurs. I'll reply on that thread to see if he can add
the additional logging

> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Konrad Rzeszutek Wilk]

> > Could be, but it is unclear - it depends on when the message pops out.
> > 
> > But that does not help with finding out why your RAID controller goes offline.
> 
> Stephan Austermuhle advises that nothing is logged via remote syslog
> when this hang occurs. I'll reply on that thread to see if he can add

He can also look at http://wiki.xensource.com/xenwiki/XenSerialConsole

> the additional logging

Pasi, is there a Wiki with this?:

When the hang happens, he needs to do two things:

 1) In the Linux kernel, hit SysRq-L, SysRQ-T

 2). Then  go in the hypervisor, hit Ctrl-A three times.  He should see a 
     prompt saying (XEN) ** Serial ...
     and hit '*' - that will collect all of the relevant information.

 3). Send the full serial log from the start of the machine to us (or in this
     case, to you Mark - or you can just CC him on this thread).

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Mark Adams]

Hi Stephan, please see debugging instructions from Konrad below.

Regards,
Mark

On Tue, Nov 16, 2010 at 11:04:22AM -0500, Konrad Rzeszutek Wilk wrote:
> > > Could be, but it is unclear - it depends on when the message pops out.
> > > 
> > > But that does not help with finding out why your RAID controller goes offline.
> > 
> > Stephan Austermuhle advises that nothing is logged via remote syslog
> > when this hang occurs. I'll reply on that thread to see if he can add
> 
> He can also look at http://wiki.xensource.com/xenwiki/XenSerialConsole
> 
> > the additional logging
> 
> Pasi, is there a Wiki with this?:
> 
> When the hang happens, he needs to do two things:
> 
>  1) In the Linux kernel, hit SysRq-L, SysRQ-T
> 
>  2). Then  go in the hypervisor, hit Ctrl-A three times.  He should see a 
>      prompt saying (XEN) ** Serial ...
>      and hit '*' - that will collect all of the relevant information.
> 
>  3). Send the full serial log from the start of the machine to us (or in this
>      case, to you Mark - or you can just CC him on this thread).

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Stephan Austermühle]

[-- Attachment #1.1: Type: text/plain, Size: 1119 bytes --]

Hello Mark!

Am 16.11.2010 17:47, schrieb Mark Adams:

>>> Stephan Austermuhle advises that nothing is logged via remote syslog
>>> when this hang occurs. I'll reply on that thread to see if he can add
>>
>> He can also look at http://wiki.xensource.com/xenwiki/XenSerialConsole
>>
>>> the additional logging
>>
>> Pasi, is there a Wiki with this?:
>>
>> When the hang happens, he needs to do two things:
>>
>>  1) In the Linux kernel, hit SysRq-L, SysRQ-T
>>
>>  2). Then  go in the hypervisor, hit Ctrl-A three times.  He should see a 
>>      prompt saying (XEN) ** Serial ...
>>      and hit '*' - that will collect all of the relevant information.
>>
>>  3). Send the full serial log from the start of the machine to us (or in this
>>      case, to you Mark - or you can just CC him on this thread).

Thanks for your support.

The server is far away from me (some hundred kilometers) with no chance
to connect a serial console. The only thing that I have access to is a
network console (kind of iLO). Is it sufficient to collect additional
debug data?

Best regards,

Stephan



[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5179 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Pasi Kärkkäinen]

On Thu, Nov 18, 2010 at 09:42:32AM +0100, Stephan Austermühle wrote:
> Hello Mark!
> 
> Am 16.11.2010 17:47, schrieb Mark Adams:
> 
> >>> Stephan Austermuhle advises that nothing is logged via remote syslog
> >>> when this hang occurs. I'll reply on that thread to see if he can add
> >>
> >> He can also look at http://wiki.xensource.com/xenwiki/XenSerialConsole
> >>
> >>> the additional logging
> >>
> >> Pasi, is there a Wiki with this?:
> >>
> >> When the hang happens, he needs to do two things:
> >>
> >>  1) In the Linux kernel, hit SysRq-L, SysRQ-T
> >>
> >>  2). Then  go in the hypervisor, hit Ctrl-A three times.  He should see a 
> >>      prompt saying (XEN) ** Serial ...
> >>      and hit '*' - that will collect all of the relevant information.
> >>
> >>  3). Send the full serial log from the start of the machine to us (or in this
> >>      case, to you Mark - or you can just CC him on this thread).
> 
> Thanks for your support.
> 
> The server is far away from me (some hundred kilometers) with no chance
> to connect a serial console. The only thing that I have access to is a
> network console (kind of iLO). Is it sufficient to collect additional
> debug data?
> 

If it's SOL (Serial Over LAN), then yes, that's enough.

See: http://wiki.xensource.com/xenwiki/XenSerialConsole

-- Pasi

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Stephan Austermühle]

[-- Attachment #1.1: Type: text/plain, Size: 458 bytes --]

Hi Pasi,

Am 18.11.2010 09:45, schrieb Pasi Kärkkäinen:

>> The server is far away from me (some hundred kilometers) with no chance
>> to connect a serial console. The only thing that I have access to is a
>> network console (kind of iLO). Is it sufficient to collect additional
>> debug data?
> 
> If it's SOL (Serial Over LAN), then yes, that's enough.
> 
> See: http://wiki.xensource.com/xenwiki/XenSerialConsole

I'll check.

Stephan


[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5179 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: [Xen-users] Re: pci-passthrough in pvops causing offline raid

[Pasi Kärkkäinen]

On Tue, Nov 16, 2010 at 11:04:22AM -0500, Konrad Rzeszutek Wilk wrote:
> > > Could be, but it is unclear - it depends on when the message pops out.
> > > 
> > > But that does not help with finding out why your RAID controller goes offline.
> > 
> > Stephan Austermuhle advises that nothing is logged via remote syslog
> > when this hang occurs. I'll reply on that thread to see if he can add
> 
> He can also look at http://wiki.xensource.com/xenwiki/XenSerialConsole
> 
> > the additional logging
> 
> Pasi, is there a Wiki with this?:
>

I don't think we have this in the wiki.. at least I haven't seen one.. 

-- Pasi
 
> When the hang happens, he needs to do two things:
> 
>  1) In the Linux kernel, hit SysRq-L, SysRQ-T
> 
>  2). Then  go in the hypervisor, hit Ctrl-A three times.  He should see a 
>      prompt saying (XEN) ** Serial ...
>      and hit '*' - that will collect all of the relevant information.
> 
>  3). Send the full serial log from the start of the machine to us (or in this
>      case, to you Mark - or you can just CC him on this thread).

Re: pci-passthrough in pvops causing offline raid

[Konrad Rzeszutek Wilk]

> > > It is an Areca 1220. I can't see anything when the device goes offline
> > > apart from 
> > > 
> > >     [77324.264270] sd 0:0:0:1: rejecting I/O to offline device
> > >     [77334.005854] sd 0:0:0:0: rejecting I/O to offline device
> > 
> > That is it? No other details from the driver? Did you poke at the driver (modinfo)
> > to see if there are any options to increase its verbosity.
> 
> I can't do anything once its happened, everything is offline so I have
> no utils...

An easy is to use netconsole. You can make all of the kernel log output
got a different machine on your network.

> > 
> > > 
> > > Unfortunately nothing get's logged because there is nothing to write to
> > > anymore. I'm not sure how I can see the IRQs otherwise. There is no
> > 
> > cat /proc/interrupts
> > 
> > > pinning being done at all, and the machine was running for a few months
> > > OK before the pciback was added.
> > 
> > Ok, what about your NICs? Are they on-board? Are they sharing the IRQ
> > with the card? You should be able to see this by looking at /proc/interrupts.
> > Which NICs are they? lspci can you help you there. As of matter of fact, run
> > lspci -vvv and send that.
> 
> It is the onboard nics, they are Intel 82574L. I can see the arcmsr
> line, but not anything for the NICS (because they are hidden?)

Your lspci tells me it is on 16 and 17. You should see in /proc/interrupts
on that line something about pciback?
> 
> 39:    1126249          0          0          0          0          0         0          0  xen-pirq-ioapic-level  arcmsr
> 
> Nothing else is on 1126249

You mean IRQ 39.
> 
> see lspci.txt attached.

thanks.
> > When this hang occurs, can you do 'xm debug-key Q', 'xm debug-key i', 'xm debug-key z'.
> > Then run 'xm dmesg' and provide that to me?
> 
> I can try this, but It probably won't work as the device is will not be
> readable.

Look on Google for 'Wiki PVOPS' and there is a section on how to connect a serial console.
With the serial console we can send those commands to the hypervisor even if your box
is hanged.

http://wiki.xen.org/xenwiki/XenSerialConsole

> > 
> > Is your boot disk on the same disk as the RAID?
> 
> There are 2 raids, a Raid1 for the OS (/boot / /var /tmp /usr) and a
> raid5 for VM's - They both dissapear at the same time so it appears the
> card is dissapearing..
> 

I wonder if we have your IRQs confused. Can you provide the full cat /proc/interrupts
and as well the serial bootup of the console? Or just the 'xm dmesg' and 'dmesg' output
if you don't have the serial console hooked up yet.

Re: Re: [Xen-devel] pci-passthrough in pvops causing offline raid

[Richie]

On 11/11/2010 11:53 AM, Konrad Rzeszutek Wilk wrote:
> On Thu, Nov 11, 2010 at 10:24:17AM +0000, Mark Adams wrote:
>    
>> Hi All,
>>
>> Running xen 4.0.1-rc6, debian squeeze 2.6.32-21.
>>
>> In a voip setup, where I have forwarded the onboard NIC interfaces
>> through to domU using the following grub config:
>>
>> module  /vmlinuz-2.6.32-5-xen-amd64 placeholder root=UUID=25c3ac79-6850-498d-afcf-ea42970e94fd ro  quiet xen-pciback.permissive xen-pciback.hide=(02:00.0)(03:00.0) pci=resource_alignment=02:00.0;03:00.0
>>
>> I'm having a serious issue where the raid card goes offline after an
>> indefinate period of time. Sometimes runs fine for a week, other times 1
>> day before I get "offline device" errors. Rebooting the machine fixes it
>> straight away, and everything is back online.
>>
>> What in the Xen pciback is causing the raid card to go offline? The
>> only devices hidden are the 2 onboard NIC's.
>>      
> You need to give more details. Is the RAID card a 3Ware? An LSI? Do you
> run with an IOMMU? When the RAID card goes offline, do you see a stop of
> IRQs going to the device? Are the IRQs for the RAID card sent to all of your
> CPUs or just a specific one? Are you pinning your guests to specific CPUs?
> Does the issue disappear if you don't passthrough the NIC interfaces? If so have
> you run this setup for "a week" to make sure?
>    
>> I know that this issue is with Xen, as I had this running on a different
>> server (same xen setup) and it had the same issues, which I initially
>> thought were to do with the raid card.
>>      
> So you never ran this setup on this kernel (2.6.32-5) without the Xen hypervisor?
>
>    
>> Is there known issues in this kernel and xen version with pciback? I'm
>>      
> No. It all works perfectly :-)
>
>    
>> going to update to the current package versions this evening (4.0.1-1
>> and 2.6.32-27) however would appreciate if anyone has any other insight
>> into this issue, or even just a note to say it is a bug that has been
>> fixed in current versions!
>>      
> Well, there were issues with the LSI cards having a hidden PCI device. But those
> are pretty obvious as you can't even use it correctly. There is also
> a problem with 3Ware 9506 IDE card - which on my box stops sending IRQs
> on the IOAPIC it has been assigned (28) and instead uses another one (17).
> Not sure if this is just the PCI card using the wrong PCI interrupt pin on the
> card and it ends up poking the wrong IOAPIC.
>
>    

Note:  I have no idea if this would be related to your issue or that my 
assessment is completely accurate.

I had an issue that I feel the debian squeeze kernel running under domU 
played a part in.  My dom0 is 2.6.34.7 Xenified w/Andrew lyon's patches 
and I running Xen 4.0.2-pre (xen testing). I passthrough a pci tuner 
card but have not considered that this could also contribute.

Sometimes when I shutdown the domU and upon halt I started getting 
libata style DRIVE_NOT_READY errors in my dom0.  Either one drive would 
drop from my mdadm raid (which houses my lvm filesystems including root 
for dom0 and domU) or perhaps they would drop and cause a panic.  A 
reboot fixes everything though a rebuild would occur.   I was not able 
to capture those errors in the few times it happened, but  I have since 
changed to use a 2.6.31 pvops kernel from jeremy's stable branch in my 
domU and I have yet to reproduce the issue.  I did note that it might 
take a number of days for the problem to manifest and so far I've tested 
a domU shutdown after 24 and 72 hours using the new kernel with no 
issues.  My next test is @ 7 days.

I wish I had more information myself, but I don't.  Regardless of the 
accuracy of this claim, I recommend trying other kernels to see if the 
problem persists.