All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
@ 2014-01-07 11:26 Wu, Feng
  2014-01-07 11:35 ` Gordan Bobic
  0 siblings, 1 reply; 40+ messages in thread
From: Wu, Feng @ 2014-01-07 11:26 UTC (permalink / raw)
  To: Gordan Bobic, Andrew Cooper; +Cc: xen-devel



> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org
> [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Gordan Bobic
> Sent: Tuesday, January 07, 2014 6:44 PM
> To: Andrew Cooper
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Multi-bridged PCIe devices (Was: Re: iommuu/vt-d
> issues with LSI MegaSAS (PERC5i))
> 
> On 2014-01-07 10:38, Andrew Cooper wrote:
> > On 07/01/14 10:35, Gordan Bobic wrote:
> >> On 2014-01-07 03:17, Zhang, Yang Z wrote:
> >>> Konrad Rzeszutek Wilk wrote on 2014-01-07:
> >>>>> Which would look like this:
> >>>>>
> >>>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
> >>>>> on the card
> >>>>>           \--------------> IEEE-1394a
> >>>>>
> >>>>> I am actually wondering if this 07:00.0 device is the one that
> >>>>> reports itself as 08:00.0 (which I think is what you alluding to
> >>>>> Jan)
> >>>>>
> >>>>
> >>>> And to double check that theory I decided to pass in the IEEE-1394a
> >>>> to a guest:
> >>>>
> >>>>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
> >>>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >>>>
> >>>>
> >>>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
> >>>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
> >>>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0]
> >>>> fault
> >>>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
> >>>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
> >>>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)
> >>>> root_entry
> >>>> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
> >>>> context
> >>>> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)
> >>>> ctxt_entry[0]
> >>>> not present
> >>>>
> >>>> So, capture card OK - Likely the Tundra bridge has an issue:
> >>>>
> >>>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
> >>>> (prog-if 01 [Subtractive decode])
> >>>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV-
> VGASnoop-
> >>>>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+
> >>>> 66MHz-
> >>>>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort+
> >>>>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07,
> >>>> secondary=08,
> >>>>         subordinate=08, sec-latency=32 Memory behind bridge:
> >>>>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
> >>>>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> >>>> BridgeCtl:
> >>>>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> >>>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat-
> DiscTmrSERREn-
> >>>>         Capabilities: [60] Subsystem: Super Micro Computer Inc
> >>>> Device 0805
> >>>>         Capabilities: [a0] Power Management version 3
> >>>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> >>>>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0
> >>>> NoSoftRst+
> >>>>                 PME-Enable- DSel=0 DScale=0 PME-
> >>>>
> >>>> or there is some unknown bridge in the motherboard.
> >>>
> >>> According your description above, the upstream Linux should also have
> >>> the same problem. Did you see it with upstream Linux?
> >>
> >> The problem I was seeing with LSI cards (phantom device doing DMA)
> >> does, indeed, also occur in upstream Linux. If I enable intel-iommu on
> >> bare metal Linux, the same problem occurs as with Xen.
> >>
> >>> There may be some buggy device that generate DMA request with
> >>> internal
> >>> BDF but it didn't expose it(not like Phantom device). For those
> >>> devices, I think we need to setup the VT-d page table manually.
> >>
> >> I think what is needed is a pci-phantom style override that tells the
> >> hypervisor to tell the IOMMU to allow DMA traffic from a specific
> >> invisible device ID.
> >>
> >> Gordan
> >
> > There is.  See "pci-phantom" in
> > http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
> 
> I thought this was only applicable to phantom _functions_ (number after
> the
> dot) rather than whole phantom _devices_. Is that not the case?

I think that's right. I go through the related code for the pci phantom device just now, I find that
the information of command line 'pci-phantom' is stored in variable ' phantom_devs[8] '
with type of s truct phantom_dev{}. This variable is used in function alloc_pdev() as follow:


                for ( i = 0; i < nr_phantom_devs; ++i )
                    if ( phantom_devs[i].seg == pseg->nr &&
                         phantom_devs[i].bus == bus &&
                         phantom_devs[i].slot == PCI_SLOT(devfn) &&
                         phantom_devs[i].stride > PCI_FUNC(devfn) )
                    {
                        pdev->phantom_stride = phantom_devs[i].stride;
                        break;
                    }

So from the code, we can see this command line only works for phantom _function_, not for whole phantom _devices_.


> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Thanks,
Feng

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07 11:26 Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i)) Wu, Feng
@ 2014-01-07 11:35 ` Gordan Bobic
  2014-01-07 12:15   ` Jan Beulich
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2014-01-07 11:35 UTC (permalink / raw)
  To: Wu, Feng; +Cc: Andrew Cooper, xen-devel

On 2014-01-07 11:26, Wu, Feng wrote:
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xen.org
>> [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Gordan Bobic
>> Sent: Tuesday, January 07, 2014 6:44 PM
>> To: Andrew Cooper
>> Cc: xen-devel@lists.xen.org
>> Subject: Re: [Xen-devel] Multi-bridged PCIe devices (Was: Re: 
>> iommuu/vt-d
>> issues with LSI MegaSAS (PERC5i))
>> 
>> On 2014-01-07 10:38, Andrew Cooper wrote:
>> > On 07/01/14 10:35, Gordan Bobic wrote:
>> >> On 2014-01-07 03:17, Zhang, Yang Z wrote:
>> >>> Konrad Rzeszutek Wilk wrote on 2014-01-07:
>> >>>>> Which would look like this:
>> >>>>>
>> >>>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
>> >>>>> on the card
>> >>>>>           \--------------> IEEE-1394a
>> >>>>>
>> >>>>> I am actually wondering if this 07:00.0 device is the one that
>> >>>>> reports itself as 08:00.0 (which I think is what you alluding to
>> >>>>> Jan)
>> >>>>>
>> >>>>
>> >>>> And to double check that theory I decided to pass in the IEEE-1394a
>> >>>> to a guest:
>> >>>>
>> >>>>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
>> >>>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>> >>>>
>> >>>>
>> >>>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
>> >>>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
>> >>>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0]
>> >>>> fault
>> >>>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
>> >>>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
>> >>>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)
>> >>>> root_entry
>> >>>> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
>> >>>> context
>> >>>> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)
>> >>>> ctxt_entry[0]
>> >>>> not present
>> >>>>
>> >>>> So, capture card OK - Likely the Tundra bridge has an issue:
>> >>>>
>> >>>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
>> >>>> (prog-if 01 [Subtractive decode])
>> >>>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV-
>> VGASnoop-
>> >>>>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+
>> >>>> 66MHz-
>> >>>>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
>> <MAbort+
>> >>>>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07,
>> >>>> secondary=08,
>> >>>>         subordinate=08, sec-latency=32 Memory behind bridge:
>> >>>>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
>> >>>>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
>> >>>> BridgeCtl:
>> >>>>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>> >>>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat-
>> DiscTmrSERREn-
>> >>>>         Capabilities: [60] Subsystem: Super Micro Computer Inc
>> >>>> Device 0805
>> >>>>         Capabilities: [a0] Power Management version 3
>> >>>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> >>>>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0
>> >>>> NoSoftRst+
>> >>>>                 PME-Enable- DSel=0 DScale=0 PME-
>> >>>>
>> >>>> or there is some unknown bridge in the motherboard.
>> >>>
>> >>> According your description above, the upstream Linux should also have
>> >>> the same problem. Did you see it with upstream Linux?
>> >>
>> >> The problem I was seeing with LSI cards (phantom device doing DMA)
>> >> does, indeed, also occur in upstream Linux. If I enable intel-iommu on
>> >> bare metal Linux, the same problem occurs as with Xen.
>> >>
>> >>> There may be some buggy device that generate DMA request with
>> >>> internal
>> >>> BDF but it didn't expose it(not like Phantom device). For those
>> >>> devices, I think we need to setup the VT-d page table manually.
>> >>
>> >> I think what is needed is a pci-phantom style override that tells the
>> >> hypervisor to tell the IOMMU to allow DMA traffic from a specific
>> >> invisible device ID.
>> >>
>> >> Gordan
>> >
>> > There is.  See "pci-phantom" in
>> > http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
>> 
>> I thought this was only applicable to phantom _functions_ (number 
>> after
>> the
>> dot) rather than whole phantom _devices_. Is that not the case?
> 
> I think that's right. I go through the related code for the pci
> phantom device just now, I find that
> the information of command line 'pci-phantom' is stored in variable '
> phantom_devs[8] '
> with type of s truct phantom_dev{}. This variable is used in function
> alloc_pdev() as follow:
> 
> 
>                 for ( i = 0; i < nr_phantom_devs; ++i )
>                     if ( phantom_devs[i].seg == pseg->nr &&
>                          phantom_devs[i].bus == bus &&
>                          phantom_devs[i].slot == PCI_SLOT(devfn) &&
>                          phantom_devs[i].stride > PCI_FUNC(devfn) )
>                     {
>                         pdev->phantom_stride = phantom_devs[i].stride;
>                         break;
>                     }
> 
> So from the code, we can see this command line only works for phantom
> _function_, not for whole phantom _devices_.

What would it take to make it work for a whole phantom device?

Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07 11:35 ` Gordan Bobic
@ 2014-01-07 12:15   ` Jan Beulich
  2014-01-07 12:42     ` Gordan Bobic
  0 siblings, 1 reply; 40+ messages in thread
From: Jan Beulich @ 2014-01-07 12:15 UTC (permalink / raw)
  To: Gordan Bobic, Feng Wu; +Cc: Andrew Cooper, xen-devel

>>> On 07.01.14 at 12:35, Gordan Bobic <gordan@bobich.net> wrote:
> On 2014-01-07 11:26, Wu, Feng wrote:
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xen.org 
>>> [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Gordan Bobic
>>> Sent: Tuesday, January 07, 2014 6:44 PM
>>> To: Andrew Cooper
>>> Cc: xen-devel@lists.xen.org 
>>> Subject: Re: [Xen-devel] Multi-bridged PCIe devices (Was: Re: 
>>> iommuu/vt-d
>>> issues with LSI MegaSAS (PERC5i))
>>> 
>>> On 2014-01-07 10:38, Andrew Cooper wrote:
>>> > On 07/01/14 10:35, Gordan Bobic wrote:
>>> >> On 2014-01-07 03:17, Zhang, Yang Z wrote:
>>> >>> Konrad Rzeszutek Wilk wrote on 2014-01-07:
>>> >>>>> Which would look like this:
>>> >>>>>
>>> >>>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
>>> >>>>> on the card
>>> >>>>>           \--------------> IEEE-1394a
>>> >>>>>
>>> >>>>> I am actually wondering if this 07:00.0 device is the one that
>>> >>>>> reports itself as 08:00.0 (which I think is what you alluding to
>>> >>>>> Jan)
>>> >>>>>
>>> >>>>
>>> >>>> And to double check that theory I decided to pass in the IEEE-1394a
>>> >>>> to a guest:
>>> >>>>
>>> >>>>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
>>> >>>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>>> >>>>
>>> >>>>
>>> >>>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
>>> >>>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
>>> >>>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0]
>>> >>>> fault
>>> >>>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
>>> >>>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
>>> >>>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)
>>> >>>> root_entry
>>> >>>> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
>>> >>>> context
>>> >>>> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)
>>> >>>> ctxt_entry[0]
>>> >>>> not present
>>> >>>>
>>> >>>> So, capture card OK - Likely the Tundra bridge has an issue:
>>> >>>>
>>> >>>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
>>> >>>> (prog-if 01 [Subtractive decode])
>>> >>>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV-
>>> VGASnoop-
>>> >>>>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+
>>> >>>> 66MHz-
>>> >>>>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
>>> <MAbort+
>>> >>>>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07,
>>> >>>> secondary=08,
>>> >>>>         subordinate=08, sec-latency=32 Memory behind bridge:
>>> >>>>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
>>> >>>>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
>>> >>>> BridgeCtl:
>>> >>>>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>>> >>>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat-
>>> DiscTmrSERREn-
>>> >>>>         Capabilities: [60] Subsystem: Super Micro Computer Inc
>>> >>>> Device 0805
>>> >>>>         Capabilities: [a0] Power Management version 3
>>> >>>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>> >>>>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0
>>> >>>> NoSoftRst+
>>> >>>>                 PME-Enable- DSel=0 DScale=0 PME-
>>> >>>>
>>> >>>> or there is some unknown bridge in the motherboard.
>>> >>>
>>> >>> According your description above, the upstream Linux should also have
>>> >>> the same problem. Did you see it with upstream Linux?
>>> >>
>>> >> The problem I was seeing with LSI cards (phantom device doing DMA)
>>> >> does, indeed, also occur in upstream Linux. If I enable intel-iommu on
>>> >> bare metal Linux, the same problem occurs as with Xen.
>>> >>
>>> >>> There may be some buggy device that generate DMA request with
>>> >>> internal
>>> >>> BDF but it didn't expose it(not like Phantom device). For those
>>> >>> devices, I think we need to setup the VT-d page table manually.
>>> >>
>>> >> I think what is needed is a pci-phantom style override that tells the
>>> >> hypervisor to tell the IOMMU to allow DMA traffic from a specific
>>> >> invisible device ID.
>>> >>
>>> >> Gordan
>>> >
>>> > There is.  See "pci-phantom" in
>>> > http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html 
>>> 
>>> I thought this was only applicable to phantom _functions_ (number 
>>> after
>>> the
>>> dot) rather than whole phantom _devices_. Is that not the case?
>> 
>> I think that's right. I go through the related code for the pci
>> phantom device just now, I find that
>> the information of command line 'pci-phantom' is stored in variable '
>> phantom_devs[8] '
>> with type of s truct phantom_dev{}. This variable is used in function
>> alloc_pdev() as follow:
>> 
>> 
>>                 for ( i = 0; i < nr_phantom_devs; ++i )
>>                     if ( phantom_devs[i].seg == pseg->nr &&
>>                          phantom_devs[i].bus == bus &&
>>                          phantom_devs[i].slot == PCI_SLOT(devfn) &&
>>                          phantom_devs[i].stride > PCI_FUNC(devfn) )
>>                     {
>>                         pdev->phantom_stride = phantom_devs[i].stride;
>>                         break;
>>                     }
>> 
>> So from the code, we can see this command line only works for phantom
>> _function_, not for whole phantom _devices_.
> 
> What would it take to make it work for a whole phantom device?

First and foremost a definition of what a phantom device is and
how one would behave. Once again - phantom functions are part
of the PCIe specification, so those don't require a definition.

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07 12:15   ` Jan Beulich
@ 2014-01-07 12:42     ` Gordan Bobic
  2014-01-07 14:38       ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2014-01-07 12:42 UTC (permalink / raw)
  To: Jan Beulich; +Cc: Andrew Cooper, Feng Wu, xen-devel

On 2014-01-07 12:15, Jan Beulich wrote:
>>>> On 07.01.14 at 12:35, Gordan Bobic <gordan@bobich.net> wrote:
>> On 2014-01-07 11:26, Wu, Feng wrote:
>>>> -----Original Message-----
>>>> From: xen-devel-bounces@lists.xen.org
>>>> [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Gordan Bobic
>>>> Sent: Tuesday, January 07, 2014 6:44 PM
>>>> To: Andrew Cooper
>>>> Cc: xen-devel@lists.xen.org
>>>> Subject: Re: [Xen-devel] Multi-bridged PCIe devices (Was: Re:
>>>> iommuu/vt-d
>>>> issues with LSI MegaSAS (PERC5i))
>>>> 
>>>> On 2014-01-07 10:38, Andrew Cooper wrote:
>>>> > On 07/01/14 10:35, Gordan Bobic wrote:
>>>> >> On 2014-01-07 03:17, Zhang, Yang Z wrote:
>>>> >>> Konrad Rzeszutek Wilk wrote on 2014-01-07:
>>>> >>>>> Which would look like this:
>>>> >>>>>
>>>> >>>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
>>>> >>>>> on the card
>>>> >>>>>           \--------------> IEEE-1394a
>>>> >>>>>
>>>> >>>>> I am actually wondering if this 07:00.0 device is the one that
>>>> >>>>> reports itself as 08:00.0 (which I think is what you alluding to
>>>> >>>>> Jan)
>>>> >>>>>
>>>> >>>>
>>>> >>>> And to double check that theory I decided to pass in the IEEE-1394a
>>>> >>>> to a guest:
>>>> >>>>
>>>> >>>>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
>>>> >>>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>>>> >>>>
>>>> >>>>
>>>> >>>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
>>>> >>>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
>>>> >>>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0]
>>>> >>>> fault
>>>> >>>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
>>>> >>>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
>>>> >>>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)
>>>> >>>> root_entry
>>>> >>>> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
>>>> >>>> context
>>>> >>>> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)
>>>> >>>> ctxt_entry[0]
>>>> >>>> not present
>>>> >>>>
>>>> >>>> So, capture card OK - Likely the Tundra bridge has an issue:
>>>> >>>>
>>>> >>>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
>>>> >>>> (prog-if 01 [Subtractive decode])
>>>> >>>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV-
>>>> VGASnoop-
>>>> >>>>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+
>>>> >>>> 66MHz-
>>>> >>>>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
>>>> <MAbort+
>>>> >>>>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07,
>>>> >>>> secondary=08,
>>>> >>>>         subordinate=08, sec-latency=32 Memory behind bridge:
>>>> >>>>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
>>>> >>>>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
>>>> >>>> BridgeCtl:
>>>> >>>>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>>>> >>>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat-
>>>> DiscTmrSERREn-
>>>> >>>>         Capabilities: [60] Subsystem: Super Micro Computer Inc
>>>> >>>> Device 0805
>>>> >>>>         Capabilities: [a0] Power Management version 3
>>>> >>>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> >>>>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0
>>>> >>>> NoSoftRst+
>>>> >>>>                 PME-Enable- DSel=0 DScale=0 PME-
>>>> >>>>
>>>> >>>> or there is some unknown bridge in the motherboard.
>>>> >>>
>>>> >>> According your description above, the upstream Linux should also have
>>>> >>> the same problem. Did you see it with upstream Linux?
>>>> >>
>>>> >> The problem I was seeing with LSI cards (phantom device doing DMA)
>>>> >> does, indeed, also occur in upstream Linux. If I enable intel-iommu on
>>>> >> bare metal Linux, the same problem occurs as with Xen.
>>>> >>
>>>> >>> There may be some buggy device that generate DMA request with
>>>> >>> internal
>>>> >>> BDF but it didn't expose it(not like Phantom device). For those
>>>> >>> devices, I think we need to setup the VT-d page table manually.
>>>> >>
>>>> >> I think what is needed is a pci-phantom style override that tells the
>>>> >> hypervisor to tell the IOMMU to allow DMA traffic from a specific
>>>> >> invisible device ID.
>>>> >>
>>>> >> Gordan
>>>> >
>>>> > There is.  See "pci-phantom" in
>>>> > http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
>>>> 
>>>> I thought this was only applicable to phantom _functions_ (number
>>>> after
>>>> the
>>>> dot) rather than whole phantom _devices_. Is that not the case?
>>> 
>>> I think that's right. I go through the related code for the pci
>>> phantom device just now, I find that
>>> the information of command line 'pci-phantom' is stored in variable '
>>> phantom_devs[8] '
>>> with type of s truct phantom_dev{}. This variable is used in function
>>> alloc_pdev() as follow:
>>> 
>>> 
>>>                 for ( i = 0; i < nr_phantom_devs; ++i )
>>>                     if ( phantom_devs[i].seg == pseg->nr &&
>>>                          phantom_devs[i].bus == bus &&
>>>                          phantom_devs[i].slot == PCI_SLOT(devfn) &&
>>>                          phantom_devs[i].stride > PCI_FUNC(devfn) )
>>>                     {
>>>                         pdev->phantom_stride = 
>>> phantom_devs[i].stride;
>>>                         break;
>>>                     }
>>> 
>>> So from the code, we can see this command line only works for phantom
>>> _function_, not for whole phantom _devices_.
>> 
>> What would it take to make it work for a whole phantom device?
> 
> First and foremost a definition of what a phantom device is and
> how one would behave. Once again - phantom functions are part
> of the PCIe specification, so those don't require a definition.

Konrad's patch from a while back seemed to do the required thing to
allow an otherwise invisible/undetected device to do DMA transfers
without freaking out the IOMMU that doesn't know about it.

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07 12:42     ` Gordan Bobic
@ 2014-01-07 14:38       ` Konrad Rzeszutek Wilk
  2014-01-07 14:47         ` Jan Beulich
  0 siblings, 1 reply; 40+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-01-07 14:38 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: Andrew Cooper, Feng Wu, Jan Beulich, xen-devel

On Tue, Jan 07, 2014 at 12:42:17PM +0000, Gordan Bobic wrote:
> On 2014-01-07 12:15, Jan Beulich wrote:
> >>>>On 07.01.14 at 12:35, Gordan Bobic <gordan@bobich.net> wrote:
> >>On 2014-01-07 11:26, Wu, Feng wrote:
> >>>>-----Original Message-----
> >>>>From: xen-devel-bounces@lists.xen.org
> >>>>[mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Gordan Bobic
> >>>>Sent: Tuesday, January 07, 2014 6:44 PM
> >>>>To: Andrew Cooper
> >>>>Cc: xen-devel@lists.xen.org
> >>>>Subject: Re: [Xen-devel] Multi-bridged PCIe devices (Was: Re:
> >>>>iommuu/vt-d
> >>>>issues with LSI MegaSAS (PERC5i))
> >>>>
> >>>>On 2014-01-07 10:38, Andrew Cooper wrote:
> >>>>> On 07/01/14 10:35, Gordan Bobic wrote:
> >>>>>> On 2014-01-07 03:17, Zhang, Yang Z wrote:
> >>>>>>> Konrad Rzeszutek Wilk wrote on 2014-01-07:
> >>>>>>>>> Which would look like this:
> >>>>>>>>>
> >>>>>>>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
> >>>>>>>>> on the card
> >>>>>>>>>           \--------------> IEEE-1394a
> >>>>>>>>>
> >>>>>>>>> I am actually wondering if this 07:00.0 device is the one that
> >>>>>>>>> reports itself as 08:00.0 (which I think is what you alluding to
> >>>>>>>>> Jan)
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> And to double check that theory I decided to pass in the IEEE-1394a
> >>>>>>>> to a guest:
> >>>>>>>>
> >>>>>>>>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
> >>>>>>>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
> >>>>>>>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
> >>>>>>>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0]
> >>>>>>>> fault
> >>>>>>>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
> >>>>>>>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
> >>>>>>>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)
> >>>>>>>> root_entry
> >>>>>>>> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
> >>>>>>>> context
> >>>>>>>> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)
> >>>>>>>> ctxt_entry[0]
> >>>>>>>> not present
> >>>>>>>>
> >>>>>>>> So, capture card OK - Likely the Tundra bridge has an issue:
> >>>>>>>>
> >>>>>>>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
> >>>>>>>> (prog-if 01 [Subtractive decode])
> >>>>>>>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV-
> >>>>VGASnoop-
> >>>>>>>>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+
> >>>>>>>> 66MHz-
> >>>>>>>>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> >>>><MAbort+
> >>>>>>>>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07,
> >>>>>>>> secondary=08,
> >>>>>>>>         subordinate=08, sec-latency=32 Memory behind bridge:
> >>>>>>>>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
> >>>>>>>>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> >>>>>>>> BridgeCtl:
> >>>>>>>>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> >>>>>>>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat-
> >>>>DiscTmrSERREn-
> >>>>>>>>         Capabilities: [60] Subsystem: Super Micro Computer Inc
> >>>>>>>> Device 0805
> >>>>>>>>         Capabilities: [a0] Power Management version 3
> >>>>>>>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> >>>>>>>>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0
> >>>>>>>> NoSoftRst+
> >>>>>>>>                 PME-Enable- DSel=0 DScale=0 PME-
> >>>>>>>>
> >>>>>>>> or there is some unknown bridge in the motherboard.
> >>>>>>>
> >>>>>>> According your description above, the upstream Linux should also have
> >>>>>>> the same problem. Did you see it with upstream Linux?

I did not even think to test. I sadly won't be able to do much of reboot/shutdown
as this is a production machine.

> >>>>>>
> >>>>>> The problem I was seeing with LSI cards (phantom device doing DMA)
> >>>>>> does, indeed, also occur in upstream Linux. If I enable intel-iommu on
> >>>>>> bare metal Linux, the same problem occurs as with Xen.
> >>>>>>
> >>>>>>> There may be some buggy device that generate DMA request with
> >>>>>>> internal
> >>>>>>> BDF but it didn't expose it(not like Phantom device). For those
> >>>>>>> devices, I think we need to setup the VT-d page table manually.
> >>>>>>
> >>>>>> I think what is needed is a pci-phantom style override that tells the
> >>>>>> hypervisor to tell the IOMMU to allow DMA traffic from a specific
> >>>>>> invisible device ID.
> >>>>>>
> >>>>>> Gordan
> >>>>>
> >>>>> There is.  See "pci-phantom" in
> >>>>> http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
> >>>>
> >>>>I thought this was only applicable to phantom _functions_ (number
> >>>>after
> >>>>the
> >>>>dot) rather than whole phantom _devices_. Is that not the case?
> >>>
> >>>I think that's right. I go through the related code for the pci
> >>>phantom device just now, I find that
> >>>the information of command line 'pci-phantom' is stored in variable '
> >>>phantom_devs[8] '
> >>>with type of s truct phantom_dev{}. This variable is used in function
> >>>alloc_pdev() as follow:
> >>>
> >>>
> >>>                for ( i = 0; i < nr_phantom_devs; ++i )
> >>>                    if ( phantom_devs[i].seg == pseg->nr &&
> >>>                         phantom_devs[i].bus == bus &&
> >>>                         phantom_devs[i].slot == PCI_SLOT(devfn) &&
> >>>                         phantom_devs[i].stride > PCI_FUNC(devfn) )
> >>>                    {
> >>>                        pdev->phantom_stride =
> >>>phantom_devs[i].stride;
> >>>                        break;
> >>>                    }
> >>>
> >>>So from the code, we can see this command line only works for phantom
> >>>_function_, not for whole phantom _devices_.
> >>
> >>What would it take to make it work for a whole phantom device?
> >
> >First and foremost a definition of what a phantom device is and
> >how one would behave. Once again - phantom functions are part
> >of the PCIe specification, so those don't require a definition.
> 
> Konrad's patch from a while back seemed to do the required thing to
> allow an otherwise invisible/undetected device to do DMA transfers
> without freaking out the IOMMU that doesn't know about it.

Except it didn't work :-) That was the first thing I tried with this
motherboard. And it looks like there are extra things I would need
to modify in the hypervisor for it to work (like make the
hypervisor create an fake PCI device with BARs and such).

Which is actually what I was going try out - see if I can make it
(hypervisor) add a PCI device for a non-existent PCI device (does
not show in the PCI configuration scan).

That requires knowing the MMIO BARs the 'fake' device has, and
.. well, whatever else the Intel VT-d code requires.

For reference, here is the code that Gordan was mentioning:


#include <linux/module.h>
#include <linux/string.h>
#include <linux/types.h>
#include <linux/init.h>
#include <linux/stat.h>
#include <linux/err.h>
#include <linux/ctype.h>
#include <linux/slab.h>
#include <linux/limits.h>
#include <linux/device.h>
#include <linux/pci.h>
#include <linux/device.h>

#include <linux/pci.h>

#include <xen/interface/xen.h>
#include <xen/interface/physdev.h>

#include <asm/xen/hypervisor.h>
#include <asm/xen/hypercall.h>

#define LSI_HACK  "0.1"

MODULE_AUTHOR("Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>");
MODULE_DESCRIPTION("lsi hack");
MODULE_LICENSE("GPL");
MODULE_VERSION(LSI_HACK);

static int __init lsi_hack_init(void)
{
        int r = 0;

        struct physdev_manage_pci manage_pci = {
                        .bus    = 0x8,
                        .devfn  = PCI_DEVFN(0,0),
                };
        r = HYPERVISOR_physdev_op(PHYSDEVOP_manage_pci_add,
                        &manage_pci);

        return r;
}

static void __exit lsi_hack_exit(void)
{
        int r = 0;
        struct physdev_manage_pci manage_pci;

        manage_pci.bus = 0x8;
        manage_pci.devfn = PCI_DEVFN(0,0);

        r = HYPERVISOR_physdev_op(PHYSDEVOP_manage_pci_remove,
                &manage_pci);
        if (r)
                printk(KERN_ERR "%s: %d\n", __FUNCTION__, r);
}

module_init(lsi_hack_init);
module_exit(lsi_hack_exit);

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07 14:38       ` Konrad Rzeszutek Wilk
@ 2014-01-07 14:47         ` Jan Beulich
  2014-01-07 15:40           ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 40+ messages in thread
From: Jan Beulich @ 2014-01-07 14:47 UTC (permalink / raw)
  To: Gordan Bobic, Konrad Rzeszutek Wilk; +Cc: Andrew Cooper, Feng Wu, xen-devel

>>> On 07.01.14 at 15:38, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> That requires knowing the MMIO BARs the 'fake' device has, and
> .. well, whatever else the Intel VT-d code requires.

Why would you need to know BAR values? Weren't we talking of
an invisible bridge (in which case one would expect that there's
no MSI-X interrupts to be used, which is the only reason I can
see us needing to know/read the BARs)?

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07 14:47         ` Jan Beulich
@ 2014-01-07 15:40           ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 40+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-01-07 15:40 UTC (permalink / raw)
  To: Jan Beulich; +Cc: Andrew Cooper, Gordan Bobic, Feng Wu, xen-devel

On Tue, Jan 07, 2014 at 02:47:23PM +0000, Jan Beulich wrote:
> >>> On 07.01.14 at 15:38, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > That requires knowing the MMIO BARs the 'fake' device has, and
> > .. well, whatever else the Intel VT-d code requires.
> 
> Why would you need to know BAR values? Weren't we talking of
> an invisible bridge (in which case one would expect that there's
> no MSI-X interrupts to be used, which is the only reason I can
> see us needing to know/read the BARs)?

I mispoke. I was thinking about the 'memory behind the bridge'
is what I need to add in somehwere.

I really need to look at the VT-d spec and implementation to see
what data I need to provide to it.

> 
> Jan
> 

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-02-21 19:08                                               ` Konrad Rzeszutek Wilk
@ 2014-02-24 10:14                                                 ` Jan Beulich
  0 siblings, 0 replies; 40+ messages in thread
From: Jan Beulich @ 2014-02-24 10:14 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk; +Cc: Andrew Cooper, Gordan Bobic, xen-devel

>>> On 21.02.14 at 20:08, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> The 0004-xen-pci-Introduce-a-way-to-deal-with-buggy-hardware-.patch
> is what is interesting.
> 
> If there is an interest in upstream this I can take a look -
> but I will need guidance from Jan how he would like to do it.

Well, I don't know. I'm not really in favor of adding such hacks,
since this doesn't scale: What if someone else breaks their
hardware in a different way, and we want to work around that
too? Such workarounds could easily start conflicting with one
another.

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07 10:44                                             ` Gordan Bobic
@ 2014-02-21 19:08                                               ` Konrad Rzeszutek Wilk
  2014-02-24 10:14                                                 ` Jan Beulich
  0 siblings, 1 reply; 40+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-02-21 19:08 UTC (permalink / raw)
  To: Gordan Bobic, jbeulich; +Cc: Andrew Cooper, xen-devel

[-- Attachment #1: Type: text/plain, Size: 6898 bytes --]

On Tue, Jan 07, 2014 at 10:44:29AM +0000, Gordan Bobic wrote:
> On 2014-01-07 10:38, Andrew Cooper wrote:
> >On 07/01/14 10:35, Gordan Bobic wrote:
> >>On 2014-01-07 03:17, Zhang, Yang Z wrote:
> >>>Konrad Rzeszutek Wilk wrote on 2014-01-07:
> >>>>>Which would look like this:
> >>>>>
> >>>>>C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
> >>>>>on the card
> >>>>>          \--------------> IEEE-1394a
> >>>>>
> >>>>>I am actually wondering if this 07:00.0 device is the one that
> >>>>>reports itself as 08:00.0 (which I think is what you alluding to
> >>>>>Jan)
> >>>>>
> >>>>
> >>>>And to double check that theory I decided to pass in the IEEE-1394a
> >>>>to a guest:
> >>>>
> >>>>           +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
> >>>>TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >>>>
> >>>>
> >>>>(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
> >>>>[VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
> >>>>[VT-D]iommu.c:865: DMAR:[DMA Read] Request device
> >>>>[0000:08:00.0] fault
> >>>>addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
> >>>>02h] Present bit in context entry is clear (XEN) print_vtd_entries:
> >>>>iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)
> >>>>root_entry
> >>>>= ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
> >>>>context
> >>>>= ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)
> >>>>ctxt_entry[0]
> >>>>not present
> >>>>
> >>>>So, capture card OK - Likely the Tundra bridge has an issue:
> >>>>
> >>>>07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
> >>>>(prog-if 01 [Subtractive decode])
> >>>>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> >>>>        ParErr- Stepping- SERR- FastB2B- DisINTx- Status:
> >>>>Cap+ 66MHz-
> >>>>        UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+
> >>>>        >SERR- <PERR- INTx- Latency: 0 Bus: primary=07,
> >>>>secondary=08,
> >>>>        subordinate=08, sec-latency=32 Memory behind bridge:
> >>>>        f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
> >>>>        DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> >>>>BridgeCtl:
> >>>>        Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> >>>>                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> >>>>        Capabilities: [60] Subsystem: Super Micro Computer Inc
> >>>>Device 0805
> >>>>        Capabilities: [a0] Power Management version 3
> >>>>                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> >>>>                PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0
> >>>>NoSoftRst+
> >>>>                PME-Enable- DSel=0 DScale=0 PME-
> >>>>
> >>>>or there is some unknown bridge in the motherboard.
> >>>
> >>>According your description above, the upstream Linux should also have
> >>>the same problem. Did you see it with upstream Linux?
> >>
> >>The problem I was seeing with LSI cards (phantom device doing DMA)
> >>does, indeed, also occur in upstream Linux. If I enable intel-iommu on
> >>bare metal Linux, the same problem occurs as with Xen.
> >>
> >>>There may be some buggy device that generate DMA request with
> >>>internal
> >>>BDF but it didn't expose it(not like Phantom device). For those
> >>>devices, I think we need to setup the VT-d page table manually.
> >>
> >>I think what is needed is a pci-phantom style override that tells the
> >>hypervisor to tell the IOMMU to allow DMA traffic from a specific
> >>invisible device ID.
> >>
> >>Gordan
> >
> >There is.  See "pci-phantom" in
> >http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
> 
> I thought this was only applicable to phantom _functions_ (number
> after the
> dot) rather than whole phantom _devices_. Is that not the case?

My new Supermicro X10SAE has this issue as well and I wrote a patch to 
"fix" this. Just to be clear - this is an issue with the motherboard
(or rather the PCI-to-PCI bridge) is that it violates VT-d chapter
3.9.2:
"For devices behind conventional PCI bridges, the source-id in the DMA
requests is the requestor-id of the bridge device.".

In my case the bridge is is:

06:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)

But all the requests from the devices behind the bridge show up with
request-id +1 (aka 07:00.0):


(XEN) [2014-02-05 22:23:23] [VT-D]iommu.c:880: iommu_fau3] [VT-D]iommu.c:882: iommu_fault_status: Primary Pending Fault
(XEN) [2014-02-05 22:23:23] [VT-D]iommu.c:860: DMAR:[DMA Read] Request device [0000:07:00.0] fault addr a57fc000, iommu reg = ffff82c000203000
(XEN) [2014-02-05 22:23:23] DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) [2014-02-05 22:23:23] print_vtd_entries: iommu ffff83023948b9c0 dev 0000:07:00.0 gmfn a57fc
(XEN) [2014-02-05 22:23:23]     root_entry = ffff82004000d000
(XEN) [2014-02-05 22:23:23]     root_entry[7] = 1b8eba001
(XEN) [2014-02-05 22:23:23]     context = ffff82004000e000
(XEN) [2014-02-05 22:23:23]     context[0] = 0_0
(XEN) [2014-02-05 22:23:23]     ctxt_entry[0] not present
[   21.305708] firewire_ohci 0000:07:03.0: added OHCI v1.10 device as card 0, 4 IR + 8 IT contexts, quirks 0x2
(XEN) [2014-02-05 22:23:24] [VT-D]iommu.c:880: iommu_fault_status: Fault Overflow
(XEN) [2014-02-05 22:23:24] [VT-D]iommu.c:882: iommu_fault_status: Primary Pending Fault
(XEN) [2014-02-05 22:23:24] [VT-D]iommu.c:860: DMAR:[DMA Read] Request device [0000:07:00.0] fault addr b76cd000, iommu reg = ffff82c000203000
(XEN) [2014-02-05 22:23:24] DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) [2014-02-05 22:23:24] print_vtd_entries: iommu ffff83023948b9c0 dev 0000:07:00.0 gmfn b76cd
(XEN) [2014-02-05 22:23:24]     root_entry = ffff82004000d000
(XEN) [2014-02-05 22:23:24]     root_entry[7] = 1b8eba001
(XEN) [2014-02-05 22:23:24]     context = ffff82004000e000
(XEN) [2014-02-05 22:23:24]     context[0] = 0_0
(XEN) [2014-02-05 22:23:24]     ctxt_entry[0] not present
[   21.386993] firewire_ohci 0000:07:03.0: bad self ID 0/1 (00000000 != ~00000000)
[   21.402892] initcall fw_ohci_init+0x0/0x1000 [firewire_ohci] returned 0 after 254567 usecs


Anyhow, the "solution" was to provide a "link" to the device
being passed in and create a fake device in Xen so that when
I passthrough my fireware: 07:03.0 it will also setup
a context for 07:00.0 device (which does not exist at all).

Attached is the Linux kernel module I use to let the hypervisor
know about the new device (this could have been written as
an user application too).

And the patch for the hypervisor, along with some extra debug
patches so when doing 'xl debug-keys Q' you can get a better
sense of what is what.

The 0004-xen-pci-Introduce-a-way-to-deal-with-buggy-hardware-.patch
is what is interesting.

If there is an interest in upstream this I can take a look -
but I will need guidance from Jan how he would like to do it.


[-- Attachment #2: hack.c --]
[-- Type: text/plain, Size: 1995 bytes --]


#include <linux/module.h>
#include <linux/string.h>
#include <linux/types.h>
#include <linux/init.h>
#include <linux/stat.h>
#include <linux/err.h>
#include <linux/ctype.h>
#include <linux/slab.h>
#include <linux/limits.h>
#include <linux/device.h>
#include <linux/pci.h>
#include <linux/device.h>

#include <linux/pci.h>

#include <xen/interface/xen.h>
#include <xen/interface/physdev.h>

#include <asm/xen/hypervisor.h>
#include <asm/xen/hypercall.h>

#define LSI_HACK  "0.1"

MODULE_AUTHOR("Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>");
MODULE_DESCRIPTION("lsi hack");
MODULE_LICENSE("GPL");
MODULE_VERSION(LSI_HACK);

/* Want to link to the device passed in the guest */
int bus = 7;
module_param(bus, int, 0644);
MODULE_PARM_DESC(bus, "bus");
int slot = 3;
module_param(slot, int, 0644);
MODULE_PARM_DESC(slot, "slot");
int func = 0;
module_param(func, int, 0644);
MODULE_PARM_DESC(func, "slot");

#define XEN_PCI_DEV_LINK 0x8
static int __init lsi_hack_init(void)
{
        int r = 0;

        struct physdev_pci_device_add add = {
			.seg	= 0,
                        .bus    = 0x7,
                        .devfn  = PCI_DEVFN(0,0),
			.physfn.bus	= bus, /* The device we want to copy from */
			.physfn.devfn	= PCI_DEVFN(slot,func),
			.flags = XEN_PCI_DEV_LINK,
                };
	printk("%s: %02x:%02x.%u, %02x:%02x.%u, %x\n",
		__func__, add.bus, PCI_SLOT(add.devfn),
		PCI_FUNC(add.devfn), add.physfn.bus,
		PCI_SLOT(add.physfn.devfn), PCI_SLOT(add.physfn.devfn),
		add.flags);
        r = HYPERVISOR_physdev_op(PHYSDEVOP_pci_device_add, &add);

        return r;
}

static void __exit lsi_hack_exit(void)
{
        int r = 0;
        struct physdev_manage_pci manage_pci;

        manage_pci.bus = 0x7;
        manage_pci.devfn = PCI_DEVFN(0,0);

        r = HYPERVISOR_physdev_op(PHYSDEVOP_manage_pci_remove,
                &manage_pci);
        if (r)
                printk(KERN_ERR "%s: %d\n", __FUNCTION__, r);
}

module_init(lsi_hack_init);
module_exit(lsi_hack_exit);

[-- Attachment #3: 0004-xen-pci-Introduce-a-way-to-deal-with-buggy-hardware-.patch --]
[-- Type: text/plain, Size: 11687 bytes --]

>From cb165429726978952f5b9e75bece1dcb5630667f Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Wed, 19 Feb 2014 10:58:19 -0500
Subject: [PATCH 4/6] xen/pci: Introduce a way to deal with buggy hardware with
 "hidden" PCI buses.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/arch/x86/physdev.c              | 14 +++++-
 xen/drivers/passthrough/pci.c       | 89 +++++++++++++++++++++++++++++++++----
 xen/drivers/passthrough/vtd/iommu.c | 31 +++++++++++++
 xen/include/public/physdev.h        |  1 +
 xen/include/xen/pci.h               |  3 ++
 5 files changed, 128 insertions(+), 10 deletions(-)

diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c
index bc0634c..f843c49 100644
--- a/xen/arch/x86/physdev.c
+++ b/xen/arch/x86/physdev.c
@@ -609,7 +609,11 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         if ( copy_from_guest(&add, arg, 1) != 0 )
             break;
 
-        pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+        if ( add.flags & XEN_PCI_DEV_EXTFN)
+            pdev_info.is_extfn = 1;
+        else
+            pdev_info.is_extfn = 0;
+
         if ( add.flags & XEN_PCI_DEV_VIRTFN )
         {
             pdev_info.is_virtfn = 1;
@@ -618,6 +622,14 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         }
         else
             pdev_info.is_virtfn = 0;
+
+        if ( add.flags & XEN_PCI_DEV_LINK )
+        {
+            pdev_info.is_link = 1;
+            pdev_info.physfn.bus = add.physfn.bus;
+            pdev_info.physfn.devfn = add.physfn.devfn;
+        } else
+            pdev_info.is_link = 0;
         ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info);
         break;
     }
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 2a6eaa4..0e59216 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -153,7 +153,8 @@ static void __init parse_phantom_dev(char *str) {
 }
 custom_param("pci-phantom", parse_phantom_dev);
 
-static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
+static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn,
+                                  int link, u8 orig_bus, u8 orig_devfn)
 {
     struct pci_dev *pdev;
 
@@ -169,8 +170,38 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
     *((u8*) &pdev->bus) = bus;
     *((u8*) &pdev->devfn) = devfn;
     pdev->domain = NULL;
+    pdev->link = NULL;
     INIT_LIST_HEAD(&pdev->msi_list);
 
+    if ( link )
+    {
+        struct pci_dev *dev;
+        list_for_each_entry ( dev, &pseg->alldevs_list, alldevs_list )
+        {
+            if ( dev->bus == orig_bus && dev->devfn == orig_devfn )
+            {
+                /* N.B. The 'bus' passed is 'new' one, while 'orig_bus' are
+                 * the ones we expect to exist. We over-write 'bus' and
+                 * 'devfn' with the original one so that this new device
+                 * will be created with the original device properties.
+                 */
+                if ( dev->link )
+                {
+                    xfree (pdev);
+                    return NULL;
+                }
+                bus = dev->bus;
+                devfn = dev->devfn;
+                dev->link = pdev;
+                pdev->link = dev;
+                pdev->info.is_link = 1;
+                printk("%04x:%02x:%02x.%u linked with %02x:%02x.%u\n",
+                       pseg->nr, pdev->bus, PCI_SLOT(pdev->devfn),
+                       PCI_FUNC(pdev->devfn), dev->bus, PCI_SLOT(dev->devfn),
+                       PCI_FUNC(dev->devfn));
+            }
+        }
+    }
     if ( pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
                              PCI_CAP_ID_MSIX) )
     {
@@ -201,12 +232,32 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
             sub_bus = pci_conf_read8(pseg->nr, bus, PCI_SLOT(devfn),
                                      PCI_FUNC(devfn), PCI_SUBORDINATE_BUS);
 
+            if ( pdev->info.is_link )
+            {
+                if ( sec_bus >= pdev->bus && pdev->bus <= sub_bus )
+                {
+#if 0
+                    u8 i = sec_bus;
+                    /* We can create an loop in bus2bridge by pointing to ourselves.
+                     * Hence destroy sec_bus up to pdev_bus values */
+                    spin_lock(&pseg->bus2bridge_lock);
+                    for ( ; i <= pdev->bus; i++ )
+                        pseg->bus2bridge[sec_bus].map = 0;
+                    spin_unlock(&pseg->bus2bridge_lock);
+                    /* And increment it so it won't cover us again*/
+                    sec_bus = pdev->bus + 1;
+                    printk("Link corrected [%02x:%02x:%u] spanning %x->%x\n", pdev->bus,
+                           PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn),sec_bus, sub_bus);
+#endif
+                    break;
+                }
+            }
             spin_lock(&pseg->bus2bridge_lock);
             for ( ; sec_bus <= sub_bus; sec_bus++ )
             {
                 pseg->bus2bridge[sec_bus].map = 1;
-                pseg->bus2bridge[sec_bus].bus = bus;
-                pseg->bus2bridge[sec_bus].devfn = devfn;
+                pseg->bus2bridge[sec_bus].bus = pdev->bus;
+                pseg->bus2bridge[sec_bus].devfn = pdev->devfn;
             }
             spin_unlock(&pseg->bus2bridge_lock);
             break;
@@ -299,7 +350,7 @@ int __init pci_hide_device(int bus, int devfn)
     int rc = -ENOMEM;
 
     spin_lock(&pcidevs_lock);
-    pdev = alloc_pdev(get_pseg(0), bus, devfn);
+    pdev = alloc_pdev(get_pseg(0), bus, devfn, 0, 0, 0);
     if ( pdev )
     {
         _pci_hide_device(pdev);
@@ -317,7 +368,7 @@ int __init pci_ro_device(int seg, int bus, int devfn)
 
     if ( !pseg )
         return -ENOMEM;
-    pdev = alloc_pdev(pseg, bus, devfn);
+    pdev = alloc_pdev(pseg, bus, devfn, 0, 0, 0);
     if ( !pdev )
         return -ENOMEM;
 
@@ -458,6 +509,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, const struct pci_dev_info *info)
     struct pci_seg *pseg;
     struct pci_dev *pdev;
     unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
+    u8 bus_link = 0, devfn_link = 0;
     const char *pdev_type;
     int ret;
 
@@ -474,6 +526,12 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, const struct pci_dev_info *info)
             pci_add_device(seg, info->physfn.bus, info->physfn.devfn, NULL);
         pdev_type = "virtual function";
     }
+    else if (info->is_link)
+    {
+        bus_link = info->physfn.bus;
+        devfn_link = info->physfn.devfn;
+        pdev_type = "link";
+    }
     else
     {
         info = NULL;
@@ -490,7 +548,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, const struct pci_dev_info *info)
     pseg = alloc_pseg(seg);
     if ( !pseg )
         goto out;
-    pdev = alloc_pdev(pseg, bus, devfn);
+    pdev = alloc_pdev(pseg, bus, devfn, (info && info->is_link), bus_link, devfn_link);
     if ( !pdev )
         goto out;
 
@@ -604,7 +662,7 @@ out:
 int pci_remove_device(u16 seg, u8 bus, u8 devfn)
 {
     struct pci_seg *pseg = get_pseg(seg);
-    struct pci_dev *pdev;
+    struct pci_dev *pdev, *link = NULL;
     int ret;
 
     ret = xsm_resource_unplug_pci(XSM_PRIV, (seg << 16) | (bus << 8) | devfn);
@@ -617,16 +675,29 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn)
         return -ENODEV;
 
     spin_lock(&pcidevs_lock);
+retry:
     list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
         if ( pdev->bus == bus && pdev->devfn == devfn )
         {
             ret = iommu_remove_device(pdev);
             if ( pdev->domain )
                 list_del(&pdev->domain_list);
-            pci_cleanup_msi(pdev);
+            if ( !pdev->info.is_link ) /* If we are not the 'fake device' */
+                pci_cleanup_msi(pdev);
+            if ( pdev->link ) {
+                /* Can be NULL if the other device was removed first. */
+                link = pdev->link;
+            }
             free_pdev(pseg, pdev);
             printk(XENLOG_DEBUG "PCI remove device %04x:%02x:%02x.%u\n",
                    seg, bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
+            if ( link )
+            {
+                bus = link->bus;
+                devfn = link->devfn;
+                link->link = NULL;
+                goto retry;
+            }
             break;
         }
 
@@ -838,7 +909,7 @@ static int __init _scan_pci_devices(struct pci_seg *pseg, void *arg)
                     continue;
                 }
 
-                pdev = alloc_pdev(pseg, bus, PCI_DEVFN(dev, func));
+                pdev = alloc_pdev(pseg, bus, PCI_DEVFN(dev, func), 0, 0, 0);
                 if ( !pdev )
                 {
                     printk("%s: alloc_pdev failed.\n", __func__);
diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c
index 5f10034..a5a4664 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1468,6 +1468,25 @@ static int domain_context_mapping(
         if ( ret )
             break;
 
+        if ( pdev->link )
+        {
+            u8 bus_link, devfn_link;
+            struct pci_dev *link_dev = pdev->link;
+
+            ASSERT ( link_dev );
+
+            bus_link = link_dev->bus;
+            devfn_link = link_dev->devfn;
+
+            if ( iommu_verbose )
+                dprintk(VTDPREFIX, "d%d:PCI: map %04x:%02x:%02x.%u\n",
+                        domain->domain_id, seg, bus_link,
+                        PCI_SLOT(devfn_link), PCI_FUNC(devfn_link));
+
+
+            ret = domain_context_mapping_one(domain, drhd->iommu, bus_link, devfn_link,
+                                             pci_get_pdev(seg, bus, devfn));
+        }
         if ( find_upstream_bridge(seg, &bus, &devfn, &secbus) < 1 )
             break;
 
@@ -1603,6 +1622,18 @@ static int domain_context_unmap(
         if ( ret )
             break;
 
+        if ( pdev->link )
+        {
+            struct pci_dev *link = pdev->link;
+
+            ASSERT(link->link == pdev);
+            tmp_bus = link->bus;
+            tmp_devfn = link->devfn;
+            if ( iommu_verbose )
+                dprintk(VTDPREFIX, "d%d:PCI: unmap %04x:%02x:%02x.%u\n",
+                        domain->domain_id, seg, tmp_bus, PCI_SLOT(tmp_devfn), PCI_FUNC(tmp_devfn));
+            ret = domain_context_unmap_one(domain, iommu, tmp_bus, tmp_devfn);
+        }
         tmp_bus = bus;
         tmp_devfn = devfn;
         if ( find_upstream_bridge(seg, &tmp_bus, &tmp_devfn, &secbus) < 1 )
diff --git a/xen/include/public/physdev.h b/xen/include/public/physdev.h
index d547928..c476175 100644
--- a/xen/include/public/physdev.h
+++ b/xen/include/public/physdev.h
@@ -281,6 +281,7 @@ DEFINE_XEN_GUEST_HANDLE(physdev_pci_mmcfg_reserved_t);
 #define XEN_PCI_DEV_EXTFN              0x1
 #define XEN_PCI_DEV_VIRTFN             0x2
 #define XEN_PCI_DEV_PXM                0x4
+#define XEN_PCI_DEV_LINK               0x8
 
 #define PHYSDEVOP_pci_device_add        25
 struct physdev_pci_device_add {
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index cadb525..b883c28 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -39,6 +39,7 @@ struct pci_dev_info {
         u8 bus;
         u8 devfn;
     } physfn;
+    bool_t is_link;
 };
 
 struct pci_dev {
@@ -75,6 +76,8 @@ struct pci_dev {
 #define PT_FAULT_THRESHOLD 10
     } fault;
     u64 vf_rlen[6];
+
+    struct pci_dev *link;
 };
 
 #define for_each_pdev(domain, pdev) \
-- 
1.8.3.1


[-- Attachment #4: 0001-pci-On-PCI-dump-keyhandler-include-bus2bridge-inform.patch --]
[-- Type: text/plain, Size: 1502 bytes --]

>From 3edf6e0b1b646a358ae14c64e726ad24350ad510 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Tue, 4 Feb 2014 12:54:53 -0500
Subject: [PATCH 1/6] pci: On PCI dump keyhandler include bus2bridge
 information

As it helps in figuring out whether they match reality if the
initial domain has altered the bus topology.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/drivers/passthrough/pci.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index c5c8344..64dfd73 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -942,6 +942,7 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
 {
     struct pci_dev *pdev;
     struct msi_desc *msi;
+    int i;
 
     printk("==== segment %04x ====\n", pseg->nr);
 
@@ -955,6 +956,17 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
                printk("%d ", msi->irq);
         printk(">\n");
     }
+    printk("==== Bus2Bridge %04x ====\n", pseg->nr);
+    spin_lock(&pseg->bus2bridge_lock);
+    for ( i = 0; i < MAX_BUSES; i++)
+    {
+        if ( !pseg->bus2bridge[i].map )
+            continue;
+        printk("%02x -> %02x:%02x.%u\n", i, pseg->bus2bridge[i].bus,
+               PCI_SLOT(pseg->bus2bridge[i].devfn),
+               PCI_FUNC(pseg->bus2bridge[i].devfn));
+    }
+    spin_unlock(&pseg->bus2bridge_lock);
 
     return 0;
 }
-- 
1.8.3.1


[-- Attachment #5: 0002-pci-On-PCI-dump-device-keyhandler-include-Device-and.patch --]
[-- Type: text/plain, Size: 1458 bytes --]

>From ad49978e083123a1068461cd7f2a8e0c2becca17 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Tue, 4 Feb 2014 12:52:35 -0500
Subject: [PATCH 2/6] pci: On PCI dump device keyhandler include Device and
 Vendor ID

As it helps in troubleshooting if the initial domain has
re-numbered the bus numbers and what Xen sees is not the
reality.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/drivers/passthrough/pci.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 64dfd73..1ad4f17 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -948,9 +948,12 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
 
     list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
     {
-        printk("%04x:%02x:%02x.%u - dom %-3d - MSIs < ",
+        int id = pci_conf_read32(pseg->nr, pdev->bus, PCI_SLOT(pdev->devfn),
+                                 PCI_FUNC(pdev->devfn), 0);
+        printk("%04x:%02x:%02x.%u (%04x:%04x)- dom %-3d - MSIs < ",
                pseg->nr, pdev->bus,
                PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn),
+               id & 0xffff, (id >> 16) & 0xffff,
                pdev->domain ? pdev->domain->domain_id : -1);
         list_for_each_entry ( msi, &pdev->msi_list, list )
                printk("%d ", msi->irq);
-- 
1.8.3.1


[-- Attachment #6: 0003-DEBUG-Include-upstream-bridge-information.patch --]
[-- Type: text/plain, Size: 1679 bytes --]

>From fb4cf64dcced21b1f98b1e60fbb9dfa70f61fc3c Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Tue, 4 Feb 2014 17:01:42 -0500
Subject: [PATCH 3/6] DEBUG: Include upstream bridge information.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/drivers/passthrough/pci.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 1ad4f17..2a6eaa4 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -950,6 +950,9 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
     {
         int id = pci_conf_read32(pseg->nr, pdev->bus, PCI_SLOT(pdev->devfn),
                                  PCI_FUNC(pdev->devfn), 0);
+        int rc = 0;
+        u8 bus, devfn, secbus;
+
         printk("%04x:%02x:%02x.%u (%04x:%04x)- dom %-3d - MSIs < ",
                pseg->nr, pdev->bus,
                PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn),
@@ -957,7 +960,14 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
                pdev->domain ? pdev->domain->domain_id : -1);
         list_for_each_entry ( msi, &pdev->msi_list, list )
                printk("%d ", msi->irq);
-        printk(">\n");
+        bus = pdev->bus;
+        devfn = pdev->devfn;
+
+        rc = find_upstream_bridge( pseg->nr, &bus, &devfn, &secbus );
+        if ( rc < 0)
+            printk(">\n");
+        else
+            printk(">[%02x:%02x.%u]\n", bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
     }
     printk("==== Bus2Bridge %04x ====\n", pseg->nr);
     spin_lock(&pseg->bus2bridge_lock);
-- 
1.8.3.1


[-- Attachment #7: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07 10:38                                           ` Andrew Cooper
@ 2014-01-07 10:44                                             ` Gordan Bobic
  2014-02-21 19:08                                               ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2014-01-07 10:44 UTC (permalink / raw)
  To: Andrew Cooper; +Cc: xen-devel

On 2014-01-07 10:38, Andrew Cooper wrote:
> On 07/01/14 10:35, Gordan Bobic wrote:
>> On 2014-01-07 03:17, Zhang, Yang Z wrote:
>>> Konrad Rzeszutek Wilk wrote on 2014-01-07:
>>>>> Which would look like this:
>>>>> 
>>>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
>>>>> on the card
>>>>>           \--------------> IEEE-1394a
>>>>> 
>>>>> I am actually wondering if this 07:00.0 device is the one that
>>>>> reports itself as 08:00.0 (which I think is what you alluding to
>>>>> Jan)
>>>>> 
>>>> 
>>>> And to double check that theory I decided to pass in the IEEE-1394a
>>>> to a guest:
>>>> 
>>>>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
>>>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>>>> 
>>>> 
>>>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
>>>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
>>>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] 
>>>> fault
>>>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
>>>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
>>>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)     
>>>> root_entry
>>>> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
>>>> context
>>>> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)     
>>>> ctxt_entry[0]
>>>> not present
>>>> 
>>>> So, capture card OK - Likely the Tundra bridge has an issue:
>>>> 
>>>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
>>>> (prog-if 01 [Subtractive decode])
>>>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 
>>>> 66MHz-
>>>>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+
>>>>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07, 
>>>> secondary=08,
>>>>         subordinate=08, sec-latency=32 Memory behind bridge:
>>>>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
>>>>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
>>>> BridgeCtl:
>>>>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>         Capabilities: [60] Subsystem: Super Micro Computer Inc
>>>> Device 0805
>>>>         Capabilities: [a0] Power Management version 3
>>>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 
>>>> NoSoftRst+
>>>>                 PME-Enable- DSel=0 DScale=0 PME-
>>>> 
>>>> or there is some unknown bridge in the motherboard.
>>> 
>>> According your description above, the upstream Linux should also have
>>> the same problem. Did you see it with upstream Linux?
>> 
>> The problem I was seeing with LSI cards (phantom device doing DMA)
>> does, indeed, also occur in upstream Linux. If I enable intel-iommu on
>> bare metal Linux, the same problem occurs as with Xen.
>> 
>>> There may be some buggy device that generate DMA request with 
>>> internal
>>> BDF but it didn't expose it(not like Phantom device). For those
>>> devices, I think we need to setup the VT-d page table manually.
>> 
>> I think what is needed is a pci-phantom style override that tells the
>> hypervisor to tell the IOMMU to allow DMA traffic from a specific
>> invisible device ID.
>> 
>> Gordan
> 
> There is.  See "pci-phantom" in
> http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html

I thought this was only applicable to phantom _functions_ (number after 
the
dot) rather than whole phantom _devices_. Is that not the case?

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07 10:35                                         ` Gordan Bobic
@ 2014-01-07 10:38                                           ` Andrew Cooper
  2014-01-07 10:44                                             ` Gordan Bobic
  0 siblings, 1 reply; 40+ messages in thread
From: Andrew Cooper @ 2014-01-07 10:38 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: xen-devel

On 07/01/14 10:35, Gordan Bobic wrote:
> On 2014-01-07 03:17, Zhang, Yang Z wrote:
>> Konrad Rzeszutek Wilk wrote on 2014-01-07:
>>>> Which would look like this:
>>>>
>>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
>>>> on the card
>>>>           \--------------> IEEE-1394a
>>>>
>>>> I am actually wondering if this 07:00.0 device is the one that
>>>> reports itself as 08:00.0 (which I think is what you alluding to
>>>> Jan)
>>>>
>>>
>>> And to double check that theory I decided to pass in the IEEE-1394a
>>> to a guest:
>>>
>>>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
>>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>>>
>>>
>>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
>>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
>>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault
>>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
>>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
>>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)     root_entry
>>> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)    
>>> context
>>> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)     ctxt_entry[0]
>>> not present
>>>
>>> So, capture card OK - Likely the Tundra bridge has an issue:
>>>
>>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
>>> (prog-if 01 [Subtractive decode])
>>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz-
>>>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+
>>>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07, secondary=08,
>>>         subordinate=08, sec-latency=32 Memory behind bridge:
>>>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
>>>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
>>> BridgeCtl:
>>>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>         Capabilities: [60] Subsystem: Super Micro Computer Inc
>>> Device 0805
>>>         Capabilities: [a0] Power Management version 3
>>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+
>>>                 PME-Enable- DSel=0 DScale=0 PME-
>>>
>>> or there is some unknown bridge in the motherboard.
>>
>> According your description above, the upstream Linux should also have
>> the same problem. Did you see it with upstream Linux?
>
> The problem I was seeing with LSI cards (phantom device doing DMA)
> does, indeed, also occur in upstream Linux. If I enable intel-iommu on
> bare metal Linux, the same problem occurs as with Xen.
>
>> There may be some buggy device that generate DMA request with internal
>> BDF but it didn't expose it(not like Phantom device). For those
>> devices, I think we need to setup the VT-d page table manually.
>
> I think what is needed is a pci-phantom style override that tells the
> hypervisor to tell the IOMMU to allow DMA traffic from a specific
> invisible device ID.
>
> Gordan

There is.  See "pci-phantom" in
http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html

~Andrew

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-07  3:17                                       ` Zhang, Yang Z
@ 2014-01-07 10:35                                         ` Gordan Bobic
  2014-01-07 10:38                                           ` Andrew Cooper
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2014-01-07 10:35 UTC (permalink / raw)
  To: xen-devel

On 2014-01-07 03:17, Zhang, Yang Z wrote:
> Konrad Rzeszutek Wilk wrote on 2014-01-07:
>>> Which would look like this:
>>> 
>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) on 
>>> the card
>>> 	      \--------------> IEEE-1394a
>>> 
>>> I am actually wondering if this 07:00.0 device is the one that
>>> reports itself as 08:00.0 (which I think is what you alluding to
>>> Jan)
>>> 
>> 
>> And to double check that theory I decided to pass in the IEEE-1394a to 
>> a guest:
>> 
>>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>> 
>> 
>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault
>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)     
>> root_entry
>> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)     
>> context
>> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)     ctxt_entry[0]
>> not present
>> 
>> So, capture card OK - Likely the Tundra bridge has an issue:
>> 
>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
>> (prog-if 01 [Subtractive decode])
>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz-
>>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+
>>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07, secondary=08,
>>         subordinate=08, sec-latency=32 Memory behind bridge:
>>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
>>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR- 
>> BridgeCtl:
>>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>         Capabilities: [60] Subsystem: Super Micro Computer Inc Device 
>> 0805
>>         Capabilities: [a0] Power Management version 3
>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+
>>                 PME-Enable- DSel=0 DScale=0 PME-
>> 
>> or there is some unknown bridge in the motherboard.
> 
> According your description above, the upstream Linux should also have
> the same problem. Did you see it with upstream Linux?

The problem I was seeing with LSI cards (phantom device doing DMA)
does, indeed, also occur in upstream Linux. If I enable intel-iommu on
bare metal Linux, the same problem occurs as with Xen.

> There may be some buggy device that generate DMA request with internal
> BDF but it didn't expose it(not like Phantom device). For those
> devices, I think we need to setup the VT-d page table manually.

I think what is needed is a pci-phantom style override that tells the
hypervisor to tell the IOMMU to allow DMA traffic from a specific
invisible device ID.

Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-06 21:45                                     ` Konrad Rzeszutek Wilk
@ 2014-01-07  3:17                                       ` Zhang, Yang Z
  2014-01-07 10:35                                         ` Gordan Bobic
  0 siblings, 1 reply; 40+ messages in thread
From: Zhang, Yang Z @ 2014-01-07  3:17 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk, Gordan Bobic; +Cc: xen-devel, Jan Beulich

Konrad Rzeszutek Wilk wrote on 2014-01-07:
>> Which would look like this:
>> 
>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) on the card
>> 	      \--------------> IEEE-1394a
>> 
>> I am actually wondering if this 07:00.0 device is the one that
>> reports itself as 08:00.0 (which I think is what you alluding to
>> Jan)
>> 
> 
> And to double check that theory I decided to pass in the IEEE-1394a to a guest:
> 
>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> 
> 
> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault
> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)     root_entry
> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)     context
> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)     ctxt_entry[0]
> not present
> 
> So, capture card OK - Likely the Tundra bridge has an issue:
> 
> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
> (prog-if 01 [Subtractive decode])
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz-
>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+
>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07, secondary=08,
>         subordinate=08, sec-latency=32 Memory behind bridge:
>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl:
>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>         Capabilities: [60] Subsystem: Super Micro Computer Inc Device 0805
>         Capabilities: [a0] Power Management version 3
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+
>                 PME-Enable- DSel=0 DScale=0 PME-
> 
> or there is some unknown bridge in the motherboard.

According your description above, the upstream Linux should also have the same problem. Did you see it with upstream Linux?
There may be some buggy device that generate DMA request with internal BDF but it didn't expose it(not like Phantom device). For those devices, I think we need to setup the VT-d page table manually.

Best regards,
Yang

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2014-01-06 20:26                                   ` Konrad Rzeszutek Wilk
@ 2014-01-06 21:45                                     ` Konrad Rzeszutek Wilk
  2014-01-07  3:17                                       ` Zhang, Yang Z
  0 siblings, 1 reply; 40+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-01-06 21:45 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: Yang Z Zhang, xen-devel, Jan Beulich

> Which would look like this:
> 
> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) on the card
> 	      \--------------> IEEE-1394a
> 
> I am actually wondering if this 07:00.0 device is the one that
> reports itself as 08:00.0 (which I think is what you alluding to Jan)
> 

And to double check that theory I decided to pass in the IEEE-1394a to a guest:

           +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]


(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 370f1000, iommu reg = ffff82c3ffd53000
(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1
(XEN)     root_entry = ffff83083d47f000
(XEN)     root_entry[8] = 72569b001
(XEN)     context = ffff83072569b000
(XEN)     context[0] = 0_0
(XEN)     ctxt_entry[0] not present

So, capture card OK - Likely the Tundra bridge has an issue:

07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) (prog-if 01 [Subtractive decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
        Latency: 0
        Bus: primary=07, secondary=08, subordinate=08, sec-latency=32
        Memory behind bridge: f0600000-f06fffff
        Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [60] Subsystem: Super Micro Computer Inc Device 0805
        Capabilities: [a0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-


or there is some unknown bridge in the motherboard.

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-12-13 15:27                                 ` Gordan Bobic
@ 2014-01-06 20:26                                   ` Konrad Rzeszutek Wilk
  2014-01-06 21:45                                     ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 40+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-01-06 20:26 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: Yang Z Zhang, xen-devel, Jan Beulich

On Fri, Dec 13, 2013 at 03:27:25PM +0000, Gordan Bobic wrote:
> On 12/13/2013 02:56 PM, Jan Beulich wrote:
> >>>>On 13.12.13 at 15:43, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> >>-[0000:00]-+-00.0  Intel Corporation 4th Gen Core Processor DRAM Controller
> >>            +-01.0-[01]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet Controller
> >>            |            \-00.1  Intel Corporation 82571EB Gigabit Ethernet Controller
> >>            +-01.1-[02]----00.0  LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
> >>            +-02.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
> >>            +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
> >>            +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
> >>            +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
> >>            +-16.3  Intel Corporation 8 Series/C220 Series Chipset Family KT Controller
> >>            +-19.0  Intel Corporation Ethernet Connection I217-LM
> >>            +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
> >>            +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
> >>            +-1c.0-[03]----00.0  Intel Corporation 82574L Gigabit Network Connection
> >>            +-1c.1-[04]----00.0  Intel Corporation 82574L Gigabit Network Connection
> >>            +-1c.3-[05]----00.0  Intel Corporation I210 Gigabit Network Connection
> >>            +-1c.4-[06]----00.0  Intel Corporation 82572EI Gigabit Ethernet Controller (Copper)
> >>            +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
> >>            |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
> >>            |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
> >>            |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
> >>            |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
> >>            |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
> >>            |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
> >>            |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
> >
> >So 00:1c.5 is the PCIe->PCI bridge, 07:00.0 and 08:01.0 are
> >PCI-PCI bridges, the capture devices are ordinary ones. The only
> >possibly unexpected aspect I can see here is that there are two
> >intermediate PCI-PCI bridges, but iirc this should be handled by
> >the VT-d code.
> 
> Not sure about multiple intermediate PCI-PCI bridges, but I can
> confirm that multiuple intermediate PCIe bridge setup works fine,
> e.g. Intel X58 -> Nvidia NF200 -> PLX -> GPU (GTX690)

To double-check whether this is a problem with the card itself,
I plucked the card in an AMD box (TA890FXE) and an Intel box (ThinkServer
TS130) and they work great there. It only has one PCIe->PCI bridge
for the PCI cards.

TA890FXE:

00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
        Flags: bus master, 66MHz, medium devsel, latency 64
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=64
        I/O behind bridge: 0000c000-0000dfff
02:05.0 PCI bridge: Hint Corp HB6 Universal PCI-PCI bridge (non-transparent mode) (rev 11) (prog-if 00 [Normal decode])
        Flags: bus master, medium devsel, latency 64
        Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
        Prefetchable memory behind bridge: fde00000-fdefffff
        Capabilities: [80] Power Management version 2
        Capabilities: [90] CompactPCI hot-swap <?> 

       +-14.4-[02-03]--+-05.0-[03]--+-08.0  Brooktree Corporation Bt878 Video Capture
           |               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
           |               |            +-09.0  Brooktree Corporation Bt878 Video Capture
           |               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
           |               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
           |               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
           |               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
           |               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
           |               \-06.0  NetMos Technology PCI 9835 Multi-I/O Controller

ThinkServer TS130:

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) (prog-if 01 [Subtractive decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=02, subordinate=03, sec-latency=32
        Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff
        Capabilities: [50] Subsystem: Lenovo Device 1025
00:00.0 Host bridge: Intel Corporation Device 0108 (rev 09)
        Subsystem: Lenovo Device 1025
        Flags: bus master, fast devsel, latency 0
        Capabilities: [e0] Vendor Specific Information: Len=0c <?>

           +-1e.0-[02-03]----00.0-[03]--+-08.0  Brooktree Corporation Bt878 Video Capture
           |                            +-08.1  Brooktree Corporation Bt878 Audio Capture
           |                            +-09.0  Brooktree Corporation Bt878 Video Capture
           |                            +-09.1  Brooktree Corporation Bt878 Audio Capture
           |                            +-0a.0  Brooktree Corporation Bt878 Video Capture
           |                            +-0a.1  Brooktree Corporation Bt878 Audio Capture
           |                            +-0b.0  Brooktree Corporation Bt878 Video Capture
           |                            \-0b.1  Brooktree Corporation Bt878 Audio Capture

SuperMicro (two PCI bridgeS):
         +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
           |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
           |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
           |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
           |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
           |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
           |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
           |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
           |                               \-03.0  Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]

00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d4) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
        Memory behind bridge: f0400000-f05fffff
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
        Capabilities: [a0] Power Management version 3
        Kernel driver in use: pcieport


00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: f0d00000-f0dfffff
        Capabilities: [88] Subsystem: Intel Corporation Device 2010
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [a0] Express Root Port (Slot+), MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [140] Root Complex Link
        Capabilities: [d94] #19
        Kernel driver in use: pcieport

07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) (prog-if 01 [Subtractive decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=07, secondary=08, subordinate=09, sec-latency=32
        Memory behind bridge: f0400000-f05fffff
        Capabilities: [60] Subsystem: Super Micro Computer Inc Device 0805
        Capabilities: [a0] Power Management version 3

Which would look like this:

C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) on the card
	      \--------------> IEEE-1394a

I am actually wondering if this 07:00.0 device is the one that
reports itself as 08:00.0 (which I think is what you alluding to Jan)

Gordan (in earlier emails) reported that the VT-d does work for him and
he has much more complex setup than I do on this motherboard:

This is on http://www.newegg.com/Product/Product.aspx?Item=N82E16813188070


 \-[0000:00]-+-00.0  Intel Corporation 5520 I/O Hub to ESI Port [8086:3406]
             +-07.0-[05-08]----00.0-[06-08]--+-00.0-[08]--+-00.0  NVIDIA Corporation GF100GL [Quadro 6000] [10de:06d8]
             |                               |            \-00.1  NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5]
             |                               \-02.0-[07]--+-00.0  NVIDIA Corporation GF100GL [Quadro 5000] [10de:06d9]
             |                                            \-00.1  NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5]

00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 [8086:340e] (rev 22) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 256 bytes
        Bus: primary=00, secondary=05, subordinate=08, sec-latency=0
        I/O behind bridge: 0000c000-0000dfff
        Memory behind bridge: f4000000-fbdfffff

05:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 256 bytes
        Bus: primary=05, secondary=06, subordinate=08, sec-latency=0
        I/O behind bridge: 0000c000-0000dfff
        Memory behind bridge: f4000000-fbdfffff
        Prefetchable memory behind bridge: 00000000a8000000-00000000bfffffff

06:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 256 bytes
        Bus: primary=06, secondary=08, subordinate=08, sec-latency=0
        I/O behind bridge: 0000d000-0000dfff
        Memory behind bridge: f8000000-fbdfffff
        Prefetchable memory behind bridge: 00000000b4000000-00000000bfffffff

Thought I am having a hard time parsing the 'lspci -vt' I think it is:

X58---> NF200 ---> Quadro 6000 (GPU and audio card)
  \---> NF200 ---> Quadro 5000 (GPU and audio card).


Which is similar setup as mine - an intermediate bridge.

So I think the code is OK - this is likely a firmware/motherboard
bug.

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-12-13 14:56                               ` Jan Beulich
@ 2013-12-13 15:27                                 ` Gordan Bobic
  2014-01-06 20:26                                   ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-12-13 15:27 UTC (permalink / raw)
  To: Jan Beulich; +Cc: Yang Z Zhang, xen-devel

On 12/13/2013 02:56 PM, Jan Beulich wrote:
>>>> On 13.12.13 at 15:43, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
>> -[0000:00]-+-00.0  Intel Corporation 4th Gen Core Processor DRAM Controller
>>             +-01.0-[01]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet Controller
>>             |            \-00.1  Intel Corporation 82571EB Gigabit Ethernet Controller
>>             +-01.1-[02]----00.0  LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
>>             +-02.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
>>             +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
>>             +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
>>             +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
>>             +-16.3  Intel Corporation 8 Series/C220 Series Chipset Family KT Controller
>>             +-19.0  Intel Corporation Ethernet Connection I217-LM
>>             +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
>>             +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
>>             +-1c.0-[03]----00.0  Intel Corporation 82574L Gigabit Network Connection
>>             +-1c.1-[04]----00.0  Intel Corporation 82574L Gigabit Network Connection
>>             +-1c.3-[05]----00.0  Intel Corporation I210 Gigabit Network Connection
>>             +-1c.4-[06]----00.0  Intel Corporation 82572EI Gigabit Ethernet Controller (Copper)
>>             +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
>>             |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
>>             |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
>>             |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
>>             |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
>>             |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
>>             |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
>>             |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
>
> So 00:1c.5 is the PCIe->PCI bridge, 07:00.0 and 08:01.0 are
> PCI-PCI bridges, the capture devices are ordinary ones. The only
> possibly unexpected aspect I can see here is that there are two
> intermediate PCI-PCI bridges, but iirc this should be handled by
> the VT-d code.

Not sure about multiple intermediate PCI-PCI bridges, but I can confirm 
that multiuple intermediate PCIe bridge setup works fine,
e.g. Intel X58 -> Nvidia NF200 -> PLX -> GPU (GTX690)

Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-12-13 14:43                             ` Konrad Rzeszutek Wilk
@ 2013-12-13 14:56                               ` Jan Beulich
  2013-12-13 15:27                                 ` Gordan Bobic
  0 siblings, 1 reply; 40+ messages in thread
From: Jan Beulich @ 2013-12-13 14:56 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk; +Cc: Yang Z Zhang, xen-devel, Gordan Bobic

>>> On 13.12.13 at 15:43, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> -[0000:00]-+-00.0  Intel Corporation 4th Gen Core Processor DRAM Controller
>            +-01.0-[01]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet Controller
>            |            \-00.1  Intel Corporation 82571EB Gigabit Ethernet Controller
>            +-01.1-[02]----00.0  LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
>            +-02.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
>            +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
>            +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
>            +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
>            +-16.3  Intel Corporation 8 Series/C220 Series Chipset Family KT Controller
>            +-19.0  Intel Corporation Ethernet Connection I217-LM
>            +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
>            +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
>            +-1c.0-[03]----00.0  Intel Corporation 82574L Gigabit Network Connection
>            +-1c.1-[04]----00.0  Intel Corporation 82574L Gigabit Network Connection
>            +-1c.3-[05]----00.0  Intel Corporation I210 Gigabit Network Connection
>            +-1c.4-[06]----00.0  Intel Corporation 82572EI Gigabit Ethernet Controller (Copper)
>            +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
>            |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
>            |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
>            |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
>            |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
>            |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
>            |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
>            |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture

So 00:1c.5 is the PCIe->PCI bridge, 07:00.0 and 08:01.0 are
PCI-PCI bridges, the capture devices are ordinary ones. The only
possibly unexpected aspect I can see here is that there are two
intermediate PCI-PCI bridges, but iirc this should be handled by
the VT-d code. You'd therefore mainly want to look at
find_upstream_bridge() getting things wrong, or
pseg->bus2bridge[] not getting set up correctly.

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-12-13 11:13                           ` Jan Beulich
@ 2013-12-13 14:43                             ` Konrad Rzeszutek Wilk
  2013-12-13 14:56                               ` Jan Beulich
  0 siblings, 1 reply; 40+ messages in thread
From: Konrad Rzeszutek Wilk @ 2013-12-13 14:43 UTC (permalink / raw)
  To: Jan Beulich; +Cc: Yang Z Zhang, xen-devel, Gordan Bobic

On Fri, Dec 13, 2013 at 11:13:11AM +0000, Jan Beulich wrote:
> >>> On 11.12.13 at 22:30, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > On Wed, Dec 11, 2013 at 09:15:17PM +0000, Gordan Bobic wrote:
> >> On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
> >> >Interestingly enough I just hit this with my brand-new Haswell CPU and
> >> >new motherboard when passing in a capture card. It shows:
> >> >
> >> >     +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video 
> > Capture
> >> >            |                               |            +-08.1  Brooktree 
> > Corporation Bt878 Audio Capture
> >> >            |                               |            +-09.0  Brooktree 
> > Corporation Bt878 Video Capture
> >> >            |                               |            +-09.1  Brooktree 
> > Corporation Bt878 Audio Capture
> >> >            |                               |            +-0a.0  Brooktree 
> > Corporation Bt878 Video Capture
> >> >            |                               |            +-0a.1  Brooktree 
> > Corporation Bt878 Audio Capture
> >> >            |                               |            +-0b.0  Brooktree 
> > Corporation Bt878 Video Capture
> >> >            |                               |            \-0b.1  Brooktree 
> > Corporation Bt878 Audio Capture
> >> >            |                               \-03.0  Texas Instruments 
> > TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >> >
> >> >And Xen says:
> >> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> >> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault 
> > addr 36aa3000, iommu reg = ffff82c3ffd53000
> >> >(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
> >> >(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
> >> >(XEN)     root_entry = ffff83083d47e000
> >> >(XEN)     root_entry[8] = 72569a001
> >> >(XEN)     context = ffff83072569a000
> >> >(XEN)     context[0] = 0_0
> >> >(XEN)     ctxt_entry[0] not present
> >> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> >> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault 
> > addr 36aa3000, iommu reg = ffff82c3ffd53000
> >> >
> >> >
> >> >Oddly enough it was working fine in a box with an AMD IOMMU. But
> >> >to be fair - that machine was running with Xen 4.1.
> >> >
> >> >The hack I developed: 
> > http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html 
> >> >ends up with this:
> >> >
> >> >(XEN) alloc_pdev: unknown type: 0000:08:00.0
> >> >(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
> >> >(XEN) [VT-D]iommu.c:1888: d0: context mapping failed
> >> >
> >> >(FYI, this Xen 4.3.1)
> >> >
> >> >Let me retry on the AMD box with the same version of Xen.
> >> 
> >> I may be wrong, but this doesn't look like the same problem (phantom
> >> PCI device on the bus). Or am I missing something?
> > 
> > It is. A phantom device as well.
> 
> Nothing in what you posted confirms this (because it doesn't
> show what the upstream bridge(s) is/are).

Ooops. Here is an output from lspci -v and lspci -vt

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: f0d00000-f0dfffff
	Capabilities: [88] Subsystem: Intel Corporation Device 2010
	Capabilities: [80] Power Management version 3
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [a0] Express Root Port (Slot+), MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [140] Root Complex Link
	Capabilities: [d94] #19
	Kernel driver in use: pcieport

00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: f0c00000-f0cfffff
	Capabilities: [88] Subsystem: Intel Corporation Device 2010
	Capabilities: [80] Power Management version 3
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [a0] Express Root Port (Slot+), MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [140] Root Complex Link
	Capabilities: [d94] #19
	Kernel driver in use: pcieport

00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) (prog-if 00 [VGA controller])
	Subsystem: Intel Corporation Device 2010
	Flags: bus master, fast devsel, latency 0, IRQ 77
	Memory at f0000000 (64-bit, non-prefetchable) [size=4M]
	Memory at e0000000 (64-bit, prefetchable) [size=256M]
	I/O ports at f000 [size=64]
	Expansion ROM at <unassigned> [disabled]
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [d0] Power Management version 2
	Capabilities: [a4] PCI Advanced Features
	Kernel driver in use: i915

00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
	Subsystem: Intel Corporation Device 2010
	Flags: fast devsel, IRQ 16
	Memory at f0e34000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
	Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00

00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04) (prog-if 30 [XHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, medium devsel, latency 0, IRQ 71
	Memory at f0e20000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
	Kernel driver in use: xhci_hcd

00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0, IRQ 78
	Memory at f0e40000 (64-bit, non-prefetchable) [size=16]
	Capabilities: [50] Power Management version 3
	Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Kernel driver in use: mei_me

00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller (rev 04) (prog-if 02 [16550])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 19
	I/O ports at f0e0 [size=8]
	Memory at f0e3e000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [c8] Power Management version 3
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Kernel driver in use: serial

00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 04)
	Subsystem: Super Micro Computer Inc Device 153a
	Flags: bus master, fast devsel, latency 0, IRQ 84
	Memory at f0e00000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0e3d000 (32-bit, non-prefetchable) [size=4K]
	I/O ports at f080 [size=32]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [e0] PCI Advanced Features
	Kernel driver in use: e1000e

00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) (prog-if 20 [EHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, medium devsel, latency 0, IRQ 16
	Memory at f0e3c000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
	Kernel driver in use: ehci-pci

00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: fast devsel, IRQ 22
	Memory at f0e30000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
	Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [100] Virtual Channel

00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: f0b00000-f0bfffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 0000b000-0000bfff
	Memory behind bridge: f0a00000-f0afffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 0000a000-0000afff
	Memory behind bridge: f0900000-f09fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
	I/O behind bridge: 00009000-00009fff
	Memory behind bridge: f0800000-f08fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
	Memory behind bridge: f0400000-f05fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #7 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0
	Memory behind bridge: f0700000-f07fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.7 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #8 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=0b, subordinate=0b, sec-latency=0
	I/O behind bridge: 00008000-00008fff
	Memory behind bridge: f0600000-f06fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) (prog-if 20 [EHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, medium devsel, latency 0, IRQ 23
	Memory at f0e3b000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
	Kernel driver in use: ehci-pci

00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, medium devsel, latency 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>
	Kernel driver in use: lpc_ich

00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 69
	I/O ports at f0d0 [size=8]
	I/O ports at f0c0 [size=4]
	I/O ports at f0b0 [size=8]
	I/O ports at f0a0 [size=4]
	I/O ports at f060 [size=32]
	Memory at f0e3a000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [70] Power Management version 3
	Capabilities: [a8] SATA HBA v1.0
	Kernel driver in use: ahci

00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: medium devsel, IRQ 18
	Memory at f0e39000 (64-bit, non-prefetchable) [size=256]
	I/O ports at f040 [size=32]
	Kernel driver in use: i801_smbus

00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0, IRQ 5
	Memory at f0e38000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [50] Power Management version 3
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-

01:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
	Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 79
	Memory at f0da0000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0d80000 (32-bit, non-prefetchable) [size=128K]
	I/O ports at e020 [size=32]
	Expansion ROM at f0d60000 [disabled] [size=128K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-15-17-ff-ff-3e-24-4c
	Kernel driver in use: e1000e

01:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
	Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 80
	Memory at f0d40000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0d20000 (32-bit, non-prefetchable) [size=128K]
	I/O ports at e000 [size=32]
	Expansion ROM at f0d00000 [disabled] [size=128K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-15-17-ff-ff-3e-24-4c
	Kernel driver in use: e1000e

02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
	Subsystem: LSI Logic / Symbios Logic Device 3020
	Flags: bus master, fast devsel, latency 0, IRQ 17
	I/O ports at d000 [size=256]
	Memory at f0cc0000 (64-bit, non-prefetchable) [size=16K]
	Memory at f0c80000 (64-bit, non-prefetchable) [size=256K]
	Expansion ROM at f0c00000 [disabled] [size=512K]
	Capabilities: [50] Power Management version 3
	Capabilities: [68] Express Endpoint, MSI 00
	Capabilities: [d0] Vital Product Data
	Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [138] Power Budgeting <?>
	Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
	Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
	Kernel driver in use: mpt2sas

03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Intel Corporation Gigabit CT Desktop Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Memory at f0bc0000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0b00000 (32-bit, non-prefetchable) [size=512K]
	I/O ports at c000 [size=32]
	Memory at f0be0000 (32-bit, non-prefetchable) [size=16K]
	Expansion ROM at f0b80000 [disabled] [size=256K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-61-49-00
	Kernel driver in use: pciback

04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Intel Corporation Gigabit CT Desktop Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 17
	Memory at f0ac0000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0a00000 (32-bit, non-prefetchable) [size=512K]
	I/O ports at b000 [size=32]
	Memory at f0ae0000 (32-bit, non-prefetchable) [size=16K]
	Expansion ROM at f0a80000 [disabled] [size=256K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-02-4c-71
	Kernel driver in use: pciback

05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
	Subsystem: Super Micro Computer Inc Device 1533
	Flags: fast devsel, IRQ 19
	Memory at f0900000 (32-bit, non-prefetchable) [disabled] [size=512K]
	I/O ports at a000 [disabled] [size=32]
	Memory at f0980000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] MSI-X: Enable- Count=5 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-25-90-ff-ff-86-c7-57
	Capabilities: [1a0] Transaction Processing Hints
	Kernel driver in use: pciback

06:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
	Subsystem: Intel Corporation PRO/1000 PT Desktop Adapter
	Flags: fast devsel, IRQ 16
	Memory at f0840000 (32-bit, non-prefetchable) [disabled] [size=128K]
	Memory at f0820000 (32-bit, non-prefetchable) [disabled] [size=128K]
	I/O ports at 9000 [disabled] [size=32]
	Expansion ROM at f0800000 [disabled] [size=128K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-00-37-02
	Kernel driver in use: pciback

07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) (prog-if 01 [Subtractive decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=07, secondary=08, subordinate=09, sec-latency=32
	Memory behind bridge: f0400000-f05fffff
	Capabilities: [60] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3

08:01.0 PCI bridge: Hint Corp HB6 Universal PCI-PCI bridge (non-transparent mode) (rev 11) (prog-if 00 [Normal decode])
	Physical Slot: 2
	Flags: bus master, medium devsel, latency 32
	Bus: primary=08, secondary=09, subordinate=09, sec-latency=32
	Memory behind bridge: f0400000-f04fffff
	Capabilities: [80] Power Management version 2
	Capabilities: [90] CompactPCI hot-swap <?>

08:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] (prog-if 10 [OHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Physical Slot: 4
	Flags: medium devsel, IRQ 16
	Memory at f0504000 (32-bit, non-prefetchable) [size=2K]
	Memory at f0500000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [44] Power Management version 2

09:08.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
	Subsystem: Device aa00:1460
	Flags: medium devsel, IRQ 18
	Memory at f0407000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:08.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev 11)
	Subsystem: Device aa00:1460
	Flags: medium devsel, IRQ 18
	Memory at f0406000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:09.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
	Subsystem: Device aa01:1461
	Flags: medium devsel, IRQ 19
	Memory at f0405000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:09.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev 11)
	Subsystem: Device aa01:1461
	Flags: medium devsel, IRQ 19
	Memory at f0404000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:0a.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
	Subsystem: Device aa02:1462
	Flags: medium devsel, IRQ 16
	Memory at f0403000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:0a.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev 11)
	Subsystem: Device aa02:1462
	Flags: medium devsel, IRQ 16
	Memory at f0402000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:0b.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
	Subsystem: Device aa03:1463
	Flags: medium devsel, IRQ 17
	Memory at f0401000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:0b.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev 11)
	Subsystem: Device aa03:1463
	Flags: medium devsel, IRQ 17
	Memory at f0400000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

0a:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0, IRQ 18
	Memory at f0700000 (64-bit, non-prefetchable) [size=8K]
	Capabilities: [50] Power Management version 3
	Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [90] MSI-X: Enable+ Count=8 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [150] Latency Tolerance Reporting
	Kernel driver in use: xhci_hcd

0b:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0, IRQ 70
	I/O ports at 8050 [size=8]
	I/O ports at 8040 [size=4]
	I/O ports at 8030 [size=8]
	I/O ports at 8020 [size=4]
	I/O ports at 8000 [size=32]
	Memory at f0600000 (32-bit, non-prefetchable) [size=512]
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [78] Power Management version 3
	Capabilities: [80] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Kernel driver in use: ahci

-[0000:00]-+-00.0  Intel Corporation 4th Gen Core Processor DRAM Controller
           +-01.0-[01]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet Controller
           |            \-00.1  Intel Corporation 82571EB Gigabit Ethernet Controller
           +-01.1-[02]----00.0  LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
           +-02.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
           +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
           +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
           +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
           +-16.3  Intel Corporation 8 Series/C220 Series Chipset Family KT Controller
           +-19.0  Intel Corporation Ethernet Connection I217-LM
           +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
           +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
           +-1c.0-[03]----00.0  Intel Corporation 82574L Gigabit Network Connection
           +-1c.1-[04]----00.0  Intel Corporation 82574L Gigabit Network Connection
           +-1c.3-[05]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1c.4-[06]----00.0  Intel Corporation 82572EI Gigabit Ethernet Controller (Copper)
           +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
           |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
           |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
           |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
           |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
           |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
           |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
           |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
           |                               \-03.0  Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
           +-1c.6-[0a]----00.0  Renesas Technology Corp. uPD720202 USB 3.0 Host Controller
           +-1c.7-[0b]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           +-1d.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1
           +-1f.0  Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller
           +-1f.2  Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]
           +-1f.3  Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller
           \-1f.6  Intel Corporation 8 Series Chipset Family Thermal Management Controller
> 
> >> As far as I can tell, the original problem was arising on cards that
> >> are PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge.
> >> Xen wasn't the only thing affected in my case - bare metal Linux
> >> kernel was also having problems with intel-iommu=1 in the kernel
> >> boot parameters. If might be worth trying that with your card to see
> >> what happens. If bare metal Linux with intel-iommu=1 works for your
> >> card, it's probably not the same problem (of course it could be
> >> similar/related).
> > 
> > That is a similar problem here. Except that I have a PCI devices and
> > it goes over an PCIe bridge (I think).
> 
> As said above - to see that's the case would require to see more
> lspci output than what you provided.
> 
> >> Out of interest, I noticed recently there is a xen parameter
> >> "pci-phantom", but I haven't been able to find documentation for it.
> >> Can you point me in the right direction? Does it, perchance, allow
> >> specifying the PCI slot ID of a phantom device so that IOMMU doesn't
> >> freak out when a seemingly non-existant device starts trying to do
> >> DMA?
> > 
> > I forgot about it!
> > 
> > t 4e3c592c93d7dbe02ca36878457515d30fe931d2
> > Author: Jan Beulich <jbeulich@suse.com>
> > Date:   Mon Jan 7 12:58:09 2013 +0100
> > 
> >     IOMMU: add option to specify devices behaving like ones using phantom functions
> 
> Note the term "functions" here: This is about a feature of the PCI
> spec that some devices apparently use without declaring they do.
> This has nothing to do with entire devices being invisible, in order
> for there to be phantom functions there always has to be an
> ordinary device at function 0.
> 
> Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-12-11 21:30                         ` Konrad Rzeszutek Wilk
@ 2013-12-13 11:13                           ` Jan Beulich
  2013-12-13 14:43                             ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 40+ messages in thread
From: Jan Beulich @ 2013-12-13 11:13 UTC (permalink / raw)
  To: Gordan Bobic, Konrad Rzeszutek Wilk; +Cc: Yang Z Zhang, xen-devel

>>> On 11.12.13 at 22:30, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> On Wed, Dec 11, 2013 at 09:15:17PM +0000, Gordan Bobic wrote:
>> On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
>> >Interestingly enough I just hit this with my brand-new Haswell CPU and
>> >new motherboard when passing in a capture card. It shows:
>> >
>> >     +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video 
> Capture
>> >            |                               |            +-08.1  Brooktree 
> Corporation Bt878 Audio Capture
>> >            |                               |            +-09.0  Brooktree 
> Corporation Bt878 Video Capture
>> >            |                               |            +-09.1  Brooktree 
> Corporation Bt878 Audio Capture
>> >            |                               |            +-0a.0  Brooktree 
> Corporation Bt878 Video Capture
>> >            |                               |            +-0a.1  Brooktree 
> Corporation Bt878 Audio Capture
>> >            |                               |            +-0b.0  Brooktree 
> Corporation Bt878 Video Capture
>> >            |                               |            \-0b.1  Brooktree 
> Corporation Bt878 Audio Capture
>> >            |                               \-03.0  Texas Instruments 
> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>> >
>> >And Xen says:
>> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
>> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
>> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault 
> addr 36aa3000, iommu reg = ffff82c3ffd53000
>> >(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
>> >(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
>> >(XEN)     root_entry = ffff83083d47e000
>> >(XEN)     root_entry[8] = 72569a001
>> >(XEN)     context = ffff83072569a000
>> >(XEN)     context[0] = 0_0
>> >(XEN)     ctxt_entry[0] not present
>> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
>> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
>> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault 
> addr 36aa3000, iommu reg = ffff82c3ffd53000
>> >
>> >
>> >Oddly enough it was working fine in a box with an AMD IOMMU. But
>> >to be fair - that machine was running with Xen 4.1.
>> >
>> >The hack I developed: 
> http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html 
>> >ends up with this:
>> >
>> >(XEN) alloc_pdev: unknown type: 0000:08:00.0
>> >(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
>> >(XEN) [VT-D]iommu.c:1888: d0: context mapping failed
>> >
>> >(FYI, this Xen 4.3.1)
>> >
>> >Let me retry on the AMD box with the same version of Xen.
>> 
>> I may be wrong, but this doesn't look like the same problem (phantom
>> PCI device on the bus). Or am I missing something?
> 
> It is. A phantom device as well.

Nothing in what you posted confirms this (because it doesn't
show what the upstream bridge(s) is/are).

>> As far as I can tell, the original problem was arising on cards that
>> are PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge.
>> Xen wasn't the only thing affected in my case - bare metal Linux
>> kernel was also having problems with intel-iommu=1 in the kernel
>> boot parameters. If might be worth trying that with your card to see
>> what happens. If bare metal Linux with intel-iommu=1 works for your
>> card, it's probably not the same problem (of course it could be
>> similar/related).
> 
> That is a similar problem here. Except that I have a PCI devices and
> it goes over an PCIe bridge (I think).

As said above - to see that's the case would require to see more
lspci output than what you provided.

>> Out of interest, I noticed recently there is a xen parameter
>> "pci-phantom", but I haven't been able to find documentation for it.
>> Can you point me in the right direction? Does it, perchance, allow
>> specifying the PCI slot ID of a phantom device so that IOMMU doesn't
>> freak out when a seemingly non-existant device starts trying to do
>> DMA?
> 
> I forgot about it!
> 
> t 4e3c592c93d7dbe02ca36878457515d30fe931d2
> Author: Jan Beulich <jbeulich@suse.com>
> Date:   Mon Jan 7 12:58:09 2013 +0100
> 
>     IOMMU: add option to specify devices behaving like ones using phantom functions

Note the term "functions" here: This is about a feature of the PCI
spec that some devices apparently use without declaring they do.
This has nothing to do with entire devices being invisible, in order
for there to be phantom functions there always has to be an
ordinary device at function 0.

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-12-11 21:15                       ` Gordan Bobic
@ 2013-12-11 21:30                         ` Konrad Rzeszutek Wilk
  2013-12-13 11:13                           ` Jan Beulich
  0 siblings, 1 reply; 40+ messages in thread
From: Konrad Rzeszutek Wilk @ 2013-12-11 21:30 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: Zhang, Yang Z, xen-devel, Jan Beulich

On Wed, Dec 11, 2013 at 09:15:17PM +0000, Gordan Bobic wrote:
> On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
> >On Thu, Sep 12, 2013 at 06:20:18AM +0000, Zhang, Yang Z wrote:
> >>Jan Beulich wrote on 2013-09-11:
> >>>>>>On 11.09.13 at 15:26, Gordan Bobic <gordan@bobich.net> wrote:
> >>>>On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
> >>>><JBeulich@suse.com>
> >>>>  wrote:
> >>>>>>>>On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
> >>>>>>On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
> >>>>>><JBeulich@suse.com>
> >>>>>>  wrote:
> >>>>>>>>>>On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
> >>>>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
> >>>>>>>>
> >>>>>>>>  I'll try adding one of my LSI cards and see the comparative
> >>>>>>>>behaviour. Right now I don't even know if the phantom device  is
> >>>>>>>>on the SAS card or the motherboard.
> >>>>>>>
> >>>>>>>The Adaptec card being the only thing on bus 0f makes it pretty
> >>>>>>>likely that this other device also is on that card.
> >>>>>>>
> >>>>>>>I guess the issue is mainly because the device itself is a PCI
> >>>>>>>one, while the immediately upstream bridge (where I mean only the
> >>>>>>>visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
> >>>>>>>them. And as long as firmware doesn't know about that bridge and
> >>>>>>>the bridge doesn't properly handle config space accesses to it,
> >>>>>>>such a device just can't be used with an IOMMU (without some yet
> >>>>>>>to be invented workaround).
> >>>>>>>
> >>>>>>  I'm actually thinking about Konrad's proposed hack in that
> >>>>>>thread from 3 years ago. If the device IDs are parameterized  out
> >>>>>>rather than hard-coded, then this could work in nearly the  same
> >>>>>>was as xen-pciback in terms of usage. Pass the phantom  device IDs
> >>>>>>as parameters to the module. Done that way it  might even be
> >>>>>>considered clean enough to be fit for public  consumption.
> >>>>>
> >>>>>Except that, short of being able to determine it via config space
> >>>>>reads, we also need the resulting command line option to tell us
> >>>>>that what kind of device that is.
> >>>>>
> >>>>  Not sure I follow. Why do we need to know the device type?
> >>>
> >>>Just look at set_msi_source_id() as well as
> >>>domain_context_{mapping,unmap}() (just the most prominent
> >>>examples): Behavior here heavily depends on the type of the device
> >>>itself _and_ that of the upstream bridge(s).
> >>Looks like there are many devices are failed to work. I wonder whether the PCI/PCIe specification tells how to detect the hidden device behind those devices (Like detection of phantom device). If not, I think those devices are buggy. Or we can say those devices are not really PCI/PCIe compatible. Since VT-d only covers the PCI/PCIe device, it's reasonable that non-PCI/PCIe device failed to work under VT-d.
> >>
> >>As Jan's suggestion, we need the user to tell us whether there is a hidden device or BDF behind anther device that the OS is unaware. We need to pass that info to Xen before pass-thought the device.
> >>
> >
> >Interestingly enough I just hit this with my brand-new Haswell CPU and
> >new motherboard when passing in a capture card. It shows:
> >
> >     +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
> >            |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
> >            |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
> >            |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
> >            |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
> >            |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
> >            |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
> >            |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
> >            |                               \-03.0  Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >
> >And Xen says:
> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000
> >(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
> >(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
> >(XEN)     root_entry = ffff83083d47e000
> >(XEN)     root_entry[8] = 72569a001
> >(XEN)     context = ffff83072569a000
> >(XEN)     context[0] = 0_0
> >(XEN)     ctxt_entry[0] not present
> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000
> >
> >
> >Oddly enough it was working fine in a box with an AMD IOMMU. But
> >to be fair - that machine was running with Xen 4.1.
> >
> >The hack I developed: http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
> >ends up with this:
> >
> >(XEN) alloc_pdev: unknown type: 0000:08:00.0
> >(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
> >(XEN) [VT-D]iommu.c:1888: d0: context mapping failed
> >
> >(FYI, this Xen 4.3.1)
> >
> >Let me retry on the AMD box with the same version of Xen.
> 
> I may be wrong, but this doesn't look like the same problem (phantom
> PCI device on the bus). Or am I missing something?

It is. A phantom device as well.
> 
> As far as I can tell, the original problem was arising on cards that
> are PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge.
> Xen wasn't the only thing affected in my case - bare metal Linux
> kernel was also having problems with intel-iommu=1 in the kernel
> boot parameters. If might be worth trying that with your card to see
> what happens. If bare metal Linux with intel-iommu=1 works for your
> card, it's probably not the same problem (of course it could be
> similar/related).

That is a similar problem here. Except that I have a PCI devices and
it goes over an PCIe bridge (I think).
> 
> Out of interest, I noticed recently there is a xen parameter
> "pci-phantom", but I haven't been able to find documentation for it.
> Can you point me in the right direction? Does it, perchance, allow
> specifying the PCI slot ID of a phantom device so that IOMMU doesn't
> freak out when a seemingly non-existant device starts trying to do
> DMA?

I forgot about it!

t 4e3c592c93d7dbe02ca36878457515d30fe931d2
Author: Jan Beulich <jbeulich@suse.com>
Date:   Mon Jan 7 12:58:09 2013 +0100

    IOMMU: add option to specify devices behaving like ones using phantom functions
    
    At least certain Marvell SATA controllers are known to issue bus master
    requests with a non-zero function as origin, despite themselves being
    single function devices.

Here is what the manpage says:

+### pci-phantom
+> `=[<seg>:]<bus>:<device>,<stride>`
+
+Mark a group of PCI devices as using phantom functions without actually
+advertising so, so the IOMMU can create translation contexts for them.
+
+All numbers specified must be hexadecimal ones.
+
+This option can be specified more than once (up to 8 times at present).
+

Hm, time to try it out.
> 
> Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-12-11 18:32                     ` Konrad Rzeszutek Wilk
@ 2013-12-11 21:15                       ` Gordan Bobic
  2013-12-11 21:30                         ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-12-11 21:15 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk; +Cc: Zhang, Yang Z, xen-devel, Jan Beulich

On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
> On Thu, Sep 12, 2013 at 06:20:18AM +0000, Zhang, Yang Z wrote:
>> Jan Beulich wrote on 2013-09-11:
>>>>>> On 11.09.13 at 15:26, Gordan Bobic <gordan@bobich.net> wrote:
>>>> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
>>>> <JBeulich@suse.com>
>>>>   wrote:
>>>>>>>> On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
>>>>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
>>>>>> <JBeulich@suse.com>
>>>>>>   wrote:
>>>>>>>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>>>>>>>>   dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>>>>>>>
>>>>>>>>   I'll try adding one of my LSI cards and see the comparative
>>>>>>>> behaviour. Right now I don't even know if the phantom device  is
>>>>>>>> on the SAS card or the motherboard.
>>>>>>>
>>>>>>> The Adaptec card being the only thing on bus 0f makes it pretty
>>>>>>> likely that this other device also is on that card.
>>>>>>>
>>>>>>> I guess the issue is mainly because the device itself is a PCI
>>>>>>> one, while the immediately upstream bridge (where I mean only the
>>>>>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>>>>>>> them. And as long as firmware doesn't know about that bridge and
>>>>>>> the bridge doesn't properly handle config space accesses to it,
>>>>>>> such a device just can't be used with an IOMMU (without some yet
>>>>>>> to be invented workaround).
>>>>>>>
>>>>>>   I'm actually thinking about Konrad's proposed hack in that
>>>>>> thread from 3 years ago. If the device IDs are parameterized  out
>>>>>> rather than hard-coded, then this could work in nearly the  same
>>>>>> was as xen-pciback in terms of usage. Pass the phantom  device IDs
>>>>>> as parameters to the module. Done that way it  might even be
>>>>>> considered clean enough to be fit for public  consumption.
>>>>>
>>>>> Except that, short of being able to determine it via config space
>>>>> reads, we also need the resulting command line option to tell us
>>>>> that what kind of device that is.
>>>>>
>>>>   Not sure I follow. Why do we need to know the device type?
>>>
>>> Just look at set_msi_source_id() as well as
>>> domain_context_{mapping,unmap}() (just the most prominent
>>> examples): Behavior here heavily depends on the type of the device
>>> itself _and_ that of the upstream bridge(s).
>> Looks like there are many devices are failed to work. I wonder whether the PCI/PCIe specification tells how to detect the hidden device behind those devices (Like detection of phantom device). If not, I think those devices are buggy. Or we can say those devices are not really PCI/PCIe compatible. Since VT-d only covers the PCI/PCIe device, it's reasonable that non-PCI/PCIe device failed to work under VT-d.
>>
>> As Jan's suggestion, we need the user to tell us whether there is a hidden device or BDF behind anther device that the OS is unaware. We need to pass that info to Xen before pass-thought the device.
>>
>
> Interestingly enough I just hit this with my brand-new Haswell CPU and
> new motherboard when passing in a capture card. It shows:
>
>      +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
>             |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
>             |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
>             |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
>             |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
>             |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
>             |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
>             |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
>             |                               \-03.0  Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>
> And Xen says:
> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000
> (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
> (XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
> (XEN)     root_entry = ffff83083d47e000
> (XEN)     root_entry[8] = 72569a001
> (XEN)     context = ffff83072569a000
> (XEN)     context[0] = 0_0
> (XEN)     ctxt_entry[0] not present
> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000
>
>
> Oddly enough it was working fine in a box with an AMD IOMMU. But
> to be fair - that machine was running with Xen 4.1.
>
> The hack I developed: http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
> ends up with this:
>
> (XEN) alloc_pdev: unknown type: 0000:08:00.0
> (XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
> (XEN) [VT-D]iommu.c:1888: d0: context mapping failed
>
> (FYI, this Xen 4.3.1)
>
> Let me retry on the AMD box with the same version of Xen.

I may be wrong, but this doesn't look like the same problem (phantom PCI 
device on the bus). Or am I missing something?

As far as I can tell, the original problem was arising on cards that are 
PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge. Xen 
wasn't the only thing affected in my case - bare metal Linux kernel was 
also having problems with intel-iommu=1 in the kernel boot parameters. 
If might be worth trying that with your card to see what happens. If 
bare metal Linux with intel-iommu=1 works for your card, it's probably 
not the same problem (of course it could be similar/related).

Out of interest, I noticed recently there is a xen parameter 
"pci-phantom", but I haven't been able to find documentation for it. Can 
you point me in the right direction? Does it, perchance, allow 
specifying the PCI slot ID of a phantom device so that IOMMU doesn't 
freak out when a seemingly non-existant device starts trying to do DMA?

Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-12  6:20                   ` Zhang, Yang Z
@ 2013-12-11 18:32                     ` Konrad Rzeszutek Wilk
  2013-12-11 21:15                       ` Gordan Bobic
  0 siblings, 1 reply; 40+ messages in thread
From: Konrad Rzeszutek Wilk @ 2013-12-11 18:32 UTC (permalink / raw)
  To: Zhang, Yang Z; +Cc: xen-devel, Gordan Bobic, Jan Beulich

On Thu, Sep 12, 2013 at 06:20:18AM +0000, Zhang, Yang Z wrote:
> Jan Beulich wrote on 2013-09-11:
> >>>> On 11.09.13 at 15:26, Gordan Bobic <gordan@bobich.net> wrote:
> >> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
> >> <JBeulich@suse.com>
> >>  wrote:
> >>>>>> On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
> >>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
> >>>> <JBeulich@suse.com>
> >>>>  wrote:
> >>>>>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
> >>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
> >>>>>>  
> >>>>>>  I'll try adding one of my LSI cards and see the comparative
> >>>>>> behaviour. Right now I don't even know if the phantom device  is
> >>>>>> on the SAS card or the motherboard.
> >>>>> 
> >>>>> The Adaptec card being the only thing on bus 0f makes it pretty
> >>>>> likely that this other device also is on that card.
> >>>>> 
> >>>>> I guess the issue is mainly because the device itself is a PCI
> >>>>> one, while the immediately upstream bridge (where I mean only the
> >>>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
> >>>>> them. And as long as firmware doesn't know about that bridge and
> >>>>> the bridge doesn't properly handle config space accesses to it,
> >>>>> such a device just can't be used with an IOMMU (without some yet
> >>>>> to be invented workaround).
> >>>>> 
> >>>>  I'm actually thinking about Konrad's proposed hack in that
> >>>> thread from 3 years ago. If the device IDs are parameterized  out
> >>>> rather than hard-coded, then this could work in nearly the  same
> >>>> was as xen-pciback in terms of usage. Pass the phantom  device IDs
> >>>> as parameters to the module. Done that way it  might even be
> >>>> considered clean enough to be fit for public  consumption.
> >>> 
> >>> Except that, short of being able to determine it via config space
> >>> reads, we also need the resulting command line option to tell us
> >>> that what kind of device that is.
> >>> 
> >>  Not sure I follow. Why do we need to know the device type?
> > 
> > Just look at set_msi_source_id() as well as
> > domain_context_{mapping,unmap}() (just the most prominent
> > examples): Behavior here heavily depends on the type of the device
> > itself _and_ that of the upstream bridge(s).
> Looks like there are many devices are failed to work. I wonder whether the PCI/PCIe specification tells how to detect the hidden device behind those devices (Like detection of phantom device). If not, I think those devices are buggy. Or we can say those devices are not really PCI/PCIe compatible. Since VT-d only covers the PCI/PCIe device, it's reasonable that non-PCI/PCIe device failed to work under VT-d.
> 
> As Jan's suggestion, we need the user to tell us whether there is a hidden device or BDF behind anther device that the OS is unaware. We need to pass that info to Xen before pass-thought the device.
> 

Interestingly enough I just hit this with my brand-new Haswell CPU and
new motherboard when passing in a capture card. It shows:

    +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
           |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
           |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
           |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
           |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
           |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
           |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
           |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
           |                               \-03.0  Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]

And Xen says:
(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000
(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
(XEN)     root_entry = ffff83083d47e000
(XEN)     root_entry[8] = 72569a001
(XEN)     context = ffff83072569a000
(XEN)     context[0] = 0_0
(XEN)     ctxt_entry[0] not present
(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000


Oddly enough it was working fine in a box with an AMD IOMMU. But
to be fair - that machine was running with Xen 4.1.

The hack I developed: http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
ends up with this:

(XEN) alloc_pdev: unknown type: 0000:08:00.0
(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
(XEN) [VT-D]iommu.c:1888: d0: context mapping failed

(FYI, this Xen 4.3.1)

Let me retry on the AMD box with the same version of Xen.

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 13:36                 ` Jan Beulich
@ 2013-09-12  6:20                   ` Zhang, Yang Z
  2013-12-11 18:32                     ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 40+ messages in thread
From: Zhang, Yang Z @ 2013-09-12  6:20 UTC (permalink / raw)
  To: Jan Beulich, Gordan Bobic; +Cc: xen-devel

Jan Beulich wrote on 2013-09-11:
>>>> On 11.09.13 at 15:26, Gordan Bobic <gordan@bobich.net> wrote:
>> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
>> <JBeulich@suse.com>
>>  wrote:
>>>>>> On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
>>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
>>>> <JBeulich@suse.com>
>>>>  wrote:
>>>>>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>>>>>  
>>>>>>  I'll try adding one of my LSI cards and see the comparative
>>>>>> behaviour. Right now I don't even know if the phantom device  is
>>>>>> on the SAS card or the motherboard.
>>>>> 
>>>>> The Adaptec card being the only thing on bus 0f makes it pretty
>>>>> likely that this other device also is on that card.
>>>>> 
>>>>> I guess the issue is mainly because the device itself is a PCI
>>>>> one, while the immediately upstream bridge (where I mean only the
>>>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>>>>> them. And as long as firmware doesn't know about that bridge and
>>>>> the bridge doesn't properly handle config space accesses to it,
>>>>> such a device just can't be used with an IOMMU (without some yet
>>>>> to be invented workaround).
>>>>> 
>>>>  I'm actually thinking about Konrad's proposed hack in that
>>>> thread from 3 years ago. If the device IDs are parameterized  out
>>>> rather than hard-coded, then this could work in nearly the  same
>>>> was as xen-pciback in terms of usage. Pass the phantom  device IDs
>>>> as parameters to the module. Done that way it  might even be
>>>> considered clean enough to be fit for public  consumption.
>>> 
>>> Except that, short of being able to determine it via config space
>>> reads, we also need the resulting command line option to tell us
>>> that what kind of device that is.
>>> 
>>  Not sure I follow. Why do we need to know the device type?
> 
> Just look at set_msi_source_id() as well as
> domain_context_{mapping,unmap}() (just the most prominent
> examples): Behavior here heavily depends on the type of the device
> itself _and_ that of the upstream bridge(s).
Looks like there are many devices are failed to work. I wonder whether the PCI/PCIe specification tells how to detect the hidden device behind those devices (Like detection of phantom device). If not, I think those devices are buggy. Or we can say those devices are not really PCI/PCIe compatible. Since VT-d only covers the PCI/PCIe device, it's reasonable that non-PCI/PCIe device failed to work under VT-d.

As Jan's suggestion, we need the user to tell us whether there is a hidden device or BDF behind anther device that the OS is unaware. We need to pass that info to Xen before pass-thought the device.

> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


Best regards,
Yang

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 13:26               ` Gordan Bobic
@ 2013-09-11 13:36                 ` Jan Beulich
  2013-09-12  6:20                   ` Zhang, Yang Z
  0 siblings, 1 reply; 40+ messages in thread
From: Jan Beulich @ 2013-09-11 13:36 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: xen-devel

 >>> On 11.09.13 at 15:26, Gordan Bobic <gordan@bobich.net> wrote:
> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich" <JBeulich@suse.com> 
>  wrote:
>>>>> On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich" 
>>> <JBeulich@suse.com>
>>>  wrote:
>>>>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>>>>
>>>>>  I'll try adding one of my LSI cards and see the comparative
>>>>>  behaviour. Right now I don't even know if the phantom device
>>>>>  is on the SAS card or the motherboard.
>>>>
>>>> The Adaptec card being the only thing on bus 0f makes it pretty
>>>> likely that this other device also is on that card.
>>>>
>>>> I guess the issue is mainly because the device itself is a PCI one,
>>>> while the immediately upstream bridge (where I mean only the
>>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>>>> them. And as long as firmware doesn't know about that bridge
>>>> and the bridge doesn't properly handle config space accesses to
>>>> it, such a device just can't be used with an IOMMU (without some
>>>> yet to be invented workaround).
>>>
>>>  I'm actually thinking about Konrad's proposed hack in that
>>>  thread from 3 years ago. If the device IDs are parameterized
>>>  out rather than hard-coded, then this could work in nearly the
>>>  same was as xen-pciback in terms of usage. Pass the phantom
>>>  device IDs as parameters to the module. Done that way it
>>>  might even be considered clean enough to be fit for public
>>>  consumption.
>>
>> Except that, short of being able to determine it via config
>> space reads, we also need the resulting command line option
>> to tell us that what kind of device that is.
> 
>  Not sure I follow. Why do we need to know the device type?

Just look at set_msi_source_id() as well as
domain_context_{mapping,unmap}() (just the most prominent
examples): Behavior here heavily depends on the type of the
device itself _and_ that of the upstream bridge(s).

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 13:23           ` Gordan Bobic
@ 2013-09-11 13:34             ` Jan Beulich
  0 siblings, 0 replies; 40+ messages in thread
From: Jan Beulich @ 2013-09-11 13:34 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: xen-devel

 >>> On 11.09.13 at 15:23, Gordan Bobic <gordan@bobich.net> wrote:
> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich" <JBeulich@suse.com> wrote:
>> I guess the issue is mainly because the device itself is a PCI one,
>> while the immediately upstream bridge (where I mean only the
>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>> them. And as long as firmware doesn't know about that bridge
>> and the bridge doesn't properly handle config space accesses to
>> it, such a device just can't be used with an IOMMU (without some
>> yet to be invented workaround).
> 
>  You don't think Konrad's patch on the old thread would work
>  for getting it working in dom0?

For just Dom0 this might suffice, but for passthrough it surely
won't. And a sufficiently generic workaround patch should deal
with both imo.

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 13:22             ` Jan Beulich
@ 2013-09-11 13:26               ` Gordan Bobic
  2013-09-11 13:36                 ` Jan Beulich
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-09-11 13:26 UTC (permalink / raw)
  To: xen-devel

 On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich" <JBeulich@suse.com> 
 wrote:
>>>> On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich" 
>> <JBeulich@suse.com>
>>  wrote:
>>>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>>>
>>>>  I'll try adding one of my LSI cards and see the comparative
>>>>  behaviour. Right now I don't even know if the phantom device
>>>>  is on the SAS card or the motherboard.
>>>
>>> The Adaptec card being the only thing on bus 0f makes it pretty
>>> likely that this other device also is on that card.
>>>
>>> I guess the issue is mainly because the device itself is a PCI one,
>>> while the immediately upstream bridge (where I mean only the
>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>>> them. And as long as firmware doesn't know about that bridge
>>> and the bridge doesn't properly handle config space accesses to
>>> it, such a device just can't be used with an IOMMU (without some
>>> yet to be invented workaround).
>>
>>  I'm actually thinking about Konrad's proposed hack in that
>>  thread from 3 years ago. If the device IDs are parameterized
>>  out rather than hard-coded, then this could work in nearly the
>>  same was as xen-pciback in terms of usage. Pass the phantom
>>  device IDs as parameters to the module. Done that way it
>>  might even be considered clean enough to be fit for public
>>  consumption.
>
> Except that, short of being able to determine it via config
> space reads, we also need the resulting command line option
> to tell us that what kind of device that is.

 Not sure I follow. Why do we need to know the device type?

 Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 13:03         ` Jan Beulich
  2013-09-11 13:10           ` Gordan Bobic
@ 2013-09-11 13:23           ` Gordan Bobic
  2013-09-11 13:34             ` Jan Beulich
  1 sibling, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-09-11 13:23 UTC (permalink / raw)
  To: xen-devel

 On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich" <JBeulich@suse.com> 
 wrote:
>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>
>>  I'll try adding one of my LSI cards and see the comparative
>>  behaviour. Right now I don't even know if the phantom device
>>  is on the SAS card or the motherboard.
>
> The Adaptec card being the only thing on bus 0f makes it pretty
> likely that this other device also is on that card.

 That's plausible, but given there are Nvidia NF200 bridges between
 the Intel 5520 PCIe hubs and each of the slots on my motherboard,
 it is not a 100% certainty.

> I guess the issue is mainly because the device itself is a PCI one,
> while the immediately upstream bridge (where I mean only the
> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
> them. And as long as firmware doesn't know about that bridge
> and the bridge doesn't properly handle config space accesses to
> it, such a device just can't be used with an IOMMU (without some
> yet to be invented workaround).

 You don't think Konrad's patch on the old thread would work
 for getting it working in dom0?

 Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 13:10           ` Gordan Bobic
@ 2013-09-11 13:22             ` Jan Beulich
  2013-09-11 13:26               ` Gordan Bobic
  0 siblings, 1 reply; 40+ messages in thread
From: Jan Beulich @ 2013-09-11 13:22 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: xen-devel

 >>> On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich" <JBeulich@suse.com> 
>  wrote:
>>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>>
>>>  I'll try adding one of my LSI cards and see the comparative
>>>  behaviour. Right now I don't even know if the phantom device
>>>  is on the SAS card or the motherboard.
>>
>> The Adaptec card being the only thing on bus 0f makes it pretty
>> likely that this other device also is on that card.
>>
>> I guess the issue is mainly because the device itself is a PCI one,
>> while the immediately upstream bridge (where I mean only the
>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>> them. And as long as firmware doesn't know about that bridge
>> and the bridge doesn't properly handle config space accesses to
>> it, such a device just can't be used with an IOMMU (without some
>> yet to be invented workaround).
> 
>  I'm actually thinking about Konrad's proposed hack in that
>  thread from 3 years ago. If the device IDs are parameterized
>  out rather than hard-coded, then this could work in nearly the
>  same was as xen-pciback in terms of usage. Pass the phantom
>  device IDs as parameters to the module. Done that way it
>  might even be considered clean enough to be fit for public
>  consumption.

Except that, short of being able to determine it via config
space reads, we also need the resulting command line option
to tell us that what kind of device that is.

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 13:03         ` Jan Beulich
@ 2013-09-11 13:10           ` Gordan Bobic
  2013-09-11 13:22             ` Jan Beulich
  2013-09-11 13:23           ` Gordan Bobic
  1 sibling, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-09-11 13:10 UTC (permalink / raw)
  To: xen-devel

 On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich" <JBeulich@suse.com> 
 wrote:
>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>
>>  I'll try adding one of my LSI cards and see the comparative
>>  behaviour. Right now I don't even know if the phantom device
>>  is on the SAS card or the motherboard.
>
> The Adaptec card being the only thing on bus 0f makes it pretty
> likely that this other device also is on that card.
>
> I guess the issue is mainly because the device itself is a PCI one,
> while the immediately upstream bridge (where I mean only the
> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
> them. And as long as firmware doesn't know about that bridge
> and the bridge doesn't properly handle config space accesses to
> it, such a device just can't be used with an IOMMU (without some
> yet to be invented workaround).

 I'm actually thinking about Konrad's proposed hack in that
 thread from 3 years ago. If the device IDs are parameterized
 out rather than hard-coded, then this could work in nearly the
 same was as xen-pciback in terms of usage. Pass the phantom
 device IDs as parameters to the module. Done that way it
 might even be considered clean enough to be fit for public
 consumption.

 Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 12:45       ` Gordan Bobic
@ 2013-09-11 13:03         ` Jan Beulich
  2013-09-11 13:10           ` Gordan Bobic
  2013-09-11 13:23           ` Gordan Bobic
  0 siblings, 2 replies; 40+ messages in thread
From: Jan Beulich @ 2013-09-11 13:03 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: xen-devel

>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
> 
>  I'll try adding one of my LSI cards and see the comparative
>  behaviour. Right now I don't even know if the phantom device
>  is on the SAS card or the motherboard.

The Adaptec card being the only thing on bus 0f makes it pretty
likely that this other device also is on that card.

I guess the issue is mainly because the device itself is a PCI one,
while the immediately upstream bridge (where I mean only the
visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
them. And as long as firmware doesn't know about that bridge
and the bridge doesn't properly handle config space accesses to
it, such a device just can't be used with an IOMMU (without some
yet to be invented workaround).

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 12:19       ` Gordan Bobic
@ 2013-09-11 12:56         ` Pasi Kärkkäinen
  0 siblings, 0 replies; 40+ messages in thread
From: Pasi Kärkkäinen @ 2013-09-11 12:56 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: xen-devel

On Wed, Sep 11, 2013 at 01:19:44PM +0100, Gordan Bobic wrote:
> On Wed, 11 Sep 2013 12:57:10 +0100, "Jan Beulich"
> <JBeulich@suse.com> wrote:
> >>>>On 11.09.13 at 13:44, Gordan Bobic <gordan@bobich.net> wrote:
> >>This got me thinking - if the problem is broken IOMMU
> >>implementation,
> >> is the IOMMU _actually_ required for PCI passthrough to HVM
> >> guests if all the memory holes and BARs are made exactly the same
> >> in dom0 and domU? If vBAR=pBAR, then surely there is no memory
> >> range remapping to be done anyway - which means that there
> >> is no need for the strict IOMMU requirements (over and above
> >> the requirements and caveats of PV domUs).
> >
> >But with this you ignore the need to handle device bus mastering
> >activities. In order to work without IOMMU, the guest's memory
> >addresses would also require guest-physical = machine-physical.
> 
> Hmm... that would be harder to achieve, mainly due to legacy
> stuff like base memory. But if (fingers crossed) DMA doesn't
> occur below 1MB, maybe the map can be bodged to emulate the
> addresses below 1MB, carve out as small a chunk of 32-bit
> memory as we can get away with for each guest, and map the
> rest with vmem=pmem in 64-bit memory range. But you are
> right - that gets way more complicated than I originally
> envisaged - all for the sake of supporting legacy
> and Steam boot-loader systems. :(
> 

I don't know the details but many years ago there was a patch
to allow PCI passthru to HVM guests without an IOMMU..

I think it's linked on Xen pcipassthrough wiki page.
It had a limitation that only one HVM guest was supported for PCI passthru.

-- Pasi

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 12:31     ` Jan Beulich
@ 2013-09-11 12:45       ` Gordan Bobic
  2013-09-11 13:03         ` Jan Beulich
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-09-11 12:45 UTC (permalink / raw)
  To: xen-devel

[-- Attachment #1: Type: text/plain, Size: 2860 bytes --]

 On Wed, 11 Sep 2013 13:31:06 +0100, "Jan Beulich" <JBeulich@suse.com> 
 wrote:
>>>> On 11.09.13 at 14:14, Gordan Bobic <gordan@bobich.net> wrote:
>> On Wed, 11 Sep 2013 12:53:09 +0100, "Jan Beulich" 
>> <JBeulich@suse.com>
>>  wrote:
>>>>>> On 11.09.13 at 13:05, Gordan Bobic <gordan@bobich.net> wrote:
>>>> I found this:
>>>>
>>>>  
>>>> http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>>>>
>>>>  while looking for a solution to a similar problem. I am
>>>>  facing a similar issue with LSI (8408E, 3081E-R) and
>>>>  Adaptec (31605) SAS cards. Was there ever a proper, more general
>>>>  fix or workaround for this issue?
>>>>
>>>>  These SAS cards experience these problems in dom0. When running
>>>>  a vanilla kernel on bare metal, they work OK without intel_iommu
>>>>  set. As soon as I set intel_iommu, the same thing happens (on
>>>>  bare metal, not dom0).
>>>>
>>>>  Clearly there is something badly broken with multiple layers
>>>>  of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>>>>  root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)
>>>
>>> The link above has some (hackish) workarounds - did you try
>>> them?
>>
>>  Not yet. The thing that bothers me is that he workaround
>>  involves hard-coding the PCI device ID which is _nasty_
>>  and unstable.
>
> I said "hackish", didn't I? Of course such a change would not
> have the slightest chance of going into any repo. But knowing
> whether it helps may allow thinking of an acceptable
> workaround.

 Indeed.

>>> In any event, seeing a hypervisor log with "iommu=debug" might
>>> shed further light on this: For one, we might be able to see which
>>> exact devices are present in the ACPI tables. And we would see
>>> which device(s) eventual faults originate from.
>>
>>  The thing that bothers me is that this happens in dom0 even
>>  with iommu=dom0-passthrough being set.
>>  iommu=dom0-passthrough,workaround_bios_bug doesn't help,
>>  either
>
> They're not meant to deal with this sort of an impossible (in
> theory) situation.

 It turns out that theoretically impossible things happen a lot
 more often than expected. :)

>>  And lo and behold, I do have phantom PCI devices after all!
>>  lspci shows no device with ID 0000:0f:01.0
>
> Not exactly: Phantom functions can't be at function 0. Irrespective
> of that - do the device coordinates somehow correlate with the
> problematic controller (IOW: lspci output and a full log would help)?

 Not necessarily a function of the same device - as I said, there is
 a PCIe bridge on the card, so there could be multiple devices hiding
 there.

 dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.

 I'll try adding one of my LSI cards and see the comparative
 behaviour. Right now I don't even know if the phantom device
 is on the SAS card or the motherboard.

 Gordan

[-- Attachment #2: dmesg.log --]
[-- Type: text/x-c++, Size: 71929 bytes --]

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 3.9.9-2.el6xen.x86_64 (root@normandy) (gcc version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Tue Jul 16 15:52:11 BST 2013
Command line: ro root=UUID=ffff5c41-4a21-4187-8af3-349580349117 rd_MD_UUID=6e517721:c6c545b7:e1c91b53:b7f83056 rd_MD_UUID=8c53a5dc:97b78841:cdbf9d19:0c4acac9 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk selinux=0 elevator=deadline video=DVI-I-1:edid/1920x2400 video=DVI-I-2:1920x2400 video=DIN-1:d quiet
Freeing 99-100 pfn range: 103 pages freed
1-1 mapping on 99->100
Freeing 3f790-100000 pfn range: 788592 pages freed
1-1 mapping on 3f790->100000
Released 788695 pages of unused memory
Set 788695 page(s) to 1-1 mapping
Populating bcf911-c901e8 pfn range: 788695 pages added
e820: BIOS-provided physical RAM map:
Xen: [mem 0x0000000000000000-0x0000000000098fff] usable
Xen: [mem 0x0000000000099800-0x00000000000fffff] reserved
Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable
Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data
Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS
Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved
Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved
Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable
NX (Execute Disable) protection: active
SMBIOS 2.6 present.
DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./EVGA Classified SR-2, BIOS 080016  04/24/2013
e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
e820: remove [mem 0x000a0000-0x000fffff] usable
No AGP bridge found
e820: last_pfn = 0xcc0000 max_arch_pfn = 0x400000000
e820: last_pfn = 0x3f790 max_arch_pfn = 0x400000000
Base memory trampoline at [ffff880000092000] 92000 size 28672
init_memory_mapping: [mem 0x00000000-0x000fffff]
 [mem 0x00000000-0x000fffff] page 4k
init_memory_mapping: [mem 0xc8fe00000-0xc8fffffff]
 [mem 0xc8fe00000-0xc8fffffff] page 4k
BRK [0x01bc0000, 0x01bc0fff] PGTABLE
BRK [0x01bc1000, 0x01bc1fff] PGTABLE
init_memory_mapping: [mem 0xc8c000000-0xc8fdfffff]
 [mem 0xc8c000000-0xc8fdfffff] page 4k
BRK [0x01bc2000, 0x01bc2fff] PGTABLE
BRK [0x01bc3000, 0x01bc3fff] PGTABLE
BRK [0x01bc4000, 0x01bc4fff] PGTABLE
init_memory_mapping: [mem 0xc80000000-0xc8bffffff]
 [mem 0xc80000000-0xc8bffffff] page 4k
init_memory_mapping: [mem 0x00100000-0x3f78ffff]
 [mem 0x00100000-0x3f78ffff] page 4k
init_memory_mapping: [mem 0x100000000-0xc7fffffff]
 [mem 0x100000000-0xc7fffffff] page 4k
init_memory_mapping: [mem 0xc90000000-0xcbfffffff]
 [mem 0xc90000000-0xcbfffffff] page 4k
RAMDISK: [mem 0x01f70000-0x0626afff]
ACPI: RSDP 00000000000f9f70 00024 (v02 ACPIAM)
ACPI: XSDT 000000003f790100 00064 (v01 042413 XSDT1438 20130424 MSFT 00000097)
ACPI: FACP 000000003f790290 000F4 (v04 042413 FACP1438 20130424 MSFT 00000097)
ACPI: DSDT 000000003f7904f0 058A3 (v02  1W555 1W555A58 00000A58 INTL 20051117)
ACPI: FACS 000000003f79e000 00040
ACPI: APIC 000000003f790390 00118 (v02 042413 APIC1438 20130424 MSFT 00000097)
ACPI: MCFG 000000003f7904b0 0003C (v01 042413 OEMMCFG  20130424 MSFT 00000097)
ACPI: OEMB 000000003f79e040 00082 (v01 042413 OEMB1438 20130424 MSFT 00000097)
ACPI: SRAT 000000003f79a4f0 00250 (v02 042413 OEMSRAT  00000001 INTL 00000001)
ACPI: HPET 000000003f79a740 00038 (v01 042413 OEMHPET  20130424 MSFT 00000097)
ACPI: XMAR 000000003f79e0d0 00120 (v01    AMI  OEMDMAR 00000001 MSFT 00000097)
ACPI: SSDT 000000003f7a4c70 00363 (v01 DpgPmm    CpuPm 00000012 INTL 20051117)
ACPI: Local APIC address 0xfee00000
NUMA turned off
Faking a node at [mem 0x0000000000000000-0x0000000cbfffffff]
Initmem setup node 0 [mem 0x00000000-0xcbfffffff]
  NODE_DATA [mem 0xc901c1000-0xc901e7fff]
Zone ranges:
  DMA      [mem 0x00001000-0x00ffffff]
  DMA32    [mem 0x01000000-0xffffffff]
  Normal   [mem 0x100000000-0xcbfffffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x00001000-0x00098fff]
  node   0: [mem 0x00100000-0x3f78ffff]
  node   0: [mem 0x100000000-0xcbfffffff]
On node 0 totalpages: 12580648
  DMA zone: 56 pages used for memmap
  DMA zone: 22 pages reserved
  DMA zone: 3992 pages, LIFO batch:0
  DMA32 zone: 3499 pages used for memmap
  DMA32 zone: 255888 pages, LIFO batch:31
  Normal zone: 168448 pages used for memmap
  Normal zone: 12320768 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x10] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x12] enabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x20] enabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x22] enabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x24] enabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x30] enabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x32] enabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x34] enabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x03] enabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x05] enabled)
ACPI: LAPIC (acpi_id[0x10] lapic_id[0x11] enabled)
ACPI: LAPIC (acpi_id[0x11] lapic_id[0x13] enabled)
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x15] enabled)
ACPI: LAPIC (acpi_id[0x13] lapic_id[0x21] enabled)
ACPI: LAPIC (acpi_id[0x14] lapic_id[0x23] enabled)
ACPI: LAPIC (acpi_id[0x15] lapic_id[0x25] enabled)
ACPI: LAPIC (acpi_id[0x16] lapic_id[0x31] enabled)
ACPI: LAPIC (acpi_id[0x17] lapic_id[0x33] enabled)
ACPI: LAPIC (acpi_id[0x18] lapic_id[0x35] enabled)
ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 6, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x07] address[0xfec8a000] gsi_base[24])
IOAPIC[1]: apic_id 7, version 32, address 0xfec8a000, GSI 24-47
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 20 low level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0xffffffff base: 0xfed00000
smpboot: Allowing 24 CPUs, 0 hotplug CPUs
nr_irqs_gsi: 64
PM: Registered nosave memory: 0000000000099000 - 000000000009a000
PM: Registered nosave memory: 000000000009a000 - 0000000000100000
PM: Registered nosave memory: 000000003f790000 - 000000003f79e000
PM: Registered nosave memory: 000000003f79e000 - 000000003f7d0000
PM: Registered nosave memory: 000000003f7d0000 - 000000003f7e0000
PM: Registered nosave memory: 000000003f7e0000 - 000000003f7e7000
PM: Registered nosave memory: 000000003f7e7000 - 0000000040000000
PM: Registered nosave memory: 0000000040000000 - 00000000fee00000
PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
PM: Registered nosave memory: 00000000fee01000 - 00000000ffc00000
PM: Registered nosave memory: 00000000ffc00000 - 0000000100000000
e820: [mem 0x40000000-0xfedfffff] available for PCI devices
Booting paravirtualized kernel on Xen
Xen version: 4.3.0 (preserve-AD)
setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:24 nr_node_ids:1
PERCPU: Embedded 28 pages/cpu @ffff880c83c00000 s84032 r8192 d22464 u131072
pcpu-alloc: s84032 r8192 d22464 u131072 alloc=1*2097152
pcpu-alloc: [0] 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 
pcpu-alloc: [0] 16 17 18 19 20 21 22 23 -- -- -- -- -- -- -- -- 
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 12408623
Policy zone: Normal
Kernel command line: ro root=UUID=ffff5c41-4a21-4187-8af3-349580349117 rd_MD_UUID=6e517721:c6c545b7:e1c91b53:b7f83056 rd_MD_UUID=8c53a5dc:97b78841:cdbf9d19:0c4acac9 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk selinux=0 elevator=deadline video=DVI-I-1:edid/1920x2400 video=DVI-I-2:1920x2400 video=DIN-1:d quiet
PID hash table entries: 4096 (order: 3, 32768 bytes)
__ex_table already sorted, skipping sort
software IO TLB [mem 0xc55800000-0xc59800000] (64MB) mapped at [ffff880c55800000-ffff880c597fffff]
Memory: 48503516k/53477376k available (5525k kernel code, 3154784k absent, 1819076k reserved, 2610k data, 1732k init)
Hierarchical RCU implementation.
	RCU restricting CPUs from NR_CPUS=32 to nr_cpu_ids=24.
NR_IRQS:4352 nr_irqs:1280 16
xen: sci override: global_irq=20 trigger=0 polarity=1
xen: registering gsi 20 triggering 0 polarity 1
xen: --> pirq=20 -> irq=9 (gsi=9)
xen: acpi sci 9
xen: --> pirq=1 -> irq=1 (gsi=1)
xen: --> pirq=2 -> irq=2 (gsi=2)
xen: --> pirq=3 -> irq=3 (gsi=3)
xen: --> pirq=4 -> irq=4 (gsi=4)
xen: --> pirq=5 -> irq=5 (gsi=5)
xen: --> pirq=6 -> irq=6 (gsi=6)
xen: --> pirq=7 -> irq=7 (gsi=7)
xen: --> pirq=8 -> irq=8 (gsi=8)
xen: --> pirq=10 -> irq=10 (gsi=10)
xen: --> pirq=11 -> irq=11 (gsi=11)
xen: --> pirq=12 -> irq=12 (gsi=12)
xen: --> pirq=13 -> irq=13 (gsi=13)
xen: --> pirq=14 -> irq=14 (gsi=14)
xen: --> pirq=15 -> irq=15 (gsi=15)
Console: colour VGA+ 80x25
console [tty0] enabled
Xen: using vcpuop timer interface
installing Xen timer for CPU 0
tsc: Detected 3321.718 MHz processor
Calibrating delay loop (skipped), value calculated using timer frequency.. 6643.43 BogoMIPS (lpj=3321718)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Disabled at boot.
Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys cpuacct
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
mce: CPU supports 2 MCE banks
Last level iTLB entries: 4KB 512, 2MB 7, 4MB 7
Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
tlb_flushall_shift: 6
Freeing SMP alternatives: 20k freed
ACPI: Core revision 20130117
ACPI: All ACPI Tables successfully acquired
ftrace: allocating 21258 entries in 84 pages
Performance Events: unsupported p6 CPU model 44 no PMU driver, software events only.
installing Xen timer for CPU 1
installing Xen timer for CPU 2
installing Xen timer for CPU 3
installing Xen timer for CPU 4
installing Xen timer for CPU 5
installing Xen timer for CPU 6
installing Xen timer for CPU 7
installing Xen timer for CPU 8
installing Xen timer for CPU 9
installing Xen timer for CPU 10
installing Xen timer for CPU 11
installing Xen timer for CPU 12
installing Xen timer for CPU 13
installing Xen timer for CPU 14
installing Xen timer for CPU 15
installing Xen timer for CPU 16
installing Xen timer for CPU 17
installing Xen timer for CPU 18
installing Xen timer for CPU 19
installing Xen timer for CPU 20
installing Xen timer for CPU 21
installing Xen timer for CPU 22
installing Xen timer for CPU 23
Brought up 24 CPUs
devtmpfs: initialized
PM: Registering ACPI NVS region [mem 0x3f79e000-0x3f7cffff] (204800 bytes)
Grant tables using version 2 layout.
Grant table initialized
regulator-dummy: no parameters
NET: Registered protocol family 16
ACPI: bus type PCI registered
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
PCI: not using MMCONFIG
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
ACPI: EC: Look up EC in DSDT
ACPI: Executed 1 blocks of module-level executable AML code
ACPI: SSDT 000000003f79e1f0 053DC (v01 DpgPmm  P001Ist 00000011 INTL 20051117)
ACPI: Dynamic OEM Table Load:
ACPI: SSDT           (null) 053DC (v01 DpgPmm  P001Ist 00000011 INTL 20051117)
ACPI: SSDT 000000003f7a35d0 00C88 (v01  PmRef  P001Cst 00003001 INTL 20051117)
ACPI: Dynamic OEM Table Load:
ACPI: SSDT           (null) 00C88 (v01  PmRef  P001Cst 00003001 INTL 20051117)
ACPI: SSDT 000000003f7a4260 00A0A (v01  PmRef  Cpu0Tst 00003000 INTL 20051117)
ACPI: Dynamic OEM Table Load:
ACPI: SSDT           (null) 00A0A (v01  PmRef  Cpu0Tst 00003000 INTL 20051117)
ACPI: Interpreter enabled
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130117/hwxface-568)
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20130117/hwxface-568)
ACPI: (supports S0 S1 S4 S5)
ACPI: Using IOAPIC for interrupt routing
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources
PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [bus 00-ff]
pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000dffff]
pci_bus 0000:00: root bus resource [mem 0x40000000-0xdfffffff]
pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfed8ffff]
pci 0000:00:00.0: [8086:3406] type 00 class 0x060000
pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
pci 0000:00:02.0: [8086:3409] type 01 class 0x060400
pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
pci 0000:00:02.0: System wakeup disabled by ACPI
pci 0000:00:03.0: [8086:340a] type 01 class 0x060400
pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
pci 0000:00:03.0: System wakeup disabled by ACPI
pci 0000:00:07.0: [8086:340e] type 01 class 0x060400
pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
pci 0000:00:07.0: System wakeup disabled by ACPI
pci 0000:00:13.0: [8086:342d] type 00 class 0x080020
pci 0000:00:13.0: reg 10: [mem 0xfec8a000-0xfec8afff]
pci 0000:00:13.0: PME# supported from D0 D3hot D3cold
pci 0000:00:14.0: [8086:342e] type 00 class 0x080000
pci 0000:00:14.1: [8086:3422] type 00 class 0x080000
pci 0000:00:14.2: [8086:3423] type 00 class 0x080000
pci 0000:00:14.3: [8086:3438] type 00 class 0x080000
pci 0000:00:1a.0: [8086:3a37] type 00 class 0x0c0300
pci 0000:00:1a.0: reg 20: [io  0x9980-0x999f]
pci 0000:00:1a.0: System wakeup disabled by ACPI
pci 0000:00:1a.1: [8086:3a38] type 00 class 0x0c0300
pci 0000:00:1a.1: reg 20: [io  0x9a00-0x9a1f]
pci 0000:00:1a.1: System wakeup disabled by ACPI
pci 0000:00:1a.2: [8086:3a39] type 00 class 0x0c0300
pci 0000:00:1a.2: reg 20: [io  0x9a80-0x9a9f]
pci 0000:00:1a.2: System wakeup disabled by ACPI
pci 0000:00:1a.7: [8086:3a3c] type 00 class 0x0c0320
pci 0000:00:1a.7: reg 10: [mem 0xf3df8000-0xf3df83ff]
pci 0000:00:1a.7: PME# supported from D0 D3hot D3cold
pci 0000:00:1a.7: System wakeup disabled by ACPI
pci 0000:00:1b.0: [8086:3a3e] type 00 class 0x040300
pci 0000:00:1b.0: reg 10: [mem 0xf3df4000-0xf3df7fff 64bit]
pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.0: [8086:3a40] type 01 class 0x060400
pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.0: System wakeup disabled by ACPI
pci 0000:00:1c.2: [8086:3a44] type 01 class 0x060400
pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.2: System wakeup disabled by ACPI
pci 0000:00:1c.4: [8086:3a48] type 01 class 0x060400
pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.4: System wakeup disabled by ACPI
pci 0000:00:1d.0: [8086:3a34] type 00 class 0x0c0300
pci 0000:00:1d.0: reg 20: [io  0x9b00-0x9b1f]
pci 0000:00:1d.0: System wakeup disabled by ACPI
pci 0000:00:1d.1: [8086:3a35] type 00 class 0x0c0300
pci 0000:00:1d.1: reg 20: [io  0x9b80-0x9b9f]
pci 0000:00:1d.1: System wakeup disabled by ACPI
pci 0000:00:1d.2: [8086:3a36] type 00 class 0x0c0300
pci 0000:00:1d.2: reg 20: [io  0x9c00-0x9c1f]
pci 0000:00:1d.2: System wakeup disabled by ACPI
pci 0000:00:1d.7: [8086:3a3a] type 00 class 0x0c0320
pci 0000:00:1d.7: reg 10: [mem 0xf3dfa000-0xf3dfa3ff]
pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
pci 0000:00:1d.7: System wakeup disabled by ACPI
pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
pci 0000:00:1e.0: System wakeup disabled by ACPI
pci 0000:00:1f.0: [8086:3a16] type 00 class 0x060100
pci 0000:00:1f.0: ICH7 LPC Generic IO decode 1 PIO at 0a00 (mask 00ff)
pci 0000:00:1f.2: [8086:3a22] type 00 class 0x010601
pci 0000:00:1f.2: reg 10: [io  0x9d00-0x9d07]
pci 0000:00:1f.2: reg 14: [io  0x9f00-0x9f03]
pci 0000:00:1f.2: reg 18: [io  0x9e80-0x9e87]
pci 0000:00:1f.2: reg 1c: [io  0x9e00-0x9e03]
pci 0000:00:1f.2: reg 20: [io  0x9d80-0x9d9f]
pci 0000:00:1f.2: reg 24: [mem 0xf3dfc000-0xf3dfc7ff]
pci 0000:00:1f.2: PME# supported from D3hot
pci 0000:00:1f.3: [8086:3a30] type 00 class 0x0c0500
pci 0000:00:1f.3: reg 10: [mem 0xf3dfe000-0xf3dfe0ff 64bit]
pci 0000:00:1f.3: reg 20: [io  0x0400-0x041f]
pci 0000:11:00.0: [1033:0194] type 00 class 0x0c0330
pci 0000:11:00.0: reg 10: [mem 0xfbefe000-0xfbefffff 64bit]
pci 0000:11:00.0: PME# supported from D0 D3hot
pci 0000:00:02.0: PCI bridge to [bus 11]
pci 0000:00:02.0:   bridge window [mem 0xfbe00000-0xfbefffff]
pci 0000:09:00.0: [10de:05b1] type 01 class 0x060400
pci 0000:09:00.0: PME# supported from D0 D3hot D3cold
pci 0000:00:03.0: PCI bridge to [bus 09-10]
pci 0000:00:03.0:   bridge window [io  0xe000-0xefff]
pci 0000:00:03.0:   bridge window [mem 0xd7b00000-0xdfffffff]
pci 0000:00:03.0:   bridge window [mem 0xc0000000-0xcfffffff 64bit pref]
pci 0000:0a:00.0: [10de:05b1] type 01 class 0x060400
pci 0000:0a:00.0: PME# supported from D0 D3hot D3cold
pci 0000:0a:02.0: [10de:05b1] type 01 class 0x060400
pci 0000:0a:02.0: PME# supported from D0 D3hot D3cold
pci 0000:0a:03.0: [10de:05b1] type 01 class 0x060400
pci 0000:0a:03.0: PME# supported from D0 D3hot D3cold
pci 0000:09:00.0: PCI bridge to [bus 0a-10]
pci 0000:09:00.0:   bridge window [io  0xe000-0xefff]
pci 0000:09:00.0:   bridge window [mem 0xd7b00000-0xdfffffff]
pci 0000:09:00.0:   bridge window [mem 0xc0000000-0xcfffffff 64bit pref]
pci 0000:10:00.0: [1002:9442] type 00 class 0x030000
pci 0000:10:00.0: reg 10: [mem 0xc0000000-0xcfffffff 64bit pref]
pci 0000:10:00.0: reg 18: [mem 0xd8000000-0xd800ffff 64bit]
pci 0000:10:00.0: reg 20: [io  0xe000-0xe0ff]
pci 0000:10:00.0: reg 30: [mem 0xd7fe0000-0xd7ffffff pref]
pci 0000:10:00.0: supports D1 D2
pci 0000:10:00.1: [1002:aa30] type 00 class 0x040300
pci 0000:10:00.1: reg 10: [mem 0xdc000000-0xdc003fff 64bit]
pci 0000:10:00.1: supports D1 D2
pci 0000:0a:00.0: PCI bridge to [bus 10]
pci 0000:0a:00.0:   bridge window [io  0xe000-0xefff]
pci 0000:0a:00.0:   bridge window [mem 0xd7f00000-0xdfffffff]
pci 0000:0a:00.0:   bridge window [mem 0xc0000000-0xcfffffff 64bit pref]
pci 0000:0d:00.0: [8086:0370] type 01 class 0x060400
pci 0000:0d:00.0: PME# supported from D0 D3hot D3cold
pci 0000:0d:00.2: [8086:0372] type 01 class 0x060400
pci 0000:0d:00.2: PME# supported from D0 D3hot D3cold
pci 0000:0d:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
pci 0000:0a:02.0: PCI bridge to [bus 0d-0f]
pci 0000:0a:02.0:   bridge window [mem 0xd7c00000-0xd7efffff]
pci 0000:0f:0e.0: [9005:0285] type 00 class 0x010400
pci 0000:0f:0e.0: reg 10: [mem 0xd7c00000-0xd7dfffff 64bit]
pci 0000:0f:0e.0: reg 30: [mem 0xd7e80000-0xd7efffff pref]
pci 0000:0f:0e.0: supports D1
pci 0000:0d:00.0: PCI bridge to [bus 0f]
pci 0000:0d:00.0:   bridge window [mem 0xd7c00000-0xd7efffff]
pci 0000:0d:00.2: PCI bridge to [bus 0e]
pci 0000:0b:00.0: [1102:7006] type 01 class 0x060400
pci 0000:0b:00.0: supports D1 D2
pci 0000:0b:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
pci 0000:0a:03.0: PCI bridge to [bus 0b-0c]
pci 0000:0a:03.0:   bridge window [mem 0xd7b00000-0xd7bfffff]
pci 0000:0c:00.0: [1102:0009] type 00 class 0x040300
pci 0000:0c:00.0: reg 10: [mem 0xd7bfc000-0xd7bfffff]
pci 0000:0c:00.0: supports D1 D2
pci 0000:0b:00.0: PCI bridge to [bus 0c]
pci 0000:0b:00.0:   bridge window [mem 0xd7b00000-0xd7bfffff]
pci 0000:05:00.0: [10de:05b1] type 01 class 0x060400
pci 0000:05:00.0: PME# supported from D0 D3hot D3cold
pci 0000:00:07.0: PCI bridge to [bus 05-08]
pci 0000:00:07.0:   bridge window [io  0xc000-0xdfff]
pci 0000:00:07.0:   bridge window [mem 0xf4000000-0xfbdfffff]
pci 0000:00:07.0:   bridge window [mem 0xa8000000-0xbfffffff 64bit pref]
pci 0000:06:00.0: [10de:05b1] type 01 class 0x060400
pci 0000:06:00.0: PME# supported from D0 D3hot D3cold
pci 0000:06:02.0: [10de:05b1] type 01 class 0x060400
pci 0000:06:02.0: PME# supported from D0 D3hot D3cold
pci 0000:05:00.0: PCI bridge to [bus 06-08]
pci 0000:05:00.0:   bridge window [io  0xc000-0xdfff]
pci 0000:05:00.0:   bridge window [mem 0xf4000000-0xfbdfffff]
pci 0000:05:00.0:   bridge window [mem 0xa8000000-0xbfffffff 64bit pref]
pci 0000:08:00.0: [10de:06d8] type 00 class 0x030000
pci 0000:08:00.0: reg 10: [mem 0xf8000000-0xf9ffffff]
pci 0000:08:00.0: reg 14: [mem 0xb8000000-0xbfffffff 64bit pref]
pci 0000:08:00.0: reg 1c: [mem 0xb4000000-0xb7ffffff 64bit pref]
pci 0000:08:00.0: reg 24: [io  0xdf80-0xdfff]
pci 0000:08:00.0: reg 30: [mem 0xfbd00000-0xfbd7ffff pref]
pci 0000:08:00.1: [10de:0be5] type 00 class 0x040300
pci 0000:08:00.1: reg 10: [mem 0xfbdfc000-0xfbdfffff]
pci 0000:06:00.0: PCI bridge to [bus 08]
pci 0000:06:00.0:   bridge window [io  0xd000-0xdfff]
pci 0000:06:00.0:   bridge window [mem 0xf8000000-0xfbdfffff]
pci 0000:06:00.0:   bridge window [mem 0xb4000000-0xbfffffff 64bit pref]
pci 0000:07:00.0: [10de:06d9] type 00 class 0x030000
pci 0000:07:00.0: reg 10: [mem 0xf4000000-0xf5ffffff]
pci 0000:07:00.0: reg 14: [mem 0xa8000000-0xafffffff 64bit pref]
pci 0000:07:00.0: reg 1c: [mem 0xb0000000-0xb3ffffff 64bit pref]
pci 0000:07:00.0: reg 24: [io  0xcf80-0xcfff]
pci 0000:07:00.0: reg 30: [mem 0xf7f00000-0xf7f7ffff pref]
pci 0000:07:00.1: [10de:0be5] type 00 class 0x040300
pci 0000:07:00.1: reg 10: [mem 0xf7ffc000-0xf7ffffff]
pci 0000:06:02.0: PCI bridge to [bus 07]
pci 0000:06:02.0:   bridge window [io  0xc000-0xcfff]
pci 0000:06:02.0:   bridge window [mem 0xf4000000-0xf7ffffff]
pci 0000:06:02.0:   bridge window [mem 0xa8000000-0xb3ffffff 64bit pref]
pci 0000:00:1c.0: PCI bridge to [bus 04]
pci 0000:03:00.0: [11ab:4380] type 00 class 0x020000
pci 0000:03:00.0: reg 10: [mem 0xf3fdc000-0xf3fdffff 64bit]
pci 0000:03:00.0: reg 18: [io  0xbf00-0xbfff]
pci 0000:03:00.0: reg 30: [mem 0xf3fe0000-0xf3ffffff pref]
pci 0000:03:00.0: supports D1 D2
pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:00:1c.2: PCI bridge to [bus 03]
pci 0000:00:1c.2:   bridge window [io  0xb000-0xbfff]
pci 0000:00:1c.2:   bridge window [mem 0xf3f00000-0xf3ffffff]
pci 0000:02:00.0: [11ab:4380] type 00 class 0x020000
pci 0000:02:00.0: reg 10: [mem 0xf3edc000-0xf3edffff 64bit]
pci 0000:02:00.0: reg 18: [io  0xaf00-0xafff]
pci 0000:02:00.0: reg 30: [mem 0xf3ee0000-0xf3efffff pref]
pci 0000:02:00.0: supports D1 D2
pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:00:1c.4: PCI bridge to [bus 02]
pci 0000:00:1c.4:   bridge window [io  0xa000-0xafff]
pci 0000:00:1c.4:   bridge window [mem 0xf3e00000-0xf3efffff]
pci 0000:00:1e.0: PCI bridge to [bus 01] (subtractive decode)
pci 0000:00:1e.0:   bridge window [io  0x0000-0x0cf7] (subtractive decode)
pci 0000:00:1e.0:   bridge window [io  0x0d00-0xffff] (subtractive decode)
pci 0000:00:1e.0:   bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
pci 0000:00:1e.0:   bridge window [mem 0x000d0000-0x000dffff] (subtractive decode)
pci 0000:00:1e.0:   bridge window [mem 0x40000000-0xdfffffff] (subtractive decode)
pci 0000:00:1e.0:   bridge window [mem 0xf0000000-0xfed8ffff] (subtractive decode)
acpi PNP0A08:00: Requesting ACPI _OSC control (0x1d)
acpi PNP0A08:00: ACPI _OSC control (0x1d) granted
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs *5)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 6 7 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 6 7 10 11 12 14 *15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 6 *7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs *3 4 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 6 7 10 11 12 *14 15)
ACPI: Enabled 1 GPEs in block 00 to 3F
acpi root: \_SB_.PCI0 notify handler is installed
Found 1 acpi root devices
ACPI: No dock devices found.
xen/balloon: Initialising balloon driver.
xen-balloon: Initialising balloon driver.
xen/balloon: Xen selfballooning driver disabled for domain0.
vgaarb: device added: PCI:0000:10:00.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: device added: PCI:0000:08:00.0,decodes=io+mem,owns=none,locks=none
vgaarb: device added: PCI:0000:07:00.0,decodes=io+mem,owns=none,locks=none
vgaarb: loaded
vgaarb: bridge control possible 0000:07:00.0
vgaarb: bridge control possible 0000:08:00.0
vgaarb: bridge control possible 0000:10:00.0
SCSI subsystem initialized
ACPI: bus type ATA registered
libata version 3.00 loaded.
ACPI: bus type USB registered
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: Discovered peer bus fe
PCI: root bus fe: using default resources
PCI: Probing PCI hardware (bus fe)
PCI host bridge to bus 0000:fe
pci_bus 0000:fe: root bus resource [io  0x0000-0xffff]
pci_bus 0000:fe: root bus resource [mem 0x00000000-0xffffffffff]
pci_bus 0000:fe: No busn resource found for root bus, will use [bus fe-ff]
pci 0000:fe:00.0: [8086:2c70] type 00 class 0x060000
pci 0000:fe:00.1: [8086:2d81] type 00 class 0x060000
pci 0000:fe:02.0: [8086:2d90] type 00 class 0x060000
pci 0000:fe:02.1: [8086:2d91] type 00 class 0x060000
pci 0000:fe:02.2: [8086:2d92] type 00 class 0x060000
pci 0000:fe:02.3: [8086:2d93] type 00 class 0x060000
pci 0000:fe:02.4: [8086:2d94] type 00 class 0x060000
pci 0000:fe:02.5: [8086:2d95] type 00 class 0x060000
pci 0000:fe:03.0: [8086:2d98] type 00 class 0x060000
pci 0000:fe:03.1: [8086:2d99] type 00 class 0x060000
pci 0000:fe:03.2: [8086:2d9a] type 00 class 0x060000
pci 0000:fe:03.4: [8086:2d9c] type 00 class 0x060000
pci 0000:fe:04.0: [8086:2da0] type 00 class 0x060000
pci 0000:fe:04.1: [8086:2da1] type 00 class 0x060000
pci 0000:fe:04.2: [8086:2da2] type 00 class 0x060000
pci 0000:fe:04.3: [8086:2da3] type 00 class 0x060000
pci 0000:fe:05.0: [8086:2da8] type 00 class 0x060000
pci 0000:fe:05.1: [8086:2da9] type 00 class 0x060000
pci 0000:fe:05.2: [8086:2daa] type 00 class 0x060000
pci 0000:fe:05.3: [8086:2dab] type 00 class 0x060000
pci 0000:fe:06.0: [8086:2db0] type 00 class 0x060000
pci 0000:fe:06.1: [8086:2db1] type 00 class 0x060000
pci 0000:fe:06.2: [8086:2db2] type 00 class 0x060000
pci 0000:fe:06.3: [8086:2db3] type 00 class 0x060000
pci_bus 0000:fe: busn_res: [bus fe-ff] end is updated to fe
PCI: Discovered peer bus ff
PCI: root bus ff: using default resources
PCI: Probing PCI hardware (bus ff)
PCI host bridge to bus 0000:ff
pci_bus 0000:ff: root bus resource [io  0x0000-0xffff]
pci_bus 0000:ff: root bus resource [mem 0x00000000-0xffffffffff]
pci_bus 0000:ff: No busn resource found for root bus, will use [bus ff-ff]
pci 0000:ff:00.0: [8086:2c70] type 00 class 0x060000
pci 0000:ff:00.1: [8086:2d81] type 00 class 0x060000
pci 0000:ff:02.0: [8086:2d90] type 00 class 0x060000
pci 0000:ff:02.1: [8086:2d91] type 00 class 0x060000
pci 0000:ff:02.2: [8086:2d92] type 00 class 0x060000
pci 0000:ff:02.3: [8086:2d93] type 00 class 0x060000
pci 0000:ff:02.4: [8086:2d94] type 00 class 0x060000
pci 0000:ff:02.5: [8086:2d95] type 00 class 0x060000
pci 0000:ff:03.0: [8086:2d98] type 00 class 0x060000
pci 0000:ff:03.1: [8086:2d99] type 00 class 0x060000
pci 0000:ff:03.2: [8086:2d9a] type 00 class 0x060000
pci 0000:ff:03.4: [8086:2d9c] type 00 class 0x060000
pci 0000:ff:04.0: [8086:2da0] type 00 class 0x060000
pci 0000:ff:04.1: [8086:2da1] type 00 class 0x060000
pci 0000:ff:04.2: [8086:2da2] type 00 class 0x060000
pci 0000:ff:04.3: [8086:2da3] type 00 class 0x060000
pci 0000:ff:05.0: [8086:2da8] type 00 class 0x060000
pci 0000:ff:05.1: [8086:2da9] type 00 class 0x060000
pci 0000:ff:05.2: [8086:2daa] type 00 class 0x060000
pci 0000:ff:05.3: [8086:2dab] type 00 class 0x060000
pci 0000:ff:06.0: [8086:2db0] type 00 class 0x060000
pci 0000:ff:06.1: [8086:2db1] type 00 class 0x060000
pci 0000:ff:06.2: [8086:2db2] type 00 class 0x060000
pci 0000:ff:06.3: [8086:2db3] type 00 class 0x060000
pci_bus 0000:ff: busn_res: [bus ff] end is updated to ff
PCI: pci_cache_line_size set to 64 bytes
e820: reserve RAM buffer [mem 0x00099000-0x0009ffff]
e820: reserve RAM buffer [mem 0x3f790000-0x3fffffff]
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI init
ACPI: bus type PNP registered
system 00:00: [mem 0xfbf00000-0xfbffffff] has been reserved
system 00:00: [mem 0xfc000000-0xfcffffff] has been reserved
system 00:00: [mem 0xfd000000-0xfdffffff] has been reserved
system 00:00: [mem 0xfe000000-0xfebfffff] has been reserved
system 00:00: [mem 0xfec8a000-0xfec8afff] could not be reserved
system 00:00: [mem 0xfed10000-0xfed10fff] has been reserved
system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active)
pnp 00:01: [dma 4]
pnp 00:01: Plug and Play ACPI device, IDs PNP0200 (active)
xen: registering gsi 8 triggering 1 polarity 0
pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active)
pnp 00:03: Plug and Play ACPI device, IDs PNP0800 (active)
xen: registering gsi 13 triggering 1 polarity 0
pnp 00:04: Plug and Play ACPI device, IDs PNP0c04 (active)
system 00:05: [io  0x0a00-0x0adf] has been reserved
system 00:05: [io  0x0ae0-0x0aef] has been reserved
system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active)
system 00:06: [io  0x04d0-0x04d1] has been reserved
system 00:06: [io  0x0800-0x087f] has been reserved
system 00:06: [io  0x0500-0x057f] has been reserved
system 00:06: [mem 0xfed1c000-0xfed1ffff] has been reserved
system 00:06: [mem 0xfed20000-0xfed3ffff] has been reserved
system 00:06: [mem 0xfed40000-0xfed8ffff] has been reserved
system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:07: Plug and Play ACPI device, IDs PNP0103 (active)
system 00:08: [mem 0xfec00000-0xfec00fff] could not be reserved
system 00:08: [mem 0xfee00000-0xfee00fff] has been reserved
system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
system 00:09: [mem 0xe0000000-0xefffffff] has been reserved
system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
system 00:0a: [mem 0x00000000-0x0009ffff] could not be reserved
system 00:0a: [mem 0x000c0000-0x000cffff] could not be reserved
system 00:0a: [mem 0x000e0000-0x000fffff] could not be reserved
system 00:0a: [mem 0x00100000-0x3fffffff] could not be reserved
system 00:0a: [mem 0xfed90000-0xffffffff] could not be reserved
system 00:0a: Plug and Play ACPI device, IDs PNP0c01 (active)
pnp: PnP ACPI: found 11 devices
ACPI: bus type PNP unregistered
PM-Timer failed consistency check  (0x0xffffff) - aborting.
pci 0000:00:1c.0: bridge window [io  0x1000-0x0fff] to [bus 04] add_size 1000
pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000
pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff] to [bus 04] add_size 200000
pci 0000:00:1c.2: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 03] add_size 200000
pci 0000:00:1c.4: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 02] add_size 200000
pci 0000:00:1c.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
pci 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
pci 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
pci 0000:00:1c.4: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
pci 0000:00:1c.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
pci 0000:00:1c.0: BAR 14: assigned [mem 0x40000000-0x401fffff]
pci 0000:00:1c.0: BAR 15: assigned [mem 0x40200000-0x403fffff 64bit pref]
pci 0000:00:1c.2: BAR 15: assigned [mem 0x40400000-0x405fffff 64bit pref]
pci 0000:00:1c.4: BAR 15: assigned [mem 0x40600000-0x407fffff 64bit pref]
pci 0000:00:1c.0: BAR 13: assigned [io  0x1000-0x1fff]
pci 0000:00:02.0: PCI bridge to [bus 11]
pci 0000:00:02.0:   bridge window [mem 0xfbe00000-0xfbefffff]
pci 0000:0a:00.0: PCI bridge to [bus 10]
pci 0000:0a:00.0:   bridge window [io  0xe000-0xefff]
pci 0000:0a:00.0:   bridge window [mem 0xd7f00000-0xdfffffff]
pci 0000:0a:00.0:   bridge window [mem 0xc0000000-0xcfffffff 64bit pref]
pci 0000:0d:00.0: PCI bridge to [bus 0f]
pci 0000:0d:00.0:   bridge window [mem 0xd7c00000-0xd7efffff]
pci 0000:0d:00.2: PCI bridge to [bus 0e]
pci 0000:0a:02.0: PCI bridge to [bus 0d-0f]
pci 0000:0a:02.0:   bridge window [mem 0xd7c00000-0xd7efffff]
pci 0000:0b:00.0: PCI bridge to [bus 0c]
pci 0000:0b:00.0:   bridge window [mem 0xd7b00000-0xd7bfffff]
pci 0000:0a:03.0: PCI bridge to [bus 0b-0c]
pci 0000:0a:03.0:   bridge window [mem 0xd7b00000-0xd7bfffff]
pci 0000:09:00.0: PCI bridge to [bus 0a-10]
pci 0000:09:00.0:   bridge window [io  0xe000-0xefff]
pci 0000:09:00.0:   bridge window [mem 0xd7b00000-0xdfffffff]
pci 0000:09:00.0:   bridge window [mem 0xc0000000-0xcfffffff 64bit pref]
pci 0000:00:03.0: PCI bridge to [bus 09-10]
pci 0000:00:03.0:   bridge window [io  0xe000-0xefff]
pci 0000:00:03.0:   bridge window [mem 0xd7b00000-0xdfffffff]
pci 0000:00:03.0:   bridge window [mem 0xc0000000-0xcfffffff 64bit pref]
pci 0000:06:00.0: PCI bridge to [bus 08]
pci 0000:06:00.0:   bridge window [io  0xd000-0xdfff]
pci 0000:06:00.0:   bridge window [mem 0xf8000000-0xfbdfffff]
pci 0000:06:00.0:   bridge window [mem 0xb4000000-0xbfffffff 64bit pref]
pci 0000:06:02.0: PCI bridge to [bus 07]
pci 0000:06:02.0:   bridge window [io  0xc000-0xcfff]
pci 0000:06:02.0:   bridge window [mem 0xf4000000-0xf7ffffff]
pci 0000:06:02.0:   bridge window [mem 0xa8000000-0xb3ffffff 64bit pref]
pci 0000:05:00.0: PCI bridge to [bus 06-08]
pci 0000:05:00.0:   bridge window [io  0xc000-0xdfff]
pci 0000:05:00.0:   bridge window [mem 0xf4000000-0xfbdfffff]
pci 0000:05:00.0:   bridge window [mem 0xa8000000-0xbfffffff 64bit pref]
pci 0000:00:07.0: PCI bridge to [bus 05-08]
pci 0000:00:07.0:   bridge window [io  0xc000-0xdfff]
pci 0000:00:07.0:   bridge window [mem 0xf4000000-0xfbdfffff]
pci 0000:00:07.0:   bridge window [mem 0xa8000000-0xbfffffff 64bit pref]
pci 0000:00:1c.0: PCI bridge to [bus 04]
pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
pci 0000:00:1c.0:   bridge window [mem 0x40000000-0x401fffff]
pci 0000:00:1c.0:   bridge window [mem 0x40200000-0x403fffff 64bit pref]
pci 0000:00:1c.2: PCI bridge to [bus 03]
pci 0000:00:1c.2:   bridge window [io  0xb000-0xbfff]
pci 0000:00:1c.2:   bridge window [mem 0xf3f00000-0xf3ffffff]
pci 0000:00:1c.2:   bridge window [mem 0x40400000-0x405fffff 64bit pref]
pci 0000:00:1c.4: PCI bridge to [bus 02]
pci 0000:00:1c.4:   bridge window [io  0xa000-0xafff]
pci 0000:00:1c.4:   bridge window [mem 0xf3e00000-0xf3efffff]
pci 0000:00:1c.4:   bridge window [mem 0x40600000-0x407fffff 64bit pref]
pci 0000:00:1e.0: PCI bridge to [bus 01]
pci 0000:00:1c.0: enabling device (0104 -> 0107)
xen: registering gsi 17 triggering 0 polarity 1
xen: --> pirq=17 -> irq=17 (gsi=17)
xen: registering gsi 18 triggering 0 polarity 1
xen: --> pirq=18 -> irq=18 (gsi=18)
xen: registering gsi 17 triggering 0 polarity 1
Already setup the GSI :17
pci 0000:00:1e.0: setting latency timer to 64
pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
pci_bus 0000:00: resource 5 [io  0x0d00-0xffff]
pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000dffff]
pci_bus 0000:00: resource 8 [mem 0x40000000-0xdfffffff]
pci_bus 0000:00: resource 9 [mem 0xf0000000-0xfed8ffff]
pci_bus 0000:11: resource 1 [mem 0xfbe00000-0xfbefffff]
pci_bus 0000:09: resource 0 [io  0xe000-0xefff]
pci_bus 0000:09: resource 1 [mem 0xd7b00000-0xdfffffff]
pci_bus 0000:09: resource 2 [mem 0xc0000000-0xcfffffff 64bit pref]
pci_bus 0000:0a: resource 0 [io  0xe000-0xefff]
pci_bus 0000:0a: resource 1 [mem 0xd7b00000-0xdfffffff]
pci_bus 0000:0a: resource 2 [mem 0xc0000000-0xcfffffff 64bit pref]
pci_bus 0000:10: resource 0 [io  0xe000-0xefff]
pci_bus 0000:10: resource 1 [mem 0xd7f00000-0xdfffffff]
pci_bus 0000:10: resource 2 [mem 0xc0000000-0xcfffffff 64bit pref]
pci_bus 0000:0d: resource 1 [mem 0xd7c00000-0xd7efffff]
pci_bus 0000:0f: resource 1 [mem 0xd7c00000-0xd7efffff]
pci_bus 0000:0b: resource 1 [mem 0xd7b00000-0xd7bfffff]
pci_bus 0000:0c: resource 1 [mem 0xd7b00000-0xd7bfffff]
pci_bus 0000:05: resource 0 [io  0xc000-0xdfff]
pci_bus 0000:05: resource 1 [mem 0xf4000000-0xfbdfffff]
pci_bus 0000:05: resource 2 [mem 0xa8000000-0xbfffffff 64bit pref]
pci_bus 0000:06: resource 0 [io  0xc000-0xdfff]
pci_bus 0000:06: resource 1 [mem 0xf4000000-0xfbdfffff]
pci_bus 0000:06: resource 2 [mem 0xa8000000-0xbfffffff 64bit pref]
pci_bus 0000:08: resource 0 [io  0xd000-0xdfff]
pci_bus 0000:08: resource 1 [mem 0xf8000000-0xfbdfffff]
pci_bus 0000:08: resource 2 [mem 0xb4000000-0xbfffffff 64bit pref]
pci_bus 0000:07: resource 0 [io  0xc000-0xcfff]
pci_bus 0000:07: resource 1 [mem 0xf4000000-0xf7ffffff]
pci_bus 0000:07: resource 2 [mem 0xa8000000-0xb3ffffff 64bit pref]
pci_bus 0000:04: resource 0 [io  0x1000-0x1fff]
pci_bus 0000:04: resource 1 [mem 0x40000000-0x401fffff]
pci_bus 0000:04: resource 2 [mem 0x40200000-0x403fffff 64bit pref]
pci_bus 0000:03: resource 0 [io  0xb000-0xbfff]
pci_bus 0000:03: resource 1 [mem 0xf3f00000-0xf3ffffff]
pci_bus 0000:03: resource 2 [mem 0x40400000-0x405fffff 64bit pref]
pci_bus 0000:02: resource 0 [io  0xa000-0xafff]
pci_bus 0000:02: resource 1 [mem 0xf3e00000-0xf3efffff]
pci_bus 0000:02: resource 2 [mem 0x40600000-0x407fffff 64bit pref]
pci_bus 0000:01: resource 4 [io  0x0000-0x0cf7]
pci_bus 0000:01: resource 5 [io  0x0d00-0xffff]
pci_bus 0000:01: resource 6 [mem 0x000a0000-0x000bffff]
pci_bus 0000:01: resource 7 [mem 0x000d0000-0x000dffff]
pci_bus 0000:01: resource 8 [mem 0x40000000-0xdfffffff]
pci_bus 0000:01: resource 9 [mem 0xf0000000-0xfed8ffff]
pci_bus 0000:fe: resource 4 [io  0x0000-0xffff]
pci_bus 0000:fe: resource 5 [mem 0x00000000-0xffffffffff]
pci_bus 0000:ff: resource 4 [io  0x0000-0xffff]
pci_bus 0000:ff: resource 5 [mem 0x00000000-0xffffffffff]
NET: Registered protocol family 2
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP: reno registered
UDP hash table entries: 32768 (order: 8, 1048576 bytes)
UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes)
NET: Registered protocol family 1
xen: registering gsi 16 triggering 0 polarity 1
xen: --> pirq=16 -> irq=16 (gsi=16)
xen: registering gsi 21 triggering 0 polarity 1
xen: --> pirq=21 -> irq=21 (gsi=21)
xen: registering gsi 19 triggering 0 polarity 1
xen: --> pirq=19 -> irq=19 (gsi=19)
xen: registering gsi 18 triggering 0 polarity 1
Already setup the GSI :18
xen: registering gsi 23 triggering 0 polarity 1
xen: --> pirq=23 -> irq=23 (gsi=23)
xen: registering gsi 19 triggering 0 polarity 1
Already setup the GSI :19
xen: registering gsi 18 triggering 0 polarity 1
Already setup the GSI :18
xen: registering gsi 23 triggering 0 polarity 1
Already setup the GSI :23
xen: registering gsi 29 triggering 0 polarity 1
xen: --> pirq=29 -> irq=29 (gsi=29)
pci 0000:10:00.0: Boot video device
pci 0000:0d:00.0: rerouting interrupts for [8086:0370]
pci 0000:0d:00.2: rerouting interrupts for [8086:0372]
PCI: CLS 256 bytes, default 64
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 68588k freed
Initialise module verification
audit: initializing netlink socket (disabled)
type=2000 audit(1378901447.667:1): initialized
bounce pool size: 64 pages
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 32768
Key type asymmetric registered
Asymmetric key parser 'x509' registered
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler deadline registered (default)
io scheduler cfq registered
aer 0000:00:02.0:pcie02: service driver aer loaded
aer 0000:00:03.0:pcie02: service driver aer loaded
aer 0000:00:07.0:pcie02: service driver aer loaded
pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt
pci 0000:11:00.0: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:02.0:pcie01: service driver pcie_pme loaded
pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt
pcieport 0000:09:00.0: Signaling PME through PCIe PME interrupt
pcieport 0000:0a:00.0: Signaling PME through PCIe PME interrupt
pci 0000:10:00.0: Signaling PME through PCIe PME interrupt
pci 0000:10:00.1: Signaling PME through PCIe PME interrupt
pcieport 0000:0a:02.0: Signaling PME through PCIe PME interrupt
pci 0000:0d:00.0: Signaling PME through PCIe PME interrupt
pci 0000:0f:0e.0: Signaling PME through PCIe PME interrupt
pci 0000:0d:00.2: Signaling PME through PCIe PME interrupt
pcieport 0000:0a:03.0: Signaling PME through PCIe PME interrupt
pci 0000:0b:00.0: Signaling PME through PCIe PME interrupt
pci 0000:0c:00.0: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:03.0:pcie01: service driver pcie_pme loaded
pcieport 0000:00:07.0: Signaling PME through PCIe PME interrupt
pcieport 0000:05:00.0: Signaling PME through PCIe PME interrupt
pcieport 0000:06:00.0: Signaling PME through PCIe PME interrupt
pci 0000:08:00.0: Signaling PME through PCIe PME interrupt
pci 0000:08:00.1: Signaling PME through PCIe PME interrupt
pcieport 0000:06:02.0: Signaling PME through PCIe PME interrupt
pci 0000:07:00.0: Signaling PME through PCIe PME interrupt
pci 0000:07:00.1: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:07.0:pcie01: service driver pcie_pme loaded
pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:1c.0:pcie01: service driver pcie_pme loaded
pcieport 0000:00:1c.2: Signaling PME through PCIe PME interrupt
pci 0000:03:00.0: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:1c.2:pcie01: service driver pcie_pme loaded
pcieport 0000:00:1c.4: Signaling PME through PCIe PME interrupt
pci 0000:02:00.0: Signaling PME through PCIe PME interrupt
pcie_pme 0000:00:1c.4:pcie01: service driver pcie_pme loaded
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
intel_idle: MWAIT substates: 0x1120
intel_idle: v0.4 model 0x2C
intel_idle: lapic_timer_reliable_states 0xffffffff
intel_idle: intel_idle yielding to none
input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
ACPI: Power Button [PWRB]
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
ACPI: Power Button [PWRF]
ACPI: Requesting acpi_cpufreq
Monitor-Mwait will be used to enter C-1 state
Monitor-Mwait will be used to enter C-3 state
Monitor-Mwait will be used to enter C-3 state
Warning: Processor Platform Limit not supported.
GHES: HEST is not enabled!
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
hpet_acpi_add: no address or irqs in _CRS
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
brd: module loaded
loop: module loaded
libphy: Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
xen: registering gsi 18 triggering 0 polarity 1
Already setup the GSI :18
ehci-pci 0000:00:1a.7: setting latency timer to 64
ehci-pci 0000:00:1a.7: EHCI Host Controller
ehci-pci 0000:00:1a.7: new USB bus registered, assigned bus number 1
ehci-pci 0000:00:1a.7: debug port 1
ehci-pci 0000:00:1a.7: cache line size of 256 is not supported
ehci-pci 0000:00:1a.7: irq 18, io mem 0xf3df8000
ehci-pci 0000:00:1a.7: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 3.9.9-2.el6xen.x86_64 ehci_hcd
usb usb1: SerialNumber: 0000:00:1a.7
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 6 ports detected
xen: registering gsi 23 triggering 0 polarity 1
Already setup the GSI :23
ehci-pci 0000:00:1d.7: setting latency timer to 64
ehci-pci 0000:00:1d.7: EHCI Host Controller
ehci-pci 0000:00:1d.7: new USB bus registered, assigned bus number 2
ehci-pci 0000:00:1d.7: debug port 1
ehci-pci 0000:00:1d.7: cache line size of 256 is not supported
ehci-pci 0000:00:1d.7: irq 23, io mem 0xf3dfa000
ehci-pci 0000:00:1d.7: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: EHCI Host Controller
usb usb2: Manufacturer: Linux 3.9.9-2.el6xen.x86_64 ehci_hcd
usb usb2: SerialNumber: 0000:00:1d.7
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 6 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
xen: registering gsi 16 triggering 0 polarity 1
Already setup the GSI :16
uhci_hcd 0000:00:1a.0: setting latency timer to 64
uhci_hcd 0000:00:1a.0: UHCI Host Controller
uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1a.0: irq 16, io base 0x00009980
usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb3: Product: UHCI Host Controller
usb usb3: Manufacturer: Linux 3.9.9-2.el6xen.x86_64 uhci_hcd
usb usb3: SerialNumber: 0000:00:1a.0
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
xen: registering gsi 21 triggering 0 polarity 1
Already setup the GSI :21
uhci_hcd 0000:00:1a.1: setting latency timer to 64
uhci_hcd 0000:00:1a.1: UHCI Host Controller
uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1a.1: irq 21, io base 0x00009a00
usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb4: Product: UHCI Host Controller
usb usb4: Manufacturer: Linux 3.9.9-2.el6xen.x86_64 uhci_hcd
usb usb4: SerialNumber: 0000:00:1a.1
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
xen: registering gsi 19 triggering 0 polarity 1
Already setup the GSI :19
uhci_hcd 0000:00:1a.2: setting latency timer to 64
uhci_hcd 0000:00:1a.2: UHCI Host Controller
uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:1a.2: irq 19, io base 0x00009a80
usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb5: Product: UHCI Host Controller
usb usb5: Manufacturer: Linux 3.9.9-2.el6xen.x86_64 uhci_hcd
usb usb5: SerialNumber: 0000:00:1a.2
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
xen: registering gsi 23 triggering 0 polarity 1
Already setup the GSI :23
uhci_hcd 0000:00:1d.0: setting latency timer to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6
uhci_hcd 0000:00:1d.0: irq 23, io base 0x00009b00
usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb6: Product: UHCI Host Controller
usb usb6: Manufacturer: Linux 3.9.9-2.el6xen.x86_64 uhci_hcd
usb usb6: SerialNumber: 0000:00:1d.0
hub 6-0:1.0: USB hub found
hub 6-0:1.0: 2 ports detected
xen: registering gsi 19 triggering 0 polarity 1
Already setup the GSI :19
uhci_hcd 0000:00:1d.1: setting latency timer to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7
uhci_hcd 0000:00:1d.1: irq 19, io base 0x00009b80
usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb7: Product: UHCI Host Controller
usb usb7: Manufacturer: Linux 3.9.9-2.el6xen.x86_64 uhci_hcd
usb usb7: SerialNumber: 0000:00:1d.1
hub 7-0:1.0: USB hub found
hub 7-0:1.0: 2 ports detected
xen: registering gsi 18 triggering 0 polarity 1
Already setup the GSI :18
uhci_hcd 0000:00:1d.2: setting latency timer to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
uhci_hcd 0000:00:1d.2: irq 18, io base 0x00009c00
usb usb8: New USB device found, idVendor=1d6b, idProduct=0001
usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb8: Product: UHCI Host Controller
usb usb8: Manufacturer: Linux 3.9.9-2.el6xen.x86_64 uhci_hcd
usb usb8: SerialNumber: 0000:00:1d.2
hub 8-0:1.0: USB hub found
hub 8-0:1.0: 2 ports detected
i8042: PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mousedev: PS/2 mouse device common for all mice
rtc_cmos 00:02: RTC can wake from S4
rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
rtc_cmos 00:02: alarms up to one month, y3k, 114 bytes nvram
EFI Variables Facility v0.08 2004-May-17
hidraw: raw HID events driver (C) Jiri Kosina
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
drop_monitor: Initializing network drop monitor service
TCP: cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
Loading module verification certificates
MODSIGN: Loaded cert 'Magrathea: Glacier signing key: 37cced82f1f9fd38d8ceb662cc0603433dd14ab8'
registered taskstats version 1
IMA: No TPM chip found, activating TPM-bypass!
rtc_cmos 00:02: setting system clock to 2013-09-11 12:10:47 UTC (1378901447)
Freeing unused kernel memory: 1732k freed
dracut: dracut-004-303.el6
dracut: rd_NO_LUKS: removing cryptoluks activation
dracut: rd_NO_LVM: removing LVM activation
scsi_transport_iscsi: module verification failed: signature and/or required key missing - tainting kernel
Loading iSCSI transport class v2.0-870.
iscsi: registered transport (qla4xxx)
QLogic iSCSI HBA Driver
libcxgbi:libcxgbi_init_module: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.
libcxgbi:ddp_setup_host_page_size: system PAGE 4096, ddp idx 0.
Chelsio T3 iSCSI Driver cxgb3i v2.0.0 (Jun. 2010)
iscsi: registered transport (cxgb3i)
Chelsio T4 iSCSI Driver cxgb4i v0.9.1 (Aug. 2010)
iscsi: registered transport (cxgb4i)
NET: Registered protocol family 10
cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.16 (Dec 05, 2012)
Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.2.2 (Apr 25, 2012)
iscsi: registered transport (bnx2i)
iscsi: registered transport (be2iscsi)
In beiscsi_module_init, tt=ffffffffa018acc0
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com
udev: starting version 147
udevd (234): /proc/234/oom_adj is deprecated, please use /proc/234/oom_score_adj instead.
[drm] Initialized drm 1.1.0 20060810
[drm] radeon kernel modesetting enabled.
xen: registering gsi 24 triggering 0 polarity 1
xen: --> pirq=24 -> irq=24 (gsi=24)
[drm] initializing kernel modesetting (RV770 0x1002:0x9442 0x174B:0xE810).
[drm] register mmio base: 0xD8000000
[drm] register mmio size: 65536
ATOM BIOS: Wekiva
radeon 0000:10:00.0: VRAM: 1024M 0x0000000000000000 - 0x000000003FFFFFFF (1024M used)
radeon 0000:10:00.0: GTT: 512M 0x0000000040000000 - 0x000000005FFFFFFF
[drm] Detected VRAM RAM=1024M, BAR=256M
[drm] RAM width 256bits DDR
[TTM] Zone  kernel: Available graphics memory: 24286928 kiB
[TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[TTM] Initializing pool allocator
[TTM] Initializing DMA pool allocator
[drm] radeon: 1024M of VRAM memory ready
[drm] radeon: 512M of GTT memory ready.
[drm] GART: num cpu pages 131072, num gpu pages 131072
[drm] enabling PCIE gen 2 link speeds, disable with radeon.pcie_gen2=0
[drm] Loading RV770 Microcode
[drm] PCIE GART of 512M enabled (table at 0x0000000000040000).
radeon 0000:10:00.0: WB enabled
radeon 0000:10:00.0: fence driver on ring 0 use gpu addr 0x0000000040000c00 and cpu addr 0xffff880c478a1c00
radeon 0000:10:00.0: fence driver on ring 3 use gpu addr 0x0000000040000c0c and cpu addr 0xffff880c478a1c0c
[drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[drm] Driver supports precise vblank timestamp query.
radeon 0000:10:00.0: radeon: using MSI.
[drm] radeon: irq initialized.
[drm] ring test on 0 succeeded in 1 usecs
[drm] ring test on 3 succeeded in 1 usecs
[drm] ib test on ring 0 succeeded in 0 usecs
[drm] ib test on ring 3 succeeded in 0 usecs
[drm] Radeon Display Connectors
[drm] Connector 0:
[drm]   DVI-I-1
[drm]   HPD1
[drm]   DDC: 0x7e60 0x7e60 0x7e64 0x7e64 0x7e68 0x7e68 0x7e6c 0x7e6c
[drm]   Encoders:
[drm]     DFP1: INTERNAL_UNIPHY
[drm]     CRT2: INTERNAL_KLDSCP_DAC2
[drm] Connector 1:
[drm]   DIN-1
[drm]   Encoders:
[drm]     TV1: INTERNAL_KLDSCP_DAC2
[drm] Connector 2:
[drm]   DVI-I-2
[drm]   HPD2
[drm]   DDC: 0x7e20 0x7e20 0x7e24 0x7e24 0x7e28 0x7e28 0x7e2c 0x7e2c
[drm]   Encoders:
[drm]     CRT1: INTERNAL_KLDSCP_DAC1
[drm]     DFP2: INTERNAL_KLDSCP_LVTMA
[drm] Internal thermal controller with fan control
[drm] radeon: power management initialized
parse error at position 4 in video mode 'edid/1920x2400'
[drm] forcing DIN-1 connector OFF
[drm] fb mappable at 0xC0142000
[drm] vram apper at 0xC0000000
[drm] size 36864000
[drm] fb depth is 24
[drm]    pitch is 15360
fbcon: radeondrmfb (fb0) is primary device
usb 4-1: new low-speed USB device number 2 using uhci_hcd
usb 4-1: New USB device found, idVendor=045e, idProduct=00dd
usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 4-1: Product: Comfort Curve Keyboard 2000
usb 4-1: Manufacturer: Microsoft
input: Microsoft Comfort Curve Keyboard 2000 as /devices/pci0000:00/0000:00:1a.1/usb4/4-1/4-1:1.0/input/input2
hid-generic 0003:045E:00DD.0001: input,hidraw0: USB HID v1.11 Keyboard [Microsoft Comfort Curve Keyboard 2000] on usb-0000:00:1a.1-1/input0
input: Microsoft Comfort Curve Keyboard 2000 as /devices/pci0000:00/0000:00:1a.1/usb4/4-1/4-1:1.1/input/input3
hid-generic 0003:045E:00DD.0002: input,hidraw1: USB HID v1.11 Device [Microsoft Comfort Curve Keyboard 2000] on usb-0000:00:1a.1-1/input1
Console: switching to colour frame buffer device 240x150
usb 4-2: new full-speed USB device number 3 using uhci_hcd
usb 4-2: New USB device found, idVendor=1532, idProduct=002f
usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 4-2: Product: Razer Imperator
usb 4-2: Manufacturer: Razer
input: Razer Razer Imperator as /devices/pci0000:00/0000:00:1a.1/usb4/4-2/4-2:1.0/input/input4
hid-generic 0003:1532:002F.0003: input,hidraw2: USB HID v1.11 Mouse [Razer Razer Imperator] on usb-0000:00:1a.1-2/input0
input: Razer Razer Imperator as /devices/pci0000:00/0000:00:1a.1/usb4/4-2/4-2:1.1/input/input5
hid-generic 0003:1532:002F.0004: input,hidraw3: USB HID v1.11 Keyboard [Razer Razer Imperator] on usb-0000:00:1a.1-2/input1
usb 5-1: new full-speed USB device number 2 using uhci_hcd
radeon 0000:10:00.0: fb0: radeondrmfb frame buffer device
radeon 0000:10:00.0: registered panic notifier
[drm] Initialized radeon 2.30.0 20080528 for 0000:10:00.0 on minor 0
dracut: Starting plymouth daemon
usb 5-1: New USB device found, idVendor=1532, idProduct=0017
usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 5-1: Product: Razer Imperator
usb 5-1: Manufacturer: Razer
dracut: rd_NO_DM: removing DM RAID activation
sky2: driver version 1.30
xen: registering gsi 18 triggering 0 polarity 1
Already setup the GSI :18
sky2 0000:03:00.0: Yukon-2 UL 2 chip revision 0
sky2 0000:03:00.0 eth0: addr 00:1f:bc:0e:0c:3f
xen: registering gsi 16 triggering 0 polarity 1
Already setup the GSI :16
sky2 0000:02:00.0: Yukon-2 UL 2 chip revision 0
sky2 0000:02:00.0 eth1: addr 00:1f:bc:0e:0c:40
ahci 0000:00:1f.2: version 3.0
xen: registering gsi 19 triggering 0 polarity 1
Already setup the GSI :19
ahci: SSS flag set, parallel bus scan disabled
ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ccc ems sxs 
ahci 0000:00:1f.2: setting latency timer to 64
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
ata1: SATA max UDMA/133 abar m2048@0xf3dfc000 port 0xf3dfc100 irq 221
ata2: SATA max UDMA/133 abar m2048@0xf3dfc000 port 0xf3dfc180 irq 221
ata3: SATA max UDMA/133 abar m2048@0xf3dfc000 port 0xf3dfc200 irq 221
ata4: SATA max UDMA/133 abar m2048@0xf3dfc000 port 0xf3dfc280 irq 221
ata5: SATA max UDMA/133 abar m2048@0xf3dfc000 port 0xf3dfc300 irq 221
ata6: SATA max UDMA/133 abar m2048@0xf3dfc000 port 0xf3dfc380 irq 221
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: HPA detected: current 402653184, native 468862128
ata1.00: ATA-9: INTEL SSDSC2CW240A3, 400i, max UDMA/133
ata1.00: 402653184 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
ata1.00: configured for UDMA/133
scsi 0:0:0:0: Direct-Access     ATA      INTEL SSDSC2CW24 400i PQ: 0 ANSI: 5
ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata2.00: ATA-8: KINGSTON SH103S3240G, 506ABBF0, max UDMA/133
ata2.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
ata2.00: configured for UDMA/133
scsi 1:0:0:0: Direct-Access     ATA      KINGSTON SH103S3 506A PQ: 0 ANSI: 5
ata3: SATA link down (SStatus 0 SControl 300)
ata4: SATA link down (SStatus 0 SControl 300)
ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata5.00: ATAPI: Optiarc DVD RW AD-7280S, 1.01, max UDMA/100
ata5.00: configured for UDMA/100
scsi 4:0:0:0: CD-ROM            Optiarc  DVD RW AD-7280S  1.01 PQ: 0 ANSI: 5
ata6: SATA link down (SStatus 0 SControl 300)
wmi: Mapper loaded
Adaptec aacraid driver 1.2-0[30000]-ms
xen: registering gsi 24 triggering 0 polarity 1
Already setup the GSI :24
AAC0: kernel 5.2-0[17342] Aug  4 2010
AAC0: monitor 5.2-0[17342]
AAC0: bios 5.2-0[17342]
AAC0: serial 7D15318191E
AAC0: Non-DASD support enabled.
AAC0: 64bit support enabled.
AAC0: 64 Bit DAC enabled
scsi6 : aacraid
scsi scan: INQUIRY result too short (5), using 36
scsi 6:1:6:0: Direct-Access                                    PQ: 0 ANSI: 0
scsi scan: INQUIRY result too short (5), using 36
scsi 6:1:7:0: Direct-Access                                    PQ: 0 ANSI: 0
scsi scan: INQUIRY result too short (5), using 36
scsi 6:1:8:0: Direct-Access                                    PQ: 0 ANSI: 0
scsi scan: INQUIRY result too short (5), using 36
scsi 6:1:15:0: Direct-Access                                    PQ: 0 ANSI: 0
sd 0:0:0:0: [sda] 402653184 512-byte logical blocks: (206 GB/192 GiB)
sd 1:0:0:0: [sdb] 468862128 512-byte logical blocks: (240 GB/223 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda4
sd 0:0:0:0: [sda] Attached SCSI disk
 sdb: sdb1 sdb2 sdb4
sd 1:0:0:0: [sdb] Attached SCSI disk
sr0: scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray
cdrom: Uniform CD-ROM driver Revision: 3.20
sr 4:0:0:0: Attached scsi CD-ROM sr0
md: bind<sda2>
md: bind<sda1>
md: bind<sdb1>
md: raid1 personality registered for level 1
md/raid1:md126: active with 2 out of 2 mirrors
md126: detected capacity change from 0 to 536805376
 md126: unknown partition table
md: bind<sdb2>
md/raid1:md127: active with 2 out of 2 mirrors
md127: detected capacity change from 0 to 34359607296
 md127: unknown partition table
input: Razer Razer Imperator as /devices/pci0000:00/0000:00:1a.2/usb5/5-1/5-1:1.0/input/input6
hid-generic 0003:1532:0017.0005: input,hidraw4: USB HID v1.11 Mouse [Razer Razer Imperator] on usb-0000:00:1a.2-1/input0
input: Razer Razer Imperator as /devices/pci0000:00/0000:00:1a.2/usb5/5-1/5-1:1.1/input/input7
hid-generic 0003:1532:0017.0006: input,hidraw5: USB HID v1.11 Keyboard [Razer Razer Imperator] on usb-0000:00:1a.2-1/input1
usb 5-2: new low-speed USB device number 3 using uhci_hcd
usb 5-2: New USB device found, idVendor=045e, idProduct=00dd
usb 5-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 5-2: Product: Comfort Curve Keyboard 2000
usb 5-2: Manufacturer: Microsoft
input: Microsoft Comfort Curve Keyboard 2000 as /devices/pci0000:00/0000:00:1a.2/usb5/5-2/5-2:1.0/input/input8
hid-generic 0003:045E:00DD.0007: input,hidraw6: USB HID v1.11 Keyboard [Microsoft Comfort Curve Keyboard 2000] on usb-0000:00:1a.2-2/input0
input: Microsoft Comfort Curve Keyboard 2000 as /devices/pci0000:00/0000:00:1a.2/usb5/5-2/5-2:1.1/input/input9
hid-generic 0003:045E:00DD.0008: input,hidraw7: USB HID v1.11 Device [Microsoft Comfort Curve Keyboard 2000] on usb-0000:00:1a.2-2/input1
EXT4-fs (md127): mounted filesystem with ordered data mode. Opts: (null)
dracut: Remounting /dev/disk/by-uuid/ffff5c41-4a21-4187-8af3-349580349117 with -o noatime,ro
EXT4-fs (md127): mounted filesystem with ordered data mode. Opts: (null)
dracut: Mounted root filesystem /dev/md127
dracut: Switching root
readahead: starting
udev: starting version 147
WARNING! power/level is deprecated; use power/control instead
xen: registering gsi 22 triggering 0 polarity 1
xen: --> pirq=22 -> irq=22 (gsi=22)
input: HDA Digital PCBeep as /devices/pci0000:00/0000:00:1b.0/input/input10
input: HDA Intel Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input11
input: HDA Intel Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input12
input: HDA Intel Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input13
input: HDA Intel Line Out Side as /devices/pci0000:00/0000:00:1b.0/sound/card0/input14
input: HDA Intel Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card0/input15
input: HDA Intel Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card0/input16
input: HDA Intel Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card0/input17
xen: registering gsi 34 triggering 0 polarity 1
xen: --> pirq=34 -> irq=34 (gsi=34)
hda-intel 0000:10:00.1: Handle VGA-switcheroo audio client
input: HDA ATI HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.0/0000:09:00.0/0000:0a:00.0/0000:10:00.1/sound/card1/input18
xen: registering gsi 36 triggering 0 polarity 1
xen: --> pirq=36 -> irq=36 (gsi=36)
input: HDA Creative Line as /devices/pci0000:00/0000:00:03.0/0000:09:00.0/0000:0a:03.0/0000:0b:00.0/0000:0c:00.0/sound/card2/input19
input: HDA Creative Mic as /devices/pci0000:00/0000:00:03.0/0000:09:00.0/0000:0a:03.0/0000:0b:00.0/0000:0c:00.0/sound/card2/input20
input: HDA Creative Front Headphone as /devices/pci0000:00/0000:00:03.0/0000:09:00.0/0000:0a:03.0/0000:0b:00.0/0000:0c:00.0/sound/card2/input21
input: HDA Creative Line Out Side as /devices/pci0000:00/0000:00:03.0/0000:09:00.0/0000:0a:03.0/0000:0b:00.0/0000:0c:00.0/sound/card2/input22
input: HDA Creative Line Out CLFE as /devices/pci0000:00/0000:00:03.0/0000:09:00.0/0000:0a:03.0/0000:0b:00.0/0000:0c:00.0/sound/card2/input23
input: HDA Creative Line Out Surround as /devices/pci0000:00/0000:00:03.0/0000:09:00.0/0000:0a:03.0/0000:0b:00.0/0000:0c:00.0/sound/card2/input24
input: HDA Creative Line Out Front as /devices/pci0000:00/0000:00:03.0/0000:09:00.0/0000:0a:03.0/0000:0b:00.0/0000:0c:00.0/sound/card2/input25
xen: registering gsi 37 triggering 0 polarity 1
xen: --> pirq=37 -> irq=37 (gsi=37)
hda_intel: Disabling MSI
hda-intel 0000:08:00.1: Handle VGA-switcheroo audio client
input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:07.0/0000:05:00.0/0000:06:00.0/0000:08:00.1/sound/card3/input26
input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:07.0/0000:05:00.0/0000:06:00.0/0000:08:00.1/sound/card3/input27
input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:07.0/0000:05:00.0/0000:06:00.0/0000:08:00.1/sound/card3/input28
input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:07.0/0000:05:00.0/0000:06:00.0/0000:08:00.1/sound/card3/input29
xen: registering gsi 38 triggering 0 polarity 1
xen: --> pirq=38 -> irq=38 (gsi=38)
hda_intel: Disabling MSI
hda-intel 0000:07:00.1: Handle VGA-switcheroo audio client
input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:07.0/0000:05:00.0/0000:06:02.0/0000:07:00.1/sound/card4/input30
input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:07.0/0000:05:00.0/0000:06:02.0/0000:07:00.1/sound/card4/input31
input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:07.0/0000:05:00.0/0000:06:02.0/0000:07:00.1/sound/card4/input32
input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:07.0/0000:05:00.0/0000:06:02.0/0000:07:00.1/sound/card4/input33
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
EDAC MC: Ver: 3.0.0
EDAC MC1: Giving out device to 'i7core_edac.c' 'i7 core #1': DEV 0000:fe:03.0
EDAC PCI0: Giving out device to module 'i7core_edac' controller 'EDAC PCI controller': DEV '0000:fe:03.0' (POLLED)
EDAC MC0: Giving out device to 'i7core_edac.c' 'i7 core #0': DEV 0000:ff:03.0
EDAC PCI1: Giving out device to module 'i7core_edac' controller 'EDAC PCI controller': DEV '0000:ff:03.0' (POLLED)
EDAC i7core: Driver loaded, 2 memory controller(s) found.
ACPI Warning: 0x0000000000000828-0x000000000000082f SystemIO conflicts with Region \PMRG 1 (20130117/utaddress-251)
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
lpc_ich: Resource conflict(s) found affecting gpio_ich
xen: registering gsi 18 triggering 0 polarity 1
Already setup the GSI :18
i801_smbus 0000:00:1f.3: SMBus using PCI Interrupt
sd 0:0:0:0: Attached scsi generic sg0 type 0
sd 1:0:0:0: Attached scsi generic sg1 type 0
sr 4:0:0:0: Attached scsi generic sg2 type 5
scsi 6:1:6:0: Attached scsi generic sg3 type 0
scsi 6:1:7:0: Attached scsi generic sg4 type 0
scsi 6:1:8:0: Attached scsi generic sg5 type 0
scsi 6:1:15:0: Attached scsi generic sg6 type 0
input: PC Speaker as /devices/platform/pcspkr/input/input34
microcode: CPU0 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU1 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU2 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU3 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU4 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU5 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU6 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU7 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU8 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU9 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU10 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU11 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU12 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU13 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU14 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU15 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU16 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU17 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU18 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU19 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU20 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU21 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU22 sig=0x206c0, pf=0x1, revision=0x0
microcode: CPU23 sig=0x206c0, pf=0x1, revision=0x0
microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.10
iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
SPL: Loaded module v0.6.1-1
zunicode: module license 'CDDL' taints kernel.
Disabling lock debugging due to kernel taint
 zd0: p1
 zd16: p1
ZFS: Loaded module v0.6.1-1, ZFS pool version 5000, ZFS filesystem version 5
SPL: using hostid 0x00000000
bonding: Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
pciback 0000:11:00.0: seizing device
pciback 0000:08:00.0: seizing device
pciback 0000:07:00.0: seizing device
pciback 0000:07:00.0: enabling device (0000 -> 0003)
xen: registering gsi 39 triggering 0 polarity 1
xen: --> pirq=39 -> irq=39 (gsi=39)
pciback 0000:08:00.0: enabling device (0000 -> 0003)
xen: registering gsi 30 triggering 0 polarity 1
xen: --> pirq=30 -> irq=30 (gsi=30)
xen: registering gsi 29 triggering 0 polarity 1
Already setup the GSI :29
xen-pciback: backend is vpci
Event-channel device installed.
zram: module is from the staging directory, the quality is unknown, you have been warned.
zram: Created 24 device(s) ...
Adding 1048572k swap on /dev/zram0.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram1.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram2.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram3.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram4.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram5.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram6.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram7.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram8.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram9.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram10.  Priority:100 extents:1 across:1048572k SS
Adding 1048572k swap on /dev/zram11.  Priority:100 extents:1 across:1048572k SS
EXT4-fs (md127): re-mounted. Opts: (null)
EXT4-fs (md126): mounted filesystem with ordered data mode. Opts: (null)
bonding: bond0: Adding slave eth0.
sky2 0000:03:00.0 eth0: enabling interface
bonding: bond0: enslaving eth0 as an active interface with an up link.
bonding: bond0: Adding slave eth1.
sky2 0000:02:00.0 eth1: enabling interface
bonding: bond0: enslaving eth1 as an active interface with an up link.
Bridge firewalling registered
device bond0 entered promiscuous mode
device eth0 entered promiscuous mode
device eth1 entered promiscuous mode
br0: port 1(bond0) entered forwarding state
br0: port 1(bond0) entered forwarding state
sky2 0000:03:00.0 eth0: Link is up at 1000 Mbps, full duplex, flow control both
sky2 0000:02:00.0 eth1: Link is up at 1000 Mbps, full duplex, flow control both
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
FS-Cache: Loaded
FS-Cache: Netfs 'nfs' registered for caching
f71882fg: Found f71808e chip at 0xa10, revision 49
f71882fg f71882fg.2576: Fan: 1 is in duty-cycle mode
f71882fg f71882fg.2576: Auto pwm controlled by raw digital data, disabling pwm auto_point sysfs attributes for fan 1
f71882fg f71882fg.2576: Fan: 2 is in duty-cycle mode
f71882fg f71882fg.2576: Fan: 3 is in duty-cycle mode
uhci_hcd 0000:00:1a.1: remove, state 1
usb usb4: USB disconnect, device number 1
usb 4-1: USB disconnect, device number 2
usb 4-2: USB disconnect, device number 3
uhci_hcd 0000:00:1a.1: USB bus 4 deregistered
pciback 0000:00:1a.1: seizing device
xen: registering gsi 21 triggering 0 polarity 1
Already setup the GSI :21
pciback 0000:07:00.1: seizing device
xen: registering gsi 38 triggering 0 polarity 1
Already setup the GSI :38
pciback 0000:08:00.1: seizing device
xen: registering gsi 37 triggering 0 polarity 1
Already setup the GSI :37
pciback 0000:0c:00.0: seizing device
xen: registering gsi 36 triggering 0 polarity 1
Already setup the GSI :36
vgaarb: device changed decodes: PCI:0000:10:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
vgaarb: transferring owner from PCI:0000:10:00.0 to PCI:0000:07:00.0

[-- Attachment #3: xl-dmesg.log --]
[-- Type: text/x-c++, Size: 23559 bytes --]

 __  __            _  _    _____  ___   _____     _  __   
 \ \/ /___ _ __   | || |  |___ / / _ \ |___  |___| |/ /_  
  \  // _ \ '_ \  | || |_   |_ \| | | |__ / // _ \ | '_ \ 
  /  \  __/ | | | |__   _| ___) | |_| |__/ /|  __/ | (_) |
 /_/\_\___|_| |_|    |_|(_)____(_)___/  /_/(_)___|_|\___/ 
                                                          
(XEN) Xen version 4.3.0 (root@shatteredsilicon.net) (gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-3)) debug=n Thu Sep  5 10:55:26 BST 2013
(XEN) Latest ChangeSet: 
(XEN) Bootloader: GNU GRUB 0.97
(XEN) Command line: noreboot dom0_vcpus_pin iommu=debug,dom0-passthrough,workaround_bios_bug loglvl=all guest_loglvl=all unrestricted_guest=1 msi=1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 2 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 0000000000099800 (usable)
(XEN)  0000000000099800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000003f790000 (usable)
(XEN)  000000003f790000 - 000000003f79e000 (ACPI data)
(XEN)  000000003f79e000 - 000000003f7d0000 (ACPI NVS)
(XEN)  000000003f7d0000 - 000000003f7e0000 (reserved)
(XEN)  000000003f7e7000 - 0000000040000000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000cc0000000 (usable)
(XEN) ACPI: RSDP 000F9F70, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT 3F790100, 0064 (r1 042413 XSDT1438 20130424 MSFT       97)
(XEN) ACPI: FACP 3F790290, 00F4 (r4 042413 FACP1438 20130424 MSFT       97)
(XEN) ACPI: DSDT 3F7904F0, 58A3 (r2  1W555 1W555A58      A58 INTL 20051117)
(XEN) ACPI: FACS 3F79E000, 0040
(XEN) ACPI: APIC 3F790390, 0118 (r2 042413 APIC1438 20130424 MSFT       97)
(XEN) ACPI: MCFG 3F7904B0, 003C (r1 042413 OEMMCFG  20130424 MSFT       97)
(XEN) ACPI: OEMB 3F79E040, 0082 (r1 042413 OEMB1438 20130424 MSFT       97)
(XEN) ACPI: SRAT 3F79A4F0, 0250 (r2 042413 OEMSRAT         1 INTL        1)
(XEN) ACPI: HPET 3F79A740, 0038 (r1 042413 OEMHPET  20130424 MSFT       97)
(XEN) ACPI: DMAR 3F79E0D0, 0120 (r1    AMI  OEMDMAR        1 MSFT       97)
(XEN) ACPI: SSDT 3F7A4C70, 0363 (r1 DpgPmm    CpuPm       12 INTL 20051117)
(XEN) System RAM: 49143MB (50322596kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 4 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 16 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 18 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 20 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 3 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 5 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 17 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 19 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 21 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 32 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 34 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 36 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 48 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 50 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 52 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 33 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 35 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 37 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 49 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 51 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 53 -> Node 1
(XEN) SRAT: Node 0 PXM 0 0-a0000
(XEN) SRAT: Node 0 PXM 0 100000-40000000
(XEN) SRAT: Node 0 PXM 0 100000000-6c0000000
(XEN) SRAT: Node 1 PXM 1 6c0000000-cc0000000
(XEN) NUMA: Allocated memnodemap from cbab5f000 - cbab6c000
(XEN) NUMA: Using 8 for the hash shift.
(XEN) Domain heap initialised DMA width 32 bits
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x808
(XEN) ACPI: SLEEP INFO: pm1x_cnt[804,0], pm1x_evt[800,0]
(XEN) ACPI:             wakeup_vec[3f79e00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) Processor #4 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x10] enabled)
(XEN) Processor #16 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x12] enabled)
(XEN) Processor #18 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled)
(XEN) Processor #20 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x20] enabled)
(XEN) Processor #32 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x22] enabled)
(XEN) Processor #34 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x24] enabled)
(XEN) Processor #36 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x30] enabled)
(XEN) Processor #48 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x32] enabled)
(XEN) Processor #50 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x34] enabled)
(XEN) Processor #52 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x01] enabled)
(XEN) Processor #1 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x03] enabled)
(XEN) Processor #3 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x05] enabled)
(XEN) Processor #5 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x11] enabled)
(XEN) Processor #17 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x13] enabled)
(XEN) Processor #19 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x15] enabled)
(XEN) Processor #21 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x21] enabled)
(XEN) Processor #33 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x23] enabled)
(XEN) Processor #35 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x25] enabled)
(XEN) Processor #37 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x31] enabled)
(XEN) Processor #49 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x33] enabled)
(XEN) Processor #51 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x18] lapic_id[0x35] enabled)
(XEN) Processor #53 6:12 APIC version 21
(XEN) Overriding APIC driver with bigsmp
(XEN) ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 6, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x07] address[0xfec8a000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 7, version 32, address 0xfec8a000, GSI 24-47
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 20 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
(XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000
(XEN) [VT-D]dmar.c:767: Host address width 40
(XEN) [VT-D]dmar.c:781: found ACPI_DMAR_DRHD:
(XEN) [VT-D]dmar.c:461:   dmaru->address = fbffe000
(XEN) [VT-D]iommu.c:1158: drhd->address = fbffe000 iommu->reg = ffff82c3ffd54000
(XEN) [VT-D]iommu.c:1160: cap = c90780106f0462 ecap = f020f7
(XEN) [VT-D]dmar.c:396:  IOAPIC: 0000:f0:1f.7
(XEN) [VT-D]dmar.c:396:  IOAPIC: 0000:00:13.0
(XEN) [VT-D]dmar.c:475:   flags: INCLUDE_ALL
(XEN) [VT-D]dmar.c:786: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1d.0
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1d.1
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1d.2
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1d.7
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1a.0
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1a.1
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1a.2
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1a.7
(XEN) [VT-D]dmar.c:655:   RMRR region: base_addr ec000 end_address f4fff
(XEN) [VT-D]dmar.c:786: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1d.0
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1d.1
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1d.2
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1d.7
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1a.0
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1a.1
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1a.2
(XEN) [VT-D]dmar.c:382:  endpoint: 0000:00:1a.7
(XEN) [VT-D]dmar.c:655:   RMRR region: base_addr 3f7e7000 end_address 3f7fffff
(XEN) [VT-D]dmar.c:791: found ACPI_DMAR_ATSR:
(XEN) [VT-D]dmar.c:684:   atsru->all_ports: 0
(XEN) [VT-D]dmar.c:353:  bridge: 0000:00:02.0 start=0 sec=11 sub=11
(XEN) [VT-D]dmar.c:353:  bridge: 0000:00:03.0 start=0 sec=9 sub=10
(XEN) [VT-D]dmar.c:353:  bridge: 0000:00:07.0 start=0 sec=5 sub=8
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 24 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 48 GSI, 4576 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3321.718 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:717: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: Not using MCFG for segment 0000 bus 00-ff
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Passthrough
(XEN) Interrupt remapping disabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Defaulting to alternative key handling; send 'A' to switch to normal mode.
(XEN) Allocated console ring of 256 KiB.
(XEN) mwait-idle: MWAIT substates: 0x1120
(XEN) mwait-idle: v0.4 model 0x2c
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 24 CPUs
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1f70000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000420000000->0000000430000000 (12301846 pages to be allocated)
(XEN)  Init. ramdisk: 0000000cbbd05000->0000000cbffffa00
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81f70000
(XEN)  Init. ramdisk: ffffffff81f70000->ffffffff8626aa00
(XEN)  Phys-Mach map: ffffffff8626b000->ffffffff8c0e7888
(XEN)  Start info:    ffffffff8c0e8000->ffffffff8c0e84b4
(XEN)  Page tables:   ffffffff8c0e9000->ffffffff8c14e000
(XEN)  Boot stack:    ffffffff8c14e000->ffffffff8c14f000
(XEN)  TOTAL:         ffffffff80000000->ffffffff8c400000
(XEN)  ENTRY ADDRESS: ffffffff818091e0
(XEN) Dom0 has maximum 24 VCPUs
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:00:00.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:13.0
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:00:14.0
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:00:14.1
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:00:14.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:14.3
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1a.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1a.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1a.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1a.7
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:00:1b.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1d.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1d.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1d.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1d.7
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1f.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1f.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:00:1f.3
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:02:00.0
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:03:00.0
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:07:00.0
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:07:00.1
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:08:00.0
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:08:00.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:0c:00.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:0f:0e.0
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:10:00.0
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:10:00.1
(XEN) [VT-D]iommu.c:1441: d0:PCIe: map 0000:11:00.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:00.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:00.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:02.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:02.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:02.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:02.3
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:02.4
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:02.5
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:03.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:03.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:03.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:03.4
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:04.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:04.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:04.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:04.3
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:05.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:05.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:05.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:05.3
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:06.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:06.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:06.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:fe:06.3
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:00.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:00.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:02.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:02.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:02.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:02.3
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:02.4
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:02.5
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:03.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:03.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:03.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:03.4
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:04.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:04.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:04.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:04.3
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:05.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:05.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:05.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:05.3
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:06.0
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:06.1
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:06.2
(XEN) [VT-D]iommu.c:1453: d0:PCI: map 0000:ff:06.3
(XEN) [VT-D]iommu.c:755: iommu_enable_translation: iommu->reg = ffff82c3ffd54000
(XEN) Scrubbing Free RAM: ..done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 272kB init memory.
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:03.0
(XEN) PCI add device 0000:00:07.0
(XEN) PCI add device 0000:00:13.0
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.1
(XEN) PCI add device 0000:00:14.2
(XEN) PCI add device 0000:00:14.3
(XEN) PCI add device 0000:00:1a.0
(XEN) PCI add device 0000:00:1a.1
(XEN) PCI add device 0000:00:1a.2
(XEN) PCI add device 0000:00:1a.7
(XEN) PCI add device 0000:00:1b.0
(XEN) PCI add device 0000:00:1c.0
(XEN) PCI add device 0000:00:1c.2
(XEN) PCI add device 0000:00:1c.4
(XEN) PCI add device 0000:00:1d.0
(XEN) PCI add device 0000:00:1d.1
(XEN) PCI add device 0000:00:1d.2
(XEN) PCI add device 0000:00:1d.7
(XEN) PCI add device 0000:00:1e.0
(XEN) PCI add device 0000:00:1f.0
(XEN) PCI add device 0000:00:1f.2
(XEN) PCI add device 0000:00:1f.3
(XEN) PCI add device 0000:11:00.0
(XEN) PCI add device 0000:09:00.0
(XEN) PCI add device 0000:0a:00.0
(XEN) PCI add device 0000:0a:02.0
(XEN) PCI add device 0000:0a:03.0
(XEN) PCI add device 0000:10:00.0
(XEN) PCI add device 0000:10:00.1
(XEN) PCI add device 0000:0d:00.0
(XEN) PCI add device 0000:0d:00.2
(XEN) PCI add device 0000:0f:0e.0
(XEN) PCI add device 0000:0b:00.0
(XEN) PCI add device 0000:0c:00.0
(XEN) PCI add device 0000:05:00.0
(XEN) PCI add device 0000:06:00.0
(XEN) PCI add device 0000:06:02.0
(XEN) PCI add device 0000:08:00.0
(XEN) PCI add device 0000:08:00.1
(XEN) PCI add device 0000:07:00.0
(XEN) PCI add device 0000:07:00.1
(XEN) PCI add device 0000:03:00.0
(XEN) PCI add device 0000:02:00.0
(XEN) PCI add device 0000:fe:00.0
(XEN) PCI add device 0000:fe:00.1
(XEN) PCI add device 0000:fe:02.0
(XEN) PCI add device 0000:fe:02.1
(XEN) PCI add device 0000:fe:02.2
(XEN) PCI add device 0000:fe:02.3
(XEN) PCI add device 0000:fe:02.4
(XEN) PCI add device 0000:fe:02.5
(XEN) PCI add device 0000:fe:03.0
(XEN) PCI add device 0000:fe:03.1
(XEN) PCI add device 0000:fe:03.2
(XEN) PCI add device 0000:fe:03.4
(XEN) PCI add device 0000:fe:04.0
(XEN) PCI add device 0000:fe:04.1
(XEN) PCI add device 0000:fe:04.2
(XEN) PCI add device 0000:fe:04.3
(XEN) PCI add device 0000:fe:05.0
(XEN) PCI add device 0000:fe:05.1
(XEN) PCI add device 0000:fe:05.2
(XEN) PCI add device 0000:fe:05.3
(XEN) PCI add device 0000:fe:06.0
(XEN) PCI add device 0000:fe:06.1
(XEN) PCI add device 0000:fe:06.2
(XEN) PCI add device 0000:fe:06.3
(XEN) PCI add device 0000:ff:00.0
(XEN) PCI add device 0000:ff:00.1
(XEN) PCI add device 0000:ff:02.0
(XEN) PCI add device 0000:ff:02.1
(XEN) PCI add device 0000:ff:02.2
(XEN) PCI add device 0000:ff:02.3
(XEN) PCI add device 0000:ff:02.4
(XEN) PCI add device 0000:ff:02.5
(XEN) PCI add device 0000:ff:03.0
(XEN) PCI add device 0000:ff:03.1
(XEN) PCI add device 0000:ff:03.2
(XEN) PCI add device 0000:ff:03.4
(XEN) PCI add device 0000:ff:04.0
(XEN) PCI add device 0000:ff:04.1
(XEN) PCI add device 0000:ff:04.2
(XEN) PCI add device 0000:ff:04.3
(XEN) PCI add device 0000:ff:05.0
(XEN) PCI add device 0000:ff:05.1
(XEN) PCI add device 0000:ff:05.2
(XEN) PCI add device 0000:ff:05.3
(XEN) PCI add device 0000:ff:06.0
(XEN) PCI add device 0000:ff:06.1
(XEN) PCI add device 0000:ff:06.2
(XEN) PCI add device 0000:ff:06.3
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) traps.c:2503:d0 Domain attempted WRMSR 00000000000001fc from 0x0000000000000002 to 0x0000000000000000.
(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] fault addr 46fc7b000, iommu reg = ffff82c3ffd54000
(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 46fc7b
(XEN)     root_entry = ffff83043ffe5000
(XEN)     root_entry[f] = e6f7001
(XEN)     context = ffff83000e6f7000
(XEN)     context[8] = 0_0
(XEN)     ctxt_entry[8] not present
(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] fault addr 46fc7b000, iommu reg = ffff82c3ffd54000
(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 46fc7b
(XEN)     root_entry = ffff83043ffe5000
(XEN)     root_entry[f] = e6f7001
(XEN)     context = ffff83000e6f7000
(XEN)     context[8] = 0_0
(XEN)     ctxt_entry[8] not present
(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] fault addr 46fc7b000, iommu reg = ffff82c3ffd54000
(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 46fc7b
(XEN)     root_entry = ffff83043ffe5000
(XEN)     root_entry[f] = e6f7001
(XEN)     context = ffff83000e6f7000
(XEN)     context[8] = 0_0
(XEN)     ctxt_entry[8] not present
(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] fault addr 472723000, iommu reg = ffff82c3ffd54000
(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 472723
(XEN)     root_entry = ffff83043ffe5000
(XEN)     root_entry[f] = e6f7001
(XEN)     context = ffff83000e6f7000
(XEN)     context[8] = 0_0
(XEN)     ctxt_entry[8] not present

[-- Attachment #4: lspci.log --]
[-- Type: text/plain, Size: 106254 bytes --]

00:00.0 Host bridge [0600]: Intel Corporation 5520 I/O Hub to ESI Port [8086:3406] (rev 22)
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit-
		Address: 00000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>

00:02.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 [8086:3409] (rev 22) (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=11, subordinate=11, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fbe00000-fbefffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device [8086:0000]
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00000  Data: 4029
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:03.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 [8086:340a] (rev 22) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=09, subordinate=10, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: d7b00000-dfffffff
	Prefetchable memory behind bridge: 00000000c0000000-00000000cfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device [8086:0000]
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00000  Data: 4031
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #51, PowerLimit 75.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 [8086:340e] (rev 22) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=05, subordinate=08, sec-latency=0
	I/O behind bridge: 0000c000-0000dfff
	Memory behind bridge: f4000000-fbdfffff
	Prefetchable memory behind bridge: 00000000a8000000-00000000bfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device [8086:0000]
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00000  Data: 4039
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #55, PowerLimit 75.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [160 v0] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:13.0 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller [8086:342d] (rev 22) (prog-if 20 [IO(X)-APIC])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Region 0: Memory at fec8a000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [6c] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

00:14.0 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e] (rev 22) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Kernel driver in use: i7core_edac
	Kernel modules: i7core_edac

00:14.1 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422] (rev 22) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

00:14.2 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers [8086:3423] (rev 22) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

00:14.3 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers [8086:3438] (rev 22) (prog-if 00 [8259])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:1a.0 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 [8086:3a37] (prog-if 00 [UHCI])
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 16
	Region 4: I/O ports at 9980 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1a.1 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 [8086:3a38] (prog-if 00 [UHCI])
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 21
	Region 4: I/O ports at 9a00 [disabled] [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: pciback

00:1a.2 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 [8086:3a39] (prog-if 00 [UHCI])
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin D routed to IRQ 19
	Region 4: I/O ports at 9a80 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1a.7 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 [8086:3a3c] (prog-if 20 [EHCI])
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at f3df8000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ehci-pci

00:1b.0 Audio device [0403]: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller [8086:3a3e]
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 222
	Region 0: Memory at f3df4000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 4089
	Capabilities: [70] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE- FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=0 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
	Capabilities: [130 v1] Root Complex Link
		Desc:	PortNumber=0f ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd-hda-intel

00:1c.0 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 [8086:3a40] (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 00001000-00001fff
	Memory behind bridge: 40000000-401fffff
	Prefetchable memory behind bridge: 0000000040200000-00000000403fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00000  Data: 4041
	Capabilities: [90] Subsystem: eVga.com. Corp. Device [3842:101a]
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Root Complex Link
		Desc:	PortNumber=01 ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:1c.2 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 3 [8086:3a44] (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 0000b000-0000bfff
	Memory behind bridge: f3f00000-f3ffffff
	Prefetchable memory behind bridge: 0000000040400000-00000000405fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #3, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00000  Data: 4049
	Capabilities: [90] Subsystem: eVga.com. Corp. Device [3842:101a]
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Root Complex Link
		Desc:	PortNumber=03 ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:1c.4 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 [8086:3a48] (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000a000-0000afff
	Memory behind bridge: f3e00000-f3efffff
	Prefetchable memory behind bridge: 0000000040600000-00000000407fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v1) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #5, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot-
		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00000  Data: 4051
	Capabilities: [90] Subsystem: eVga.com. Corp. Device [3842:101a]
	Capabilities: [a0] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Root Complex Link
		Desc:	PortNumber=05 ComponentID=00 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed1c000
	Kernel driver in use: pcieport
	Kernel modules: shpchp

00:1d.0 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 [8086:3a34] (prog-if 00 [UHCI])
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 23
	Region 4: I/O ports at 9b00 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1d.1 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 [8086:3a35] (prog-if 00 [UHCI])
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 19
	Region 4: I/O ports at 9b80 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1d.2 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 [8086:3a36] (prog-if 00 [UHCI])
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin C routed to IRQ 18
	Region 4: I/O ports at 9c00 [size=32]
	Capabilities: [50] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: uhci_hcd

00:1d.7 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 [8086:3a3a] (prog-if 20 [EHCI])
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 23
	Region 0: Memory at f3dfa000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ehci-pci

00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 90) (prog-if 01 [Subtractive decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] Subsystem: eVga.com. Corp. Device [3842:101a]

00:1f.0 ISA bridge [0601]: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller [8086:3a16]
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>
	Kernel driver in use: lpc_ich
	Kernel modules: lpc_ich

00:1f.2 SATA controller [0106]: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller [8086:3a22] (prog-if 01 [AHCI 1.0])
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 221
	Region 0: I/O ports at 9d00 [size=8]
	Region 1: I/O ports at 9f00 [size=4]
	Region 2: I/O ports at 9e80 [size=8]
	Region 3: I/O ports at 9e00 [size=4]
	Region 4: I/O ports at 9d80 [size=32]
	Region 5: Memory at f3dfc000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: [80] MSI: Enable+ Count=1/16 Maskable- 64bit-
		Address: fee00000  Data: 4079
	Capabilities: [70] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
	Capabilities: [b0] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: ahci
	Kernel modules: ahci

00:1f.3 SMBus [0c05]: Intel Corporation 82801JI (ICH10 Family) SMBus Controller [8086:3a30]
	Subsystem: eVga.com. Corp. Device [3842:101a]
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at f3dfe000 (64-bit, non-prefetchable) [size=256]
	Region 4: I/O ports at 0400 [size=32]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c-i801

02:00.0 Ethernet controller [0200]: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller [11ab:4380] (rev 10)
	Subsystem: eVga.com. Corp. Device [3842:abcd]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 220
	Region 0: Memory at f3edc000 (64-bit, non-prefetchable) [size=16K]
	Region 2: I/O ports at af00 [size=256]
	Expansion ROM at f3ee0000 [disabled] [size=128K]
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 4071
	Capabilities: [c0] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <32us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L0s L1 Enabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP+ Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [130 v1] Device Serial Number 40-0c-0e-ff-ff-bc-1f-00
	Kernel driver in use: sky2
	Kernel modules: sky2

03:00.0 Ethernet controller [0200]: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller [11ab:4380] (rev 10)
	Subsystem: eVga.com. Corp. Device [3842:abcd]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 219
	Region 0: Memory at f3fdc000 (64-bit, non-prefetchable) [size=16K]
	Region 2: I/O ports at bf00 [size=256]
	Expansion ROM at f3fe0000 [disabled] [size=128K]
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 4069
	Capabilities: [c0] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <32us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L0s L1 Enabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [130 v1] Device Serial Number 3f-0c-0e-ff-ff-bc-1f-00
	Kernel driver in use: sky2
	Kernel modules: sky2

05:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=05, secondary=06, subordinate=08, sec-latency=0
	I/O behind bridge: 0000c000-0000dfff
	Memory behind bridge: f4000000-fbdfffff
	Prefetchable memory behind bridge: 00000000a8000000-00000000bfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] Express (v2) Upstream Port, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-SlotPowerLimit 75.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] Subsystem: NVIDIA Corporation Device [10de:c55e]
	Kernel driver in use: pcieport
	Kernel modules: shpchp

06:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=06, secondary=08, subordinate=08, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: f8000000-fbdfffff
	Prefetchable memory behind bridge: 00000000b4000000-00000000bfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #1, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Kernel driver in use: pcieport
	Kernel modules: shpchp

06:02.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: f4000000-f7ffffff
	Prefetchable memory behind bridge: 00000000a8000000-00000000b3ffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 5GT/s, Width x16, ASPM L0s, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #3, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Kernel driver in use: pcieport
	Kernel modules: shpchp

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF100GL [Quadro 5000] [10de:06d9] (rev a3) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. Device [1043:8342]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 39
	Region 0: Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=32M]
	Region 1: Memory at a8000000 (64-bit, prefetchable) [disabled] [size=128M]
	Region 3: Memory at b0000000 (64-bit, prefetchable) [disabled] [size=64M]
	Region 5: I/O ports at cf80 [disabled] [size=128]
	Expansion ROM at f7f00000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Kernel driver in use: pciback
	Kernel modules: nouveau, nvidiafb

07:00.1 Audio device [0403]: NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:8342]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 38
	Region 0: Memory at f7ffc000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: pciback
	Kernel modules: snd-hda-intel

08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF100GL [Quadro 6000] [10de:06d8] (rev a3) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation Device [10de:075f]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 30
	Region 0: Memory at f8000000 (32-bit, non-prefetchable) [disabled] [size=32M]
	Region 1: Memory at b8000000 (64-bit, prefetchable) [disabled] [size=128M]
	Region 3: Memory at b4000000 (64-bit, prefetchable) [disabled] [size=64M]
	Region 5: I/O ports at df80 [disabled] [size=128]
	Expansion ROM at fbd00000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Kernel driver in use: pciback
	Kernel modules: nouveau, nvidiafb

08:00.1 Audio device [0403]: NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5] (rev a1)
	Subsystem: NVIDIA Corporation Device [10de:075f]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 37
	Region 0: Memory at fbdfc000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: pciback
	Kernel modules: snd-hda-intel

09:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=09, secondary=0a, subordinate=10, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: d7b00000-dfffffff
	Prefetchable memory behind bridge: 00000000c0000000-00000000cfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] Express (v2) Upstream Port, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-SlotPowerLimit 75.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] Subsystem: NVIDIA Corporation Device [10de:c55e]
	Kernel driver in use: pcieport
	Kernel modules: shpchp

0a:00.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=0a, secondary=10, subordinate=10, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: d7f00000-dfffffff
	Prefetchable memory behind bridge: 00000000c0000000-00000000cfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #1, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Kernel driver in use: pcieport
	Kernel modules: shpchp

0a:02.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=0a, secondary=0d, subordinate=0f, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: d7c00000-d7efffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #3, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Kernel driver in use: pcieport
	Kernel modules: shpchp

0a:03.0 PCI bridge [0604]: NVIDIA Corporation NF200 PCIe 2.0 switch [10de:05b1] (rev a3) (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=0a, secondary=0b, subordinate=0c, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: d7b00000-d7bfffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [60] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #3, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #4, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Kernel driver in use: pcieport
	Kernel modules: shpchp

0b:00.0 PCI bridge [0604]: Creative Labs [SB X-Fi Xtreme Audio] CA0110-IBG PCI to PCIe Bridge [1102:7006] (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=0b, secondary=0c, subordinate=0c, sec-latency=64
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: d7b00000-d7bfffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
		Bridge: PM- B3+
	Capabilities: [60] MSI: Enable- Count=1/16 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [80] Subsystem: Creative Labs Device [1102:0010]
	Capabilities: [90] Express (v1) PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel modules: shpchp

0c:00.0 Audio device [0403]: Creative Labs [SB X-Fi Xtreme Audio] CA0110-IBG [1102:0009]
	Subsystem: Creative Labs SB1040 [1102:0018]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 36
	Region 0: Memory at d7bfc000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: [dc] Power Management version 3
		Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: pciback
	Kernel modules: snd-hda-intel

0d:00.0 PCI bridge [0604]: Intel Corporation 80333 Segment-A PCI Express-to-PCI Express Bridge [8086:0370] (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=0d, secondary=0f, subordinate=0f, sec-latency=64
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: d7c00000-d7efffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [44] Express (v1) PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [5c] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [6c] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [d8] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=133MHz
		Status: Dev=0d:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
		Upstream: Capacity=65535 CommitmentLimit=65535
		Downstream: Capacity=65535 CommitmentLimit=65535
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [300 v1] Power Budgeting <?>
	Kernel modules: shpchp

0d:00.2 PCI bridge [0604]: Intel Corporation 80333 Segment-B PCI Express-to-PCI Express Bridge [8086:0372] (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Bus: primary=0d, secondary=0e, subordinate=0e, sec-latency=64
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [44] Express (v1) PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
	Capabilities: [5c] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [6c] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [d8] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=133MHz
		Status: Dev=0d:00.2 64bit- 133MHz- SCD- USC- SCO- SRD-
		Upstream: Capacity=65535 CommitmentLimit=65535
		Downstream: Capacity=65535 CommitmentLimit=65535
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [300 v1] Power Budgeting <?>
	Kernel modules: shpchp

0f:0e.0 RAID bus controller [0104]: Adaptec AAC-RAID [9005:0285]
	Subsystem: Adaptec 31605 [9005:02be]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping+ SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64 (250ns min, 250ns max), Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 24
	Region 0: Memory at d7c00000 (64-bit, non-prefetchable) [size=2M]
	Expansion ROM at d7e80000 [disabled] [size=512K]
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [d0] MSI: Enable- Count=1/2 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [e0] PCI-X non-bridge device
		Command: DPERE- ERO- RBC=512 OST=4
		Status: Dev=0f:0e.0 64bit+ 133MHz+ SCD- USC- DC=bridge DMMRBC=1024 DMOST=4 DMCRS=16 RSCEM- 266MHz- 533MHz-
	Kernel driver in use: aacraid
	Kernel modules: aacraid

10:00.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI RV770 [Radeon HD 4850] [1002:9442] (prog-if 00 [VGA controller])
	Subsystem: PC Partner Limited Sapphire HD 4850 512MB GDDR3 PCI-E Dual Slot Fansink [174b:e810]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin A routed to IRQ 218
	Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at d8000000 (64-bit, non-prefetchable) [size=64K]
	Region 4: I/O ports at e000 [size=256]
	Expansion ROM at d7fe0000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 4061
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Kernel driver in use: radeon
	Kernel modules: radeon

10:00.1 Audio device [0403]: Advanced Micro Devices [AMD] nee ATI RV770 HDMI Audio [Radeon HD 4850/4870] [1002:aa30]
	Subsystem: PC Partner Limited Sapphire HD 4850 512MB GDDR3 PCI-E Dual Slot Fansink [174b:aa30]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 256 bytes
	Interrupt: pin B routed to IRQ 223
	Region 0: Memory at dc000000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 4099
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd-hda-intel

11:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194] (rev 03) (prog-if 30 [XHCI])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 29
	Region 0: Memory at fbefe000 (64-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [90] MSI-X: Enable- Count=8 Masked-
		Vector table: BAR=0 offset=00001000
		PBA: BAR=0 offset=00001080
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP+ Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Device Serial Number ff-ff-ff-ff-ff-ff-ff-ff
	Capabilities: [150 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Kernel driver in use: pciback
	Kernel modules: xhci-hcd

fe:00.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers [8086:2c70]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:00.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder [8086:2d81]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:02.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 0 [8086:2d90]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:02.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 0 [8086:2d91]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:02.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 0 [8086:2d92]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:02.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 1 [8086:2d93]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:02.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 1 [8086:2d94]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:02.5 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 1 [8086:2d95]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:03.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers [8086:2d98]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:03.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder [8086:2d99]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:03.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers [8086:2d9a]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:03.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers [8086:2d9c]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:04.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control [8086:2da0]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:04.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address [8086:2da1]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:04.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank [8086:2da2]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:04.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control [8086:2da3]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:05.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control [8086:2da8]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:05.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address [8086:2da9]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:05.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank [8086:2daa]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:05.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control [8086:2dab]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:06.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control [8086:2db0]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:06.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address [8086:2db1]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:06.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank [8086:2db2]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

fe:06.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control [8086:2db3]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:00.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers [8086:2c70]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:00.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder [8086:2d81]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 0 [8086:2d90]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 0 [8086:2d91]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 0 [8086:2d92]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 1 [8086:2d93]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 1 [8086:2d94]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:02.5 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 1 [8086:2d95]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:03.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers [8086:2d98]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:03.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder [8086:2d99]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:03.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers [8086:2d9a]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:03.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers [8086:2d9c]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:04.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control [8086:2da0]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:04.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address [8086:2da1]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:04.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank [8086:2da2]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:04.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control [8086:2da3]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:05.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control [8086:2da8]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:05.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address [8086:2da9]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:05.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank [8086:2daa]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:05.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control [8086:2dab]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:06.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control [8086:2db0]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:06.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address [8086:2db1]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:06.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank [8086:2db2]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

ff:06.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control [8086:2db3]
	Subsystem: Intel Corporation Device [8086:8086]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0


[-- Attachment #5: lspci-t.log --]
[-- Type: text/plain, Size: 8425 bytes --]

-+-[0000:ff]-+-00.0  Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers [8086:2c70]
 |           +-00.1  Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder [8086:2d81]
 |           +-02.0  Intel Corporation Xeon 5600 Series QPI Link 0 [8086:2d90]
 |           +-02.1  Intel Corporation Xeon 5600 Series QPI Physical 0 [8086:2d91]
 |           +-02.2  Intel Corporation Xeon 5600 Series Mirror Port Link 0 [8086:2d92]
 |           +-02.3  Intel Corporation Xeon 5600 Series Mirror Port Link 1 [8086:2d93]
 |           +-02.4  Intel Corporation Xeon 5600 Series QPI Link 1 [8086:2d94]
 |           +-02.5  Intel Corporation Xeon 5600 Series QPI Physical 1 [8086:2d95]
 |           +-03.0  Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers [8086:2d98]
 |           +-03.1  Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder [8086:2d99]
 |           +-03.2  Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers [8086:2d9a]
 |           +-03.4  Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers [8086:2d9c]
 |           +-04.0  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control [8086:2da0]
 |           +-04.1  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address [8086:2da1]
 |           +-04.2  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank [8086:2da2]
 |           +-04.3  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control [8086:2da3]
 |           +-05.0  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control [8086:2da8]
 |           +-05.1  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address [8086:2da9]
 |           +-05.2  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank [8086:2daa]
 |           +-05.3  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control [8086:2dab]
 |           +-06.0  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control [8086:2db0]
 |           +-06.1  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address [8086:2db1]
 |           +-06.2  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank [8086:2db2]
 |           \-06.3  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control [8086:2db3]
 +-[0000:fe]-+-00.0  Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers [8086:2c70]
 |           +-00.1  Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder [8086:2d81]
 |           +-02.0  Intel Corporation Xeon 5600 Series QPI Link 0 [8086:2d90]
 |           +-02.1  Intel Corporation Xeon 5600 Series QPI Physical 0 [8086:2d91]
 |           +-02.2  Intel Corporation Xeon 5600 Series Mirror Port Link 0 [8086:2d92]
 |           +-02.3  Intel Corporation Xeon 5600 Series Mirror Port Link 1 [8086:2d93]
 |           +-02.4  Intel Corporation Xeon 5600 Series QPI Link 1 [8086:2d94]
 |           +-02.5  Intel Corporation Xeon 5600 Series QPI Physical 1 [8086:2d95]
 |           +-03.0  Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers [8086:2d98]
 |           +-03.1  Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder [8086:2d99]
 |           +-03.2  Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers [8086:2d9a]
 |           +-03.4  Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers [8086:2d9c]
 |           +-04.0  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control [8086:2da0]
 |           +-04.1  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address [8086:2da1]
 |           +-04.2  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank [8086:2da2]
 |           +-04.3  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control [8086:2da3]
 |           +-05.0  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control [8086:2da8]
 |           +-05.1  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address [8086:2da9]
 |           +-05.2  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank [8086:2daa]
 |           +-05.3  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control [8086:2dab]
 |           +-06.0  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control [8086:2db0]
 |           +-06.1  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address [8086:2db1]
 |           +-06.2  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank [8086:2db2]
 |           \-06.3  Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control [8086:2db3]
 \-[0000:00]-+-00.0  Intel Corporation 5520 I/O Hub to ESI Port [8086:3406]
             +-02.0-[11]----00.0  NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194]
             +-03.0-[09-10]----00.0-[0a-10]--+-00.0-[10]--+-00.0  Advanced Micro Devices [AMD] nee ATI RV770 [Radeon HD 4850] [1002:9442]
             |                               |            \-00.1  Advanced Micro Devices [AMD] nee ATI RV770 HDMI Audio [Radeon HD 4850/4870] [1002:aa30]
             |                               +-02.0-[0d-0f]--+-00.0-[0f]----0e.0  Adaptec AAC-RAID [9005:0285]
             |                               |               \-00.2-[0e]--
             |                               \-03.0-[0b-0c]----00.0-[0c]----00.0  Creative Labs [SB X-Fi Xtreme Audio] CA0110-IBG [1102:0009]
             +-07.0-[05-08]----00.0-[06-08]--+-00.0-[08]--+-00.0  NVIDIA Corporation GF100GL [Quadro 6000] [10de:06d8]
             |                               |            \-00.1  NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5]
             |                               \-02.0-[07]--+-00.0  NVIDIA Corporation GF100GL [Quadro 5000] [10de:06d9]
             |                                            \-00.1  NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5]
             +-13.0  Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller [8086:342d]
             +-14.0  Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e]
             +-14.1  Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422]
             +-14.2  Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers [8086:3423]
             +-14.3  Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers [8086:3438]
             +-1a.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 [8086:3a37]
             +-1a.1  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 [8086:3a38]
             +-1a.2  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 [8086:3a39]
             +-1a.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 [8086:3a3c]
             +-1b.0  Intel Corporation 82801JI (ICH10 Family) HD Audio Controller [8086:3a3e]
             +-1c.0-[04]--
             +-1c.2-[03]----00.0  Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller [11ab:4380]
             +-1c.4-[02]----00.0  Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller [11ab:4380]
             +-1d.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 [8086:3a34]
             +-1d.1  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 [8086:3a35]
             +-1d.2  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 [8086:3a36]
             +-1d.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 [8086:3a3a]
             +-1e.0-[01]--
             +-1f.0  Intel Corporation 82801JIR (ICH10R) LPC Interface Controller [8086:3a16]
             +-1f.2  Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller [8086:3a22]
             \-1f.3  Intel Corporation 82801JI (ICH10 Family) SMBus Controller [8086:3a30]

[-- Attachment #6: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 12:14   ` Gordan Bobic
@ 2013-09-11 12:31     ` Jan Beulich
  2013-09-11 12:45       ` Gordan Bobic
  0 siblings, 1 reply; 40+ messages in thread
From: Jan Beulich @ 2013-09-11 12:31 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: xen-devel

 >>> On 11.09.13 at 14:14, Gordan Bobic <gordan@bobich.net> wrote:
> On Wed, 11 Sep 2013 12:53:09 +0100, "Jan Beulich" <JBeulich@suse.com> 
>  wrote:
>>>>> On 11.09.13 at 13:05, Gordan Bobic <gordan@bobich.net> wrote:
>>> I found this:
>>>
>>>  http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html 
>>>
>>>  while looking for a solution to a similar problem. I am
>>>  facing a similar issue with LSI (8408E, 3081E-R) and
>>>  Adaptec (31605) SAS cards. Was there ever a proper, more general
>>>  fix or workaround for this issue?
>>>
>>>  These SAS cards experience these problems in dom0. When running
>>>  a vanilla kernel on bare metal, they work OK without intel_iommu
>>>  set. As soon as I set intel_iommu, the same thing happens (on
>>>  bare metal, not dom0).
>>>
>>>  Clearly there is something badly broken with multiple layers
>>>  of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>>>  root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)
>>
>> The link above has some (hackish) workarounds - did you try
>> them?
> 
>  Not yet. The thing that bothers me is that he workaround
>  involves hard-coding the PCI device ID which is _nasty_
>  and unstable.

I said "hackish", didn't I? Of course such a change would not
have the slightest chance of going into any repo. But knowing
whether it helps may allow thinking of an acceptable
workaround.

>> In any event, seeing a hypervisor log with "iommu=debug" might
>> shed further light on this: For one, we might be able to see which
>> exact devices are present in the ACPI tables. And we would see
>> which device(s) eventual faults originate from.
> 
>  The thing that bothers me is that this happens in dom0 even
>  with iommu=dom0-passthrough being set.
>  iommu=dom0-passthrough,workaround_bios_bug doesn't help,
>  either

They're not meant to deal with this sort of an impossible (in
theory) situation.

>  And lo and behold, I do have phantom PCI devices after all!
>  lspci shows no device with ID 0000:0f:01.0

Not exactly: Phantom functions can't be at function 0. Irrespective
of that - do the device coordinates somehow correlate with the
problematic controller (IOW: lspci output and a full log would help)?

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 11:57     ` Jan Beulich
@ 2013-09-11 12:19       ` Gordan Bobic
  2013-09-11 12:56         ` Pasi Kärkkäinen
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-09-11 12:19 UTC (permalink / raw)
  To: xen-devel

 On Wed, 11 Sep 2013 12:57:10 +0100, "Jan Beulich" <JBeulich@suse.com> 
 wrote:
>>>> On 11.09.13 at 13:44, Gordan Bobic <gordan@bobich.net> wrote:
>> This got me thinking - if the problem is broken IOMMU 
>> implementation,
>>  is the IOMMU _actually_ required for PCI passthrough to HVM
>>  guests if all the memory holes and BARs are made exactly the same
>>  in dom0 and domU? If vBAR=pBAR, then surely there is no memory
>>  range remapping to be done anyway - which means that there
>>  is no need for the strict IOMMU requirements (over and above
>>  the requirements and caveats of PV domUs).
>
> But with this you ignore the need to handle device bus mastering
> activities. In order to work without IOMMU, the guest's memory
> addresses would also require guest-physical = machine-physical.

 Hmm... that would be harder to achieve, mainly due to legacy
 stuff like base memory. But if (fingers crossed) DMA doesn't
 occur below 1MB, maybe the map can be bodged to emulate the
 addresses below 1MB, carve out as small a chunk of 32-bit
 memory as we can get away with for each guest, and map the
 rest with vmem=pmem in 64-bit memory range. But you are
 right - that gets way more complicated than I originally
 envisaged - all for the sake of supporting legacy
 and Steam boot-loader systems. :(

 Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 11:53 ` Jan Beulich
@ 2013-09-11 12:14   ` Gordan Bobic
  2013-09-11 12:31     ` Jan Beulich
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-09-11 12:14 UTC (permalink / raw)
  To: xen-devel

 On Wed, 11 Sep 2013 12:53:09 +0100, "Jan Beulich" <JBeulich@suse.com> 
 wrote:
>>>> On 11.09.13 at 13:05, Gordan Bobic <gordan@bobich.net> wrote:
>> I found this:
>>
>>  http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>>
>>  while looking for a solution to a similar problem. I am
>>  facing a similar issue with LSI (8408E, 3081E-R) and
>>  Adaptec (31605) SAS cards. Was there ever a proper, more general
>>  fix or workaround for this issue?
>>
>>  These SAS cards experience these problems in dom0. When running
>>  a vanilla kernel on bare metal, they work OK without intel_iommu
>>  set. As soon as I set intel_iommu, the same thing happens (on
>>  bare metal, not dom0).
>>
>>  Clearly there is something badly broken with multiple layers
>>  of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>>  root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)
>
> The link above has some (hackish) workarounds - did you try
> them?

 Not yet. The thing that bothers me is that he workaround
 involves hard-coding the PCI device ID which is _nasty_
 and unstable.

> The link above, however, doesn't indicate any relationship to
> multiple bridges being in between, so it may not match what
> you're observing.

 The impression I got was that the "invisible" devices the
 previous thread was referring to were bridges on the SAS
 card. But I may have misunderstood.

> In any event, seeing a hypervisor log with "iommu=debug" might
> shed further light on this: For one, we might be able to see which
> exact devices are present in the ACPI tables. And we would see
> which device(s) eventual faults originate from.

 The thing that bothers me is that this happens in dom0 even
 with iommu=dom0-passthrough being set.
 iommu=dom0-passthrough,workaround_bios_bug doesn't help,
 either

 And lo and behold, I do have phantom PCI devices after all!
 lspci shows no device with ID 0000:0f:01.0


 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 857f15000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 857f15
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present
 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 858a35000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 858a35
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present
 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 471df8000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 471df8
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present
 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 46fc22000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 46fc22
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present

 Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 11:44   ` Gordan Bobic
@ 2013-09-11 11:57     ` Jan Beulich
  2013-09-11 12:19       ` Gordan Bobic
  0 siblings, 1 reply; 40+ messages in thread
From: Jan Beulich @ 2013-09-11 11:57 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: xen-devel

 >>> On 11.09.13 at 13:44, Gordan Bobic <gordan@bobich.net> wrote:
> This got me thinking - if the problem is broken IOMMU implementation,
>  is the IOMMU _actually_ required for PCI passthrough to HVM
>  guests if all the memory holes and BARs are made exactly the same
>  in dom0 and domU? If vBAR=pBAR, then surely there is no memory
>  range remapping to be done anyway - which means that there
>  is no need for the strict IOMMU requirements (over and above
>  the requirements and caveats of PV domUs).

But with this you ignore the need to handle device bus mastering
activities. In order to work without IOMMU, the guest's memory
addresses would also require guest-physical = machine-physical.

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 11:05 Gordan Bobic
  2013-09-11 11:25 ` Gordan Bobic
@ 2013-09-11 11:53 ` Jan Beulich
  2013-09-11 12:14   ` Gordan Bobic
  1 sibling, 1 reply; 40+ messages in thread
From: Jan Beulich @ 2013-09-11 11:53 UTC (permalink / raw)
  To: Gordan Bobic; +Cc: xen-devel

 >>> On 11.09.13 at 13:05, Gordan Bobic <gordan@bobich.net> wrote:
> I found this:
> 
>  http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html 
> 
>  while looking for a solution to a similar problem. I am
>  facing a similar issue with LSI (8408E, 3081E-R) and
>  Adaptec (31605) SAS cards. Was there ever a proper, more general
>  fix or workaround for this issue?
> 
>  These SAS cards experience these problems in dom0. When running
>  a vanilla kernel on bare metal, they work OK without intel_iommu
>  set. As soon as I set intel_iommu, the same thing happens (on
>  bare metal, not dom0).
> 
>  Clearly there is something badly broken with multiple layers
>  of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>  root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)

The link above has some (hackish) workarounds - did you try
them?

The link above, however, doesn't indicate any relationship to
multiple bridges being in between, so it may not match what
you're observing.

In any event, seeing a hypervisor log with "iommu=debug" might
shed further light on this: For one, we might be able to see which
exact devices are present in the ACPI tables. And we would see
which device(s) eventual faults originate from.

Jan

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 11:25 ` Gordan Bobic
@ 2013-09-11 11:44   ` Gordan Bobic
  2013-09-11 11:57     ` Jan Beulich
  0 siblings, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-09-11 11:44 UTC (permalink / raw)
  To: xen-devel

 This got me thinking - if the problem is broken IOMMU implementation,
 is the IOMMU _actually_ required for PCI passthrough to HVM
 guests if all the memory holes and BARs are made exactly the same
 in dom0 and domU? If vBAR=pBAR, then surely there is no memory
 range remapping to be done anyway - which means that there
 is no need for the strict IOMMU requirements (over and above
 the requirements and caveats of PV domUs).

 In turn, this would enable PCI passthrough (incl. secondary VGA,
 unless I am very much mistaken) to HVM guests while running xen
 with iommu=0. It shifts the design from virtualization to
 partitioning, which I see having obvious advantages and no
 disadvantages (e.g. VM migration doesn't work with PCI
 passthrough anyway).

 The reason I am mentioning this is because I'm working on
 a vhole=phole+vBAR=pBAR patch set anyway, and this would
 be a neat logical extension that would help me work around
 yet more problems on what appears to be a fairly common
 hardware implementation bug.

 Gordan

 On Wed, 11 Sep 2013 12:25:18 +0100, Gordan Bobic <gordan@bobich.net> 
 wrote:
> It looks like I'm definitely not the first person to
> hit this problem:
>
> 
> http://www.gossamer-threads.com/lists/xen/users/168557?do=post_view_threaded#168557
>
> No responses or workarounds suggested back then. :(
>
> Gordan
>
> On Wed, 11 Sep 2013 12:05:35 +0100, Gordan Bobic <gordan@bobich.net> 
> wrote:
>> I found this:
>>
>> http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>>
>> while looking for a solution to a similar problem. I am
>> facing a similar issue with LSI (8408E, 3081E-R) and
>> Adaptec (31605) SAS cards. Was there ever a proper, more general
>> fix or workaround for this issue?
>>
>> These SAS cards experience these problems in dom0. When running
>> a vanilla kernel on bare metal, they work OK without intel_iommu
>> set. As soon as I set intel_iommu, the same thing happens (on
>> bare metal, not dom0).
>>
>> Clearly there is something badly broken with multiple layers
>> of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>> root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)
>>
>> I tried iommu=dom0-passthrough and it doesn't appear to have
>> helped.
>>
>> I am not seeing similar problems with other PCIe devices that
>> are also, in theory doing DMA (e.g. GPUs), but LSI and Adapted
>> controllers appear to be affected for some reason.
>>
>> Is there anything else I could try/do to make this work?
>>
>> Gordan
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
  2013-09-11 11:05 Gordan Bobic
@ 2013-09-11 11:25 ` Gordan Bobic
  2013-09-11 11:44   ` Gordan Bobic
  2013-09-11 11:53 ` Jan Beulich
  1 sibling, 1 reply; 40+ messages in thread
From: Gordan Bobic @ 2013-09-11 11:25 UTC (permalink / raw)
  To: xen-devel

 It looks like I'm definitely not the first person to
 hit this problem:

 http://www.gossamer-threads.com/lists/xen/users/168557?do=post_view_threaded#168557

 No responses or workarounds suggested back then. :(

 Gordan

 On Wed, 11 Sep 2013 12:05:35 +0100, Gordan Bobic <gordan@bobich.net> 
 wrote:
> I found this:
>
> http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>
> while looking for a solution to a similar problem. I am
> facing a similar issue with LSI (8408E, 3081E-R) and
> Adaptec (31605) SAS cards. Was there ever a proper, more general
> fix or workaround for this issue?
>
> These SAS cards experience these problems in dom0. When running
> a vanilla kernel on bare metal, they work OK without intel_iommu
> set. As soon as I set intel_iommu, the same thing happens (on
> bare metal, not dom0).
>
> Clearly there is something badly broken with multiple layers
> of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
> root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)
>
> I tried iommu=dom0-passthrough and it doesn't appear to have
> helped.
>
> I am not seeing similar problems with other PCIe devices that
> are also, in theory doing DMA (e.g. GPUs), but LSI and Adapted
> controllers appear to be affected for some reason.
>
> Is there anything else I could try/do to make this work?
>
> Gordan
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))
@ 2013-09-11 11:05 Gordan Bobic
  2013-09-11 11:25 ` Gordan Bobic
  2013-09-11 11:53 ` Jan Beulich
  0 siblings, 2 replies; 40+ messages in thread
From: Gordan Bobic @ 2013-09-11 11:05 UTC (permalink / raw)
  To: xen-devel

 I found this:

 http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html

 while looking for a solution to a similar problem. I am
 facing a similar issue with LSI (8408E, 3081E-R) and
 Adaptec (31605) SAS cards. Was there ever a proper, more general
 fix or workaround for this issue?

 These SAS cards experience these problems in dom0. When running
 a vanilla kernel on bare metal, they work OK without intel_iommu
 set. As soon as I set intel_iommu, the same thing happens (on
 bare metal, not dom0).

 Clearly there is something badly broken with multiple layers
 of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
 root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)

 I tried iommu=dom0-passthrough and it doesn't appear to have
 helped.

 I am not seeing similar problems with other PCIe devices that
 are also, in theory doing DMA (e.g. GPUs), but LSI and Adapted
 controllers appear to be affected for some reason.

 Is there anything else I could try/do to make this work?

 Gordan

^ permalink raw reply	[flat|nested] 40+ messages in thread

end of thread, other threads:[~2014-02-24 10:14 UTC | newest]

Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-01-07 11:26 Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i)) Wu, Feng
2014-01-07 11:35 ` Gordan Bobic
2014-01-07 12:15   ` Jan Beulich
2014-01-07 12:42     ` Gordan Bobic
2014-01-07 14:38       ` Konrad Rzeszutek Wilk
2014-01-07 14:47         ` Jan Beulich
2014-01-07 15:40           ` Konrad Rzeszutek Wilk
  -- strict thread matches above, loose matches on Subject: below --
2013-09-11 11:05 Gordan Bobic
2013-09-11 11:25 ` Gordan Bobic
2013-09-11 11:44   ` Gordan Bobic
2013-09-11 11:57     ` Jan Beulich
2013-09-11 12:19       ` Gordan Bobic
2013-09-11 12:56         ` Pasi Kärkkäinen
2013-09-11 11:53 ` Jan Beulich
2013-09-11 12:14   ` Gordan Bobic
2013-09-11 12:31     ` Jan Beulich
2013-09-11 12:45       ` Gordan Bobic
2013-09-11 13:03         ` Jan Beulich
2013-09-11 13:10           ` Gordan Bobic
2013-09-11 13:22             ` Jan Beulich
2013-09-11 13:26               ` Gordan Bobic
2013-09-11 13:36                 ` Jan Beulich
2013-09-12  6:20                   ` Zhang, Yang Z
2013-12-11 18:32                     ` Konrad Rzeszutek Wilk
2013-12-11 21:15                       ` Gordan Bobic
2013-12-11 21:30                         ` Konrad Rzeszutek Wilk
2013-12-13 11:13                           ` Jan Beulich
2013-12-13 14:43                             ` Konrad Rzeszutek Wilk
2013-12-13 14:56                               ` Jan Beulich
2013-12-13 15:27                                 ` Gordan Bobic
2014-01-06 20:26                                   ` Konrad Rzeszutek Wilk
2014-01-06 21:45                                     ` Konrad Rzeszutek Wilk
2014-01-07  3:17                                       ` Zhang, Yang Z
2014-01-07 10:35                                         ` Gordan Bobic
2014-01-07 10:38                                           ` Andrew Cooper
2014-01-07 10:44                                             ` Gordan Bobic
2014-02-21 19:08                                               ` Konrad Rzeszutek Wilk
2014-02-24 10:14                                                 ` Jan Beulich
2013-09-11 13:23           ` Gordan Bobic
2013-09-11 13:34             ` Jan Beulich

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.