Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya]

+Linux-PCI.

Please use mailing list email syntax moving forward. (inline and 75 characters per line)

On 1/18/2018 1:21 AM, Ron Yuan wrote:
> Bjorn, 
> I believe no one in current CC list would against be cc'd in larger group. 
> 
> Sinan, 
> we do understand that BIOS have full control of MRRS. But the problem is once kernel boot with pcie_bus_perf, MRRS setting would be overwritten by the code we complained. 
> Btw, the reason we need to use pcie_bus_perf is because Intel white paper indicated system should use pcie_bus_perf to support NVMe SSD hot plug, specifically for a correct MPS to be set. 
> 
> So I think at the moment back to the code, we should 
> 1. when we don’t agree on what's the best value to set, at least remove the code MRRS=MPS, leave MRRS to default value instead of changing it incorrectly. The setting is for one device, hence the device should be responsible for itself, choosing a suitable value as default.  

As I indicated below, MRRS is not a value that a PCIe endpoint device can choose.
It depends on the root complex silicon. Some chip internal bus may handle 4k writes
poorly; whereas, another one can handle it better.

Also, according to the spec; the default value of MRRS is 512 not 4096. 

Coming back to why MRRS=MPS? I think this is because of a isochronous device statement
in the spec.

"Software must limit the Max_Payload_Size for each path that supports isochronous to meet
the isochronous latency. For example, all traffic flowing on a path from an isochronous
capable device to the Root Complex should be limited to packets that do not exceed
the Max_Payload_Size required to meet the isochronous latency requirements."

This is again a hefty price to pay for some specific PCIE endpoint type that not everybody
cares about.

> 2. try to get a common understanding what is the best performance for MRRS, maybe try largest MRRS supported by the system/device, or have additional command line input. 
> 
> Item 1 is something we should do as first step.
> For item 2, we need bigger group of experienced PCIe engineers to discuss.  

We need to understand that MPS and MRRS have much broader impact. All silicon is different.

I don't think there is a common solution to this and my preference is for these parameters
to be specified along with the perf option. Then, we can adjust things accordingly.

Let's see what others think about this.

> Here is one link for your reference:
> https://www.xilinx.com/support/documentation/white_papers/wp350.pdf 
> 

Sinan

> Thanks!
> Ron
> -----Original Message-----
> From: Bjorn Helgaas [mailto:bjorn.helgaas@gmail.com] 
> Sent: 2018年1月17日 22:27
> To: Sinan Kaya <okaya@codeaurora.org>
> Cc: Ron Yuan <ron.yuan@memblaze.com>; Bo Chen <bo.chen@memblaze.com>; William Huang <william.huang@memblaze.com>; Fengming Wu <fengming.wu@memblaze.com>; Bjorn Helgaas <helgaas@kernel.org>; Jason Jiang <jason.jiang@microsemi.com>; Radjendirane Codandaramane <radjendirane.codanda@microsemi.com>; Ramyakanth Edupuganti <Ramyakanth.Edupuganti@microsemi.com>; William Cheng <william.cheng@microsemi.com>; Kim Helper (khelper) <khelper@micron.com>
> Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device
> 
> I'd be happy to participate in this discussion, but I'd prefer to include the linux-pci@vger.kernel.org mailing list because that is public, archived, and searchable.  It also includes people who have worked on MPS and MRRs in the past.  If anyone doesn't want to be cc'd on that list, let me know and I'll remove you from the cc list before I respond.
> 
> On Wed, Jan 17, 2018 at 7:30 AM, Sinan Kaya <okaya@codeaurora.org> wrote:
>> On 1/17/2018 12:04 AM, Ron Yuan wrote:
>>>
>>> The code we refer to is
>>> static void pcie_write_mrrs(struct pci_dev *dev) { ...
>>>     /* For Max performance, the MRRS must be set to the largest supported
>>>      * value.  However, it cannot be configured larger than the MPS the
>>>      * device or the bus can support.  This should already be properly
>>>      * configured by a prior call to pcie_write_mps.
>>>      */
>>>     mrrs = pcie_get_mps(dev);
>>> ...
>>> }
>>>
>>> Here the statement " However, it cannot be configured larger than the MPS the device or the bus can support." is obviously incorrect.
>>> A simple against statement is from PCIe spec, which define default MPS to 128B, and *default MRRS to 512B*.
>>> From HW understanding, MRRS is multiple of MPS in the system, it must be no less than MPS.
>>
>> Yeah, I mentioned it in my presentation too. MPS is not equal to MRRS. The only requirement is for MRRS to be a multiple of MPS.
>>
>>>
>>> Look at the code of Pcie_bus_safe/Pcie_bus_perf/Pcie_bus_peer2peer, it is not handling MRRS correctly.
>>> Peer2peer mode should actually use the smallest 128B for MRRS, which 
>>> is also discussed in https://patchwork.kernel.org/patch/8747521/
>>> Safe mode may be better match MRRS to MDS for safety reason.
>>> Perf mode should set MRRS to largest accepted value, 4k for example.
>>
>> I don't disagree.
>>
>> Choosing 4k is problematic. It could be 1k for one person and 2k for another person.We can have some configuration around this via kernel command line if Bjorn agrees.
>>
>>>
>>> Your reply says some hw logic may have poorer performance with larger MRRS sounds new to me, some HW engineer may have better look.
>>> Then in this case, "leave device recommended MRRS untouched" is more appropriate for performance purpose.
>>> Currently I can see a lot of device is using 256B MPS, and force downgrade default 4k/512B MRRS to 256B is actually getting poorer performance.
>>>
>>
>> Most BIOS allows you to choose the MPS and MRRS settings. If your BIOS does, then I recommend you to use that option at this moment for tuning your system along with the safe boot option.
>>
>>> Thanks again for your time!
>>>
>>> Ron
>>> Beijing Memblaze Technology
>>>
>>> -----Original Message-----
>>> From: Sinan Kaya [mailto:okaya@codeaurora.org]
>>> Sent: 2018年1月17日 12:21
>>> To: Bo Chen <bo.chen@memblaze.com>
>>> Cc: Ron Yuan <ron.yuan@memblaze.com>; William Huang 
>>> <william.huang@memblaze.com>; Fengming Wu <fengming.wu@memblaze.com>; 
>>> Bjorn Helgaas <helgaas@kernel.org>
>>> Subject: Re: One Question About PCIe BUS Config Type with 
>>> pcie_bus_safe or pcie_bus_perf On NVMe Device
>>>
>>> +Bjorn,
>>>
>>> Hi Bo,
>>>
>>> On 1/16/2018 9:58 PM, Bo Chen wrote:
>>>> Hi Sinan,
>>>>
>>>> We went through linux pci related code found that nvme device has different performance when kernel boot with pcie_bus_safe or pcie_bus_perf.
>>>> For details, when boot with pcie_bus_safe (eg: nvme device A: MaxPayload 128B, MaxReadReq 4096B), after pci bus init and enumerate, device A devCtl remains 128B for MaxPayload and 4096B for MaxReadReq.
>>>
>>> pcie_bus_safe option is used to balance the maximum payload size setting across the bus. The goal is to make sure that maximum payload size is consistent. That's why, it is called safe.
>>> It also honors the maximum read request size setting done by the BIOS. It does not touch the MRRS setting.
>>>
>>>> While boot with pcie_bus_perf, we found pcie_write_mrrs set the read request size to the same with MPS. For nvme device A, it means both MaxPayload and MaxReadReq are set to 128B so that we observed it has limited device performance in read req.
>>>> >From our point of view on nvme device, the setting of pcie_bus_perf has lower performance compared with setting with pcie_bus_safe.
>>>> Is there any consideration for pcie_bus_perf to set MaxReadReq the same value with pcie_bus_safe? Is there any risk when we boot with pcie_bus_perf for nvme device?
>>>> Thank you very much!
>>>>
>>>       /*
>>>        * If using the "performance" PCIe config, we clamp the
>>>        * read rq size to the max packet size to prevent the
>>>        * host bridge generating requests larger than we can
>>>        * cope with
>>>        */
>>>
>>> I found this comment in the code that seems to match the behavior you are seeing.
>>>
>>> Problem with maximum read request size is that there is no one-size-fits-all solution. It depends on the bus-fabric of the silicon you are using. An MRRS of 4k might behave slower than 1k. 1k might behave better than 512 bytes.
>>>
>>> The recommendation is for these settings to be adjustable in BIOS and tweak it according to the application.
>>>
>>> I was planning to make a change in this area but I was warned by our hardware engineers that performance does not always scale with MRRS setting.
>>>
>>> Sinan
>>>
>>
>>
>> --
>> Sinan Kaya
>> Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
>> Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Bjorn Helgaas]

On Thu, Jan 18, 2018 at 11:24:52AM -0500, Sinan Kaya wrote:
> On 1/18/2018 1:21 AM, Ron Yuan wrote:
> > we do understand that BIOS have full control of MRRS. But the
> > problem is once kernel boot with pcie_bus_perf, MRRS setting would
> > be overwritten by the code we complained.  Btw, the reason we need
> > to use pcie_bus_perf is because Intel white paper indicated system
> > should use pcie_bus_perf to support NVMe SSD hot plug,
> > specifically for a correct MPS to be set. 
> > 
> > So I think at the moment back to the code, we should 
> > 1. when we don’t agree on what's the best value to set, at least
> > remove the code MRRS=MPS, leave MRRS to default value instead of
> > changing it incorrectly. The setting is for one device, hence the
> > device should be responsible for itself, choosing a suitable value
> > as default.  
> 
> As I indicated below, MRRS is not a value that a PCIe endpoint
> device can choose.  It depends on the root complex silicon. Some
> chip internal bus may handle 4k writes poorly; whereas, another one
> can handle it better.
> 
> Also, according to the spec; the default value of MRRS is 512 not
> 4096. 
> 
> Coming back to why MRRS=MPS? I think this is because of a
> isochronous device statement in the spec.
> 
> "Software must limit the Max_Payload_Size for each path that
> supports isochronous to meet the isochronous latency. For example,
> all traffic flowing on a path from an isochronous capable device to
> the Root Complex should be limited to packets that do not exceed the
> Max_Payload_Size required to meet the isochronous latency
> requirements."
> 
> This is again a hefty price to pay for some specific PCIE endpoint
> type that not everybody cares about.

In PCIE_BUS_PERFORMANCE mode, pcie_write_mrrs() does try to set
MRRS=MPS, but the reason is not to support isochronous.  The reason is
to allow us to use larger MPS settings than we otherwise would.

Consider a switch leading to an endpoint that supports only MPS=128.
The simplest approach would be to configure every device in the fabric
with MPS=128.  That guarantees the endpoint will never receive a TLP
with a payload larger than 128 bytes.

Here's my understanding of how PCIE_BUS_PERFORMANCE works:

There are two ways an endpoint may receive TLPs with data payloads:
(1) Memory Write Requests that target the endpoint, and (2)
Completions with Data in response to Memory Read Requests generated by
the endpoint.

In PCIE_BUS_PERFORMANCE mode, we assume Memory Write Requests are not
an issue because:

  - We assume a CPU Memory Write Request is never larger than MPS (128
    bytes in this case).  This is fairly safe because CPUs generally
    can't write more than a cache line in one request, and most CPUs
    have cache lines of 128 bytes or less.

  - We assume there's no peer-to-peer DMA, so other devices in the
    fabric will never send Memory Write Requests to the endpoint, so
    we don't need to limit their MPS settings.

That leaves Completions.  We limit the size of Completions by limiting
MRRS.  If we set the endpoint's MRRS to its MPS (128 in this case), it
will never request more than MPS bytes at a time, so it will never
receive a Completion with more than MPS bytes.

Therefore, we may be able to configure other devices in the fabric
with MPS larger than 128, which may benefit those devices.

> > 2. try to get a common understanding what is the best performance
> > for MRRS, maybe try largest MRRS supported by the system/device,
> > or have additional command line input. 
> > 
> > Item 1 is something we should do as first step.  For item 2, we
> > need bigger group of experienced PCIe engineers to discuss.  
> 
> We need to understand that MPS and MRRS have much broader impact.
> All silicon is different.
> 
> I don't think there is a common solution to this and my preference
> is for these parameters to be specified along with the perf option.
> Then, we can adjust things accordingly.
> 
> Let's see what others think about this.
> 
> > Here is one link for your reference:
> > https://www.xilinx.com/support/documentation/white_papers/wp350.pdf 

> > On Wed, Jan 17, 2018 at 7:30 AM, Sinan Kaya <okaya@codeaurora.org> wrote:
> >> On 1/17/2018 12:04 AM, Ron Yuan wrote:
> >>> The code we refer to is
> >>> static void pcie_write_mrrs(struct pci_dev *dev) { ...
> >>>     /* For Max performance, the MRRS must be set to the largest supported
> >>>      * value.  However, it cannot be configured larger than the MPS the
> >>>      * device or the bus can support.  This should already be properly
> >>>      * configured by a prior call to pcie_write_mps.
> >>>      */
> >>>     mrrs = pcie_get_mps(dev);
> >>> ...
> >>> }
> >>>
> >>> Here the statement " However, it cannot be configured larger
> >>> than the MPS the device or the bus can support." is obviously
> >>> incorrect.  A simple against statement is from PCIe spec, which
> >>> define default MPS to 128B, and *default MRRS to 512B*.  From HW
> >>> understanding, MRRS is multiple of MPS in the system, it must be
> >>> no less than MPS.
> >>
> >> Yeah, I mentioned it in my presentation too. MPS is not equal to
> >> MRRS. The only requirement is for MRRS to be a multiple of MPS.

>From a hardware point of view, I do not think there is any requirement
that MRRS be related to MPS.  PCIe r4.0, sec 7.5.3.4 does not mention
any constraints on MRRS.

>From sec 2.2.2, MPS sets the maximum TLP payload size for a function:
  - A function never generates a TLP larger than its MPS
  - A function treats received TLPs larger than its MPS as malformed

MPS applies to TLPs with data payloads.  These are either Memory Write
Requests that carry data directly, or Completions with Data in
response to a Memory Read Request.

MRRS sets the maximum Read Request size a function will generate.  A
Memory Read Request itself contains no data payload and MPS doesn't
apply to the request.

The completer of the Read Request will generate Completion TLPs
carrying the requested data.  The payload size of these Completion
TLPs is determined by the completer's MPS setting, and it is
software's responsibility to ensure that the completer's MPS is
smaller than or equal to the requester's MPS.

The requester's MRRS indirectly affects the payload size because the
completer will not generate a payload larger than the read request
size.

Consider the case where a function has MPS=256 and MRRS=1024: if the
function generates a 1024-byte read request, it should receive 4
completions, each with a 256-byte data payload.

Or, consider the case where MPS=1024 and MRRS=256: if the function
generates a 256-byte read request, it should receive 1 completion with
a 256-byte data payload.

In this case, the function *could* receive a Memory Write Request with
a 1024-byte data payload.  CPUs normally write at most a cacheline,
which is probably smaller than 1024 bytes, so these large writes would
most likely be peer-to-peer DMA writes.

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b03e7495a862

> >>> Look at the code of
> >>> Pcie_bus_safe/Pcie_bus_perf/Pcie_bus_peer2peer, it is not
> >>> handling MRRS correctly.  Peer2peer mode should actually use the
> >>> smallest 128B for MRRS, which is also discussed in
> >>> https://patchwork.kernel.org/patch/8747521/
> >>> Safe mode may be better match MRRS to MDS for safety reason.
> >>> Perf mode should set MRRS to largest accepted value, 4k for
> >>> example.
> >>
> >> I don't disagree.
> >>
> >> Choosing 4k is problematic. It could be 1k for one person and 2k
> >> for another person. We can have some configuration around this via
> >> kernel command line if Bjorn agrees.

In peer-to-peer mode, we currently set MPS=128 for all devices and we
don't touch MRRS.  Why do you think we should set MRRS=128?

As far as I can tell, that is not required by spec.  Limiting MRRS may
prevent a device from monopolizing bandwidth, and that may be
worthwhile to do, but it is not specifically a peer-to-peer issue and
I don't want to artificially tie it to MPS.

I try to avoid command-line parameters because they're hard for users
and distros to deal with.  In this case particularly, the issues are
hard to understand so I want to untangle them a bit before we add more
knobs.

> >>> Your reply says some hw logic may have poorer performance with
> >>> larger MRRS sounds new to me, some HW engineer may have better
> >>> look.  Then in this case, "leave device recommended MRRS
> >>> untouched" is more appropriate for performance purpose.
> >>> Currently I can see a lot of device is using 256B MPS, and force
> >>> downgrade default 4k/512B MRRS to 256B is actually getting
> >>> poorer performance.
> >>
> >> Most BIOS allows you to choose the MPS and MRRS settings. If your
> >> BIOS does, then I recommend you to use that option at this moment
> >> for tuning your system along with the safe boot option.
> >>
> >>> Thanks again for your time!
> >>>
> >>> Ron
> >>> Beijing Memblaze Technology
> >>>
> >>> -----Original Message-----
> >>> From: Sinan Kaya [mailto:okaya@codeaurora.org]
> >>> Sent: 2018年1月17日 12:21
> >>> To: Bo Chen <bo.chen@memblaze.com>
> >>> Cc: Ron Yuan <ron.yuan@memblaze.com>; William Huang 
> >>> <william.huang@memblaze.com>; Fengming Wu <fengming.wu@memblaze.com>; 
> >>> Bjorn Helgaas <helgaas@kernel.org>
> >>> Subject: Re: One Question About PCIe BUS Config Type with 
> >>> pcie_bus_safe or pcie_bus_perf On NVMe Device
> >>>
> >>> +Bjorn,
> >>>
> >>> Hi Bo,
> >>>
> >>> On 1/16/2018 9:58 PM, Bo Chen wrote:
> >>>> Hi Sinan,
> >>>>
> >>>> We went through linux pci related code found that nvme device
> >>>> has different performance when kernel boot with pcie_bus_safe
> >>>> or pcie_bus_perf.  For details, when boot with pcie_bus_safe
> >>>> (eg: nvme device A: MaxPayload 128B, MaxReadReq 4096B), after
> >>>> pci bus init and enumerate, device A devCtl remains 128B for
> >>>> MaxPayload and 4096B for MaxReadReq.
> >>>
> >>> pcie_bus_safe option is used to balance the maximum payload size
> >>> setting across the bus. The goal is to make sure that maximum
> >>> payload size is consistent. That's why, it is called safe.  It
> >>> also honors the maximum read request size setting done by the
> >>> BIOS. It does not touch the MRRS setting.
> >>>
> >>>> While boot with pcie_bus_perf, we found pcie_write_mrrs set the
> >>>> read request size to the same with MPS. For nvme device A, it
> >>>> means both MaxPayload and MaxReadReq are set to 128B so that we
> >>>> observed it has limited device performance in read req.
> >>>>
> >>>> >From our point of view on nvme device, the setting of
> >>>> >pcie_bus_perf has lower performance compared with setting with
> >>>> >pcie_bus_safe.
> >>>>
> >>>> Is there any consideration for pcie_bus_perf to set MaxReadReq
> >>>> the same value with pcie_bus_safe? Is there any risk when we
> >>>> boot with pcie_bus_perf for nvme device?

Correct me if I'm wrong:

  - The NVMe device has a maximum MPS of 128
  - The NVMe device supports MRRS of 4096
  - In SAFE mode, Linux sets MPS=128 and doesn't touch MRRS, so the
    BIOS setting of MRRS=4096 remains
  - In PERFORMANCE mode, Linux sets MPS=128 and MRRS=128

In SAFE mode, if Linux sets MPS=128, it means there's some device that
doesn't support MPS larger than 128.  That's obvious in this case
because the NVMe device only supports MPS=128.

In PERFORMANCE mode, Linux tries to keep larger MPS settings above the
NVMe endpoint and sets MRRS as described above to make this safe.
However, in this case, we apparently can't use larger MPS settings
anywhere else, so all we did is reduce MRRS, which reduces your
performance.

We should be able to be smarter about this, but I don't have a patch
to propose.  Maybe we can work on this during the next cycle.

> >>>       /*
> >>>        * If using the "performance" PCIe config, we clamp the
> >>>        * read rq size to the max packet size to prevent the
> >>>        * host bridge generating requests larger than we can
> >>>        * cope with
> >>>        */
> >>>
> >>> I found this comment in the code that seems to match the
> >>> behavior you are seeing.
> >>>
> >>> Problem with maximum read request size is that there is no
> >>> one-size-fits-all solution. It depends on the bus-fabric of the
> >>> silicon you are using. An MRRS of 4k might behave slower than
> >>> 1k. 1k might behave better than 512 bytes.
> >>>
> >>> The recommendation is for these settings to be adjustable in
> >>> BIOS and tweak it according to the application.
> >>>
> >>> I was planning to make a change in this area but I was warned by
> >>> our hardware engineers that performance does not always scale
> >>> with MRRS setting.

Bjorn

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya]

On 1/19/2018 3:51 PM, Bjorn Helgaas wrote:
> On Thu, Jan 18, 2018 at 11:24:52AM -0500, Sinan Kaya wrote:
>> On 1/18/2018 1:21 AM, Ron Yuan wrote:
>>> we do understand that BIOS have full control of MRRS. But the
>>> problem is once kernel boot with pcie_bus_perf, MRRS setting would
>>> be overwritten by the code we complained.  Btw, the reason we need
>>> to use pcie_bus_perf is because Intel white paper indicated system
>>> should use pcie_bus_perf to support NVMe SSD hot plug,
>>> specifically for a correct MPS to be set. 
>>>
>>> So I think at the moment back to the code, we should 
>>> 1. when we don’t agree on what's the best value to set, at least
>>> remove the code MRRS=MPS, leave MRRS to default value instead of
>>> changing it incorrectly. The setting is for one device, hence the
>>> device should be responsible for itself, choosing a suitable value
>>> as default.  
>>
>> As I indicated below, MRRS is not a value that a PCIe endpoint
>> device can choose.  It depends on the root complex silicon. Some
>> chip internal bus may handle 4k writes poorly; whereas, another one
>> can handle it better.
>>
>> Also, according to the spec; the default value of MRRS is 512 not
>> 4096. 
>>
>> Coming back to why MRRS=MPS? I think this is because of a
>> isochronous device statement in the spec.
>>
>> "Software must limit the Max_Payload_Size for each path that
>> supports isochronous to meet the isochronous latency. For example,
>> all traffic flowing on a path from an isochronous capable device to
>> the Root Complex should be limited to packets that do not exceed the
>> Max_Payload_Size required to meet the isochronous latency
>> requirements."
>>
>> This is again a hefty price to pay for some specific PCIE endpoint
>> type that not everybody cares about.
> 
> In PCIE_BUS_PERFORMANCE mode, pcie_write_mrrs() does try to set
> MRRS=MPS, but the reason is not to support isochronous.  The reason is
> to allow us to use larger MPS settings than we otherwise would.
>

OK. I didn't know the history behind the change. I was looking at MPS
and MRRS references in the spec.
 
> Consider a switch leading to an endpoint that supports only MPS=128.
> The simplest approach would be to configure every device in the fabric
> with MPS=128.  That guarantees the endpoint will never receive a TLP
> with a payload larger than 128 bytes.
> 
> Here's my understanding of how PCIE_BUS_PERFORMANCE works:
> 
> There are two ways an endpoint may receive TLPs with data payloads:
> (1) Memory Write Requests that target the endpoint, and (2)
> Completions with Data in response to Memory Read Requests generated by
> the endpoint.
> 
> In PCIE_BUS_PERFORMANCE mode, we assume Memory Write Requests are not
> an issue because:
> 
>   - We assume a CPU Memory Write Request is never larger than MPS (128
>     bytes in this case).  This is fairly safe because CPUs generally
>     can't write more than a cache line in one request, and most CPUs
>     have cache lines of 128 bytes or less.
> 

Fair assumption.

>   - We assume there's no peer-to-peer DMA, so other devices in the
>     fabric will never send Memory Write Requests to the endpoint, so
>     we don't need to limit their MPS settings.
> 
> That leaves Completions.  We limit the size of Completions by limiting
> MRRS.  If we set the endpoint's MRRS to its MPS (128 in this case), it
> will never request more than MPS bytes at a time, so it will never
> receive a Completion with more than MPS bytes.
> 
> Therefore, we may be able to configure other devices in the fabric
> with MPS larger than 128, which may benefit those devices.
> 

This is still problematic. One application may be doing a lot of writes
compared to reads. We prefer maximizing endpoint write performance compared
to read performance by reducing the MRRS setting.

>>> 2. try to get a common understanding what is the best performance
>>> for MRRS, maybe try largest MRRS supported by the system/device,
>>> or have additional command line input. 
>>>
>>> Item 1 is something we should do as first step.  For item 2, we
>>> need bigger group of experienced PCIe engineers to discuss.  
>>
>> We need to understand that MPS and MRRS have much broader impact.
>> All silicon is different.
>>
>> I don't think there is a common solution to this and my preference
>> is for these parameters to be specified along with the perf option.
>> Then, we can adjust things accordingly.
>>
>> Let's see what others think about this.
>>
>>> Here is one link for your reference:
>>> https://www.xilinx.com/support/documentation/white_papers/wp350.pdf 
> 
>>> On Wed, Jan 17, 2018 at 7:30 AM, Sinan Kaya <okaya@codeaurora.org> wrote:
>>>> On 1/17/2018 12:04 AM, Ron Yuan wrote:
>>>>> The code we refer to is
>>>>> static void pcie_write_mrrs(struct pci_dev *dev) { ...
>>>>>     /* For Max performance, the MRRS must be set to the largest supported
>>>>>      * value.  However, it cannot be configured larger than the MPS the
>>>>>      * device or the bus can support.  This should already be properly
>>>>>      * configured by a prior call to pcie_write_mps.
>>>>>      */
>>>>>     mrrs = pcie_get_mps(dev);
>>>>> ...
>>>>> }
>>>>>
>>>>> Here the statement " However, it cannot be configured larger
>>>>> than the MPS the device or the bus can support." is obviously
>>>>> incorrect.  A simple against statement is from PCIe spec, which
>>>>> define default MPS to 128B, and *default MRRS to 512B*.  From HW
>>>>> understanding, MRRS is multiple of MPS in the system, it must be
>>>>> no less than MPS.
>>>>
>>>> Yeah, I mentioned it in my presentation too. MPS is not equal to
>>>> MRRS. The only requirement is for MRRS to be a multiple of MPS.
> 
>>From a hardware point of view, I do not think there is any requirement
> that MRRS be related to MPS.  PCIe r4.0, sec 7.5.3.4 does not mention
> any constraints on MRRS.
> 
>>From sec 2.2.2, MPS sets the maximum TLP payload size for a function:
>   - A function never generates a TLP larger than its MPS
>   - A function treats received TLPs larger than its MPS as malformed
> 
> MPS applies to TLPs with data payloads.  These are either Memory Write
> Requests that carry data directly, or Completions with Data in
> response to a Memory Read Request> 
> MRRS sets the maximum Read Request size a function will generate.  A
> Memory Read Request itself contains no data payload and MPS doesn't
> apply to the request.

Agreed, MRRS has an indirect effect on the completions. It determines the
number of completions that endpoint will receive. As the ratio of MPS/MRRS
gets closer to 0, the overhead becomes more significant.

> 
> The completer of the Read Request will generate Completion TLPs
> carrying the requested data.  The payload size of these Completion
> TLPs is determined by the completer's MPS setting, and it is
> software's responsibility to ensure that the completer's MPS is
> smaller than or equal to the requester's MPS.
> 
> The requester's MRRS indirectly affects the payload size because the
> completer will not generate a payload larger than the read request
> size.
> 
> Consider the case where a function has MPS=256 and MRRS=1024: if the
> function generates a 1024-byte read request, it should receive 4
> completions, each with a 256-byte data payload.
> 
> Or, consider the case where MPS=1024 and MRRS=256: if the function
> generates a 256-byte read request, it should receive 1 completion with
> a 256-byte data payload.
> 
> In this case, the function *could* receive a Memory Write Request with
> a 1024-byte data payload.  CPUs normally write at most a cacheline,
> which is probably smaller than 1024 bytes, so these large writes would
> most likely be peer-to-peer DMA writes.

Why do you think that CPU is involved in fetching a memory read response
to a read request?

If an endpoint tries to read 4k from system memory, this would be typically
done by SOC internal fabric not by the CPU. How fast a fabric can feed 1k
to the endpoint depends on the capability of the fabric. Some fabric might
do better with 1k, others could be better with 256 bytes.

CPU would be only involved if it was doing a memory write to the endpoint by
itself. Then, mostly cache line or CPU write instruction size is in effect.

> 
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b03e7495a862
> 
>>>>> Look at the code of
>>>>> Pcie_bus_safe/Pcie_bus_perf/Pcie_bus_peer2peer, it is not
>>>>> handling MRRS correctly.  Peer2peer mode should actually use the
>>>>> smallest 128B for MRRS, which is also discussed in
>>>>> https://patchwork.kernel.org/patch/8747521/
>>>>> Safe mode may be better match MRRS to MDS for safety reason.
>>>>> Perf mode should set MRRS to largest accepted value, 4k for
>>>>> example.
>>>>
>>>> I don't disagree.
>>>>
>>>> Choosing 4k is problematic. It could be 1k for one person and 2k
>>>> for another person. We can have some configuration around this via
>>>> kernel command line if Bjorn agrees.
> 
> In peer-to-peer mode, we currently set MPS=128 for all devices and we
> don't touch MRRS.  Why do you think we should set MRRS=128?
> 
> As far as I can tell, that is not required by spec.  Limiting MRRS may
> prevent a device from monopolizing bandwidth, and that may be
> worthwhile to do, but it is not specifically a peer-to-peer issue and
> I don't want to artificially tie it to MPS.
> 
> I try to avoid command-line parameters because they're hard for users
> and distros to deal with.  In this case particularly, the issues are
> hard to understand so I want to untangle them a bit before we add more
> knobs.

OK.

> 
>>>>> Your reply says some hw logic may have poorer performance with
>>>>> larger MRRS sounds new to me, some HW engineer may have better
>>>>> look.  Then in this case, "leave device recommended MRRS
>>>>> untouched" is more appropriate for performance purpose.
>>>>> Currently I can see a lot of device is using 256B MPS, and force
>>>>> downgrade default 4k/512B MRRS to 256B is actually getting
>>>>> poorer performance.
>>>>
>>>> Most BIOS allows you to choose the MPS and MRRS settings. If your
>>>> BIOS does, then I recommend you to use that option at this moment
>>>> for tuning your system along with the safe boot option.
>>>>
>>>>> Thanks again for your time!
>>>>>
>>>>> Ron
>>>>> Beijing Memblaze Technology
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sinan Kaya [mailto:okaya@codeaurora.org]
>>>>> Sent: 2018年1月17日 12:21
>>>>> To: Bo Chen <bo.chen@memblaze.com>
>>>>> Cc: Ron Yuan <ron.yuan@memblaze.com>; William Huang 
>>>>> <william.huang@memblaze.com>; Fengming Wu <fengming.wu@memblaze.com>; 
>>>>> Bjorn Helgaas <helgaas@kernel.org>
>>>>> Subject: Re: One Question About PCIe BUS Config Type with 
>>>>> pcie_bus_safe or pcie_bus_perf On NVMe Device
>>>>>
>>>>> +Bjorn,
>>>>>
>>>>> Hi Bo,
>>>>>
>>>>> On 1/16/2018 9:58 PM, Bo Chen wrote:
>>>>>> Hi Sinan,
>>>>>>
>>>>>> We went through linux pci related code found that nvme device
>>>>>> has different performance when kernel boot with pcie_bus_safe
>>>>>> or pcie_bus_perf.  For details, when boot with pcie_bus_safe
>>>>>> (eg: nvme device A: MaxPayload 128B, MaxReadReq 4096B), after
>>>>>> pci bus init and enumerate, device A devCtl remains 128B for
>>>>>> MaxPayload and 4096B for MaxReadReq.
>>>>>
>>>>> pcie_bus_safe option is used to balance the maximum payload size
>>>>> setting across the bus. The goal is to make sure that maximum
>>>>> payload size is consistent. That's why, it is called safe.  It
>>>>> also honors the maximum read request size setting done by the
>>>>> BIOS. It does not touch the MRRS setting.
>>>>>
>>>>>> While boot with pcie_bus_perf, we found pcie_write_mrrs set the
>>>>>> read request size to the same with MPS. For nvme device A, it
>>>>>> means both MaxPayload and MaxReadReq are set to 128B so that we
>>>>>> observed it has limited device performance in read req.
>>>>>>
>>>>>> >From our point of view on nvme device, the setting of
>>>>>>> pcie_bus_perf has lower performance compared with setting with
>>>>>>> pcie_bus_safe.
>>>>>>
>>>>>> Is there any consideration for pcie_bus_perf to set MaxReadReq
>>>>>> the same value with pcie_bus_safe? Is there any risk when we
>>>>>> boot with pcie_bus_perf for nvme device?
> 
> Correct me if I'm wrong:
> 
>   - The NVMe device has a maximum MPS of 128
>   - The NVMe device supports MRRS of 4096
>   - In SAFE mode, Linux sets MPS=128 and doesn't touch MRRS, so the
>     BIOS setting of MRRS=4096 remains
>   - In PERFORMANCE mode, Linux sets MPS=128 and MRRS=128
> 
> In SAFE mode, if Linux sets MPS=128, it means there's some device that
> doesn't support MPS larger than 128.  That's obvious in this case
> because the NVMe device only supports MPS=128.
> 
> In PERFORMANCE mode, Linux tries to keep larger MPS settings above the
> NVMe endpoint and sets MRRS as described above to make this safe.
> However, in this case, we apparently can't use larger MPS settings
> anywhere else, so all we did is reduce MRRS, which reduces your
> performance.
> 
> We should be able to be smarter about this, but I don't have a patch
> to propose.  Maybe we can work on this during the next cycle.
> 

Sure, I was planning to work on this once the CRS change gets merged. I
was doing feasibility analysis in the meantime.

>>>>>       /*
>>>>>        * If using the "performance" PCIe config, we clamp the
>>>>>        * read rq size to the max packet size to prevent the
>>>>>        * host bridge generating requests larger than we can
>>>>>        * cope with
>>>>>        */
>>>>>
>>>>> I found this comment in the code that seems to match the
>>>>> behavior you are seeing.
>>>>>
>>>>> Problem with maximum read request size is that there is no
>>>>> one-size-fits-all solution. It depends on the bus-fabric of the
>>>>> silicon you are using. An MRRS of 4k might behave slower than
>>>>> 1k. 1k might behave better than 512 bytes.
>>>>>
>>>>> The recommendation is for these settings to be adjustable in
>>>>> BIOS and tweak it according to the application.
>>>>>
>>>>> I was planning to make a change in this area but I was warned by
>>>>> our hardware engineers that performance does not always scale
>>>>> with MRRS setting.
> 
> Bjorn
> 


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya]

On 1/20/2018 2:20 PM, Sinan Kaya wrote:
>> In this case, the function *could* receive a Memory Write Request with
>> a 1024-byte data payload.  CPUs normally write at most a cacheline,
>> which is probably smaller than 1024 bytes, so these large writes would
>> most likely be peer-to-peer DMA writes.
> Why do you think that CPU is involved in fetching a memory read response
> to a read request?

Never mind. I realized you started talking about Memory Writes after the
completions in the above paragraph now. Another source for memory write could
be an SOC internal DMA memcpy engine as well as a peer-to-peer access.

> 
> If an endpoint tries to read 4k from system memory, this would be typically
> done by SOC internal fabric not by the CPU. How fast a fabric can feed 1k
> to the endpoint depends on the capability of the fabric. Some fabric might
> do better with 1k, others could be better with 256 bytes.


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Bjorn Helgaas]

On Sat, Jan 20, 2018 at 02:20:06PM -0500, Sinan Kaya wrote:
> On 1/19/2018 3:51 PM, Bjorn Helgaas wrote:
> > Consider a switch leading to an endpoint that supports only MPS=128.
> > The simplest approach would be to configure every device in the fabric
> > with MPS=128.  That guarantees the endpoint will never receive a TLP
> > with a payload larger than 128 bytes.
> > 
> > Here's my understanding of how PCIE_BUS_PERFORMANCE works:
> > 
> > There are two ways an endpoint may receive TLPs with data payloads:
> > (1) Memory Write Requests that target the endpoint, and (2)
> > Completions with Data in response to Memory Read Requests generated by
> > the endpoint.
> > 
> > In PCIE_BUS_PERFORMANCE mode, we assume Memory Write Requests are not
> > an issue because:
> > 
> >   - We assume a CPU Memory Write Request is never larger than MPS (128
> >     bytes in this case).  This is fairly safe because CPUs generally
> >     can't write more than a cache line in one request, and most CPUs
> >     have cache lines of 128 bytes or less.
> 
> Fair assumption.
> 
> >   - We assume there's no peer-to-peer DMA, so other devices in the
> >     fabric will never send Memory Write Requests to the endpoint, so
> >     we don't need to limit their MPS settings.
> > 
> > That leaves Completions.  We limit the size of Completions by limiting
> > MRRS.  If we set the endpoint's MRRS to its MPS (128 in this case), it
> > will never request more than MPS bytes at a time, so it will never
> > receive a Completion with more than MPS bytes.
> > 
> > Therefore, we may be able to configure other devices in the fabric
> > with MPS larger than 128, which may benefit those devices.
> 
> This is still problematic. One application may be doing a lot of
> writes compared to reads. We prefer maximizing endpoint write
> performance compared to read performance by reducing the MRRS
> setting.

Help me understand exactly what is problematic.  No matter what your
read/write mix is, a single device in isolation should get the best
performance with both MPS and MRRS at the highest possible settings.

Reducing MPS may be necessary if there are several devices in the
hierarchy and one requires a smaller MPS than the others.  That
obviously reduces the maximum read and write performance.

Reducing the MRRS may be useful to prevent one device from hogging a
link, but of course, it reduces read performance for that device
because we need more read requests.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya]

On 1/22/2018 4:36 PM, Bjorn Helgaas wrote:
>>> That leaves Completions.  We limit the size of Completions by limiting
>>> MRRS.  If we set the endpoint's MRRS to its MPS (128 in this case), it
>>> will never request more than MPS bytes at a time, so it will never
>>> receive a Completion with more than MPS bytes.
>>>
>>> Therefore, we may be able to configure other devices in the fabric
>>> with MPS larger than 128, which may benefit those devices.

> Help me understand exactly what is problematic.  No matter what your
> read/write mix is, a single device in isolation should get the best
> performance with both MPS and MRRS at the highest possible settings.

The performance approach is trying to maximize MPS while reducing MRRS
value to MPS value. Meaning improving write performance while trading
off read performance.

> 
> Reducing MPS may be necessary if there are several devices in the
> hierarchy and one requires a smaller MPS than the others.  That
> obviously reduces the maximum read and write performance.
> 
> Reducing the MRRS may be useful to prevent one device from hogging a
> link, but of course, it reduces read performance for that device
> because we need more read requests.
> 

Maybe, a picture could help.

			root (MPS=256)
			 |
		 ------------------
		/		   \
	   bridge0 (MPS=256)      bridge1 (MPS=128)
	    /				\
	   EP0 (MPS=256)		EP1 (MPS=128)

If I understood this right, code allows the configuration above with
the performance mode so that MPS doesn't have to be uniform across
the tree. 

It just needs to be consistent between the root port and endpoints.

Why are we reducing MRRS in this case?

Are we assuming that root bus cannot handle more than 256 bytes and bridge1
would be starved while root bus is passing the completions to bridge0?

If yes, the answer to the assumption depends on the architecture. One silicon
could serve any completions in any speed. 

It depends on how root bus was implemented. If it is a real one or an emulated one.
I think most silicon emulate root bus.


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Bjorn Helgaas]

On Mon, Jan 22, 2018 at 05:04:03PM -0500, Sinan Kaya wrote:
> On 1/22/2018 4:36 PM, Bjorn Helgaas wrote:
> >>> That leaves Completions.  We limit the size of Completions by
> >>> limiting MRRS.  If we set the endpoint's MRRS to its MPS (128 in
> >>> this case), it will never request more than MPS bytes at a time,
> >>> so it will never receive a Completion with more than MPS bytes.
> >>>
> >>> Therefore, we may be able to configure other devices in the
> >>> fabric with MPS larger than 128, which may benefit those
> >>> devices.
> 
> > Help me understand exactly what is problematic.  No matter what
> > your read/write mix is, a single device in isolation should get
> > the best performance with both MPS and MRRS at the highest
> > possible settings.
> 
> The performance approach is trying to maximize MPS while reducing
> MRRS value to MPS value. Meaning improving write performance while
> trading off read performance.

Right.  The intent of the PERFORMANCE mode is exactly to maximize MPS
(which maximizes write performance) by reducing MRRS in some cases
(which reduces read performance).

You had said:

>>> This is still problematic. One application may be doing a lot of
>>> writes compared to reads. We prefer maximizing endpoint write
>>> performance compared to read performance by reducing the MRRS
>>> setting.

so I thought you had an issue with this, and I was trying to
understand what you wanted instead.

> > Reducing MPS may be necessary if there are several devices in the
> > hierarchy and one requires a smaller MPS than the others.  That
> > obviously reduces the maximum read and write performance.
> > 
> > Reducing the MRRS may be useful to prevent one device from hogging
> > a link, but of course, it reduces read performance for that device
> > because we need more read requests.
> 
> Maybe, a picture could help.
> 
>                root (MPS=256)
>                  |
>          ------------------
>         /                  \
>    bridge0 (MPS=256)      bridge1 (MPS=128)
>       /                       \
>     EP0 (MPS=256)            EP1 (MPS=128)
> 
> If I understood this right, code allows the configuration above with
> the performance mode so that MPS doesn't have to be uniform across
> the tree. 

Yes.  In PERFORMANCE mode, we will set EP1's MRRS=128 and
EP0's MRRS=256, just as you show.

> It just needs to be consistent between the root port and endpoints.

No, it doesn't need to be consistent.  In PERFORMANCE mode, we'll set
the root's MPS=256 and EP1's MPS=128.

(I'm not actually 100% convinced that the PERFORMANCE mode approach of
reducing MRRS is safe, necessary, and maintainable.  I suspect that in
many of the interesting cases, the device we care about is the only
one below a Root Port, and we can get the performance we need by
maximizing MPS and MRRS for that Root Port and its children,
independent of the rest of the system.)

> Why are we reducing MRRS in this case?

We have to set EP1's MRRS=128 so it will never receive a completion
larger than 128 bytes.  If we set EP1's MRRS=256, it could receive
256-byte TLPs, which it would treat as malformed.  (We also assume no
peer-to-peer DMA that targets EP1.)

> Are we assuming that root bus cannot handle more than 256 bytes and
> bridge1 would be starved while root bus is passing the completions
> to bridge0?

We don't have to assume.  Every device tells us via Dev Cap what size
TLPs it can handle.  In your example, I assume the root's Dev Cap
tells us it supports 256-byte TLPs.

PERFORMANCE mode reduces MRRS not because of a starvation issue, but
because reducing EP1's MRRS allows EP0 to use a larger MPS.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya]

On 1/22/2018 5:51 PM, Bjorn Helgaas wrote:
> On Mon, Jan 22, 2018 at 05:04:03PM -0500, Sinan Kaya wrote:
>> On 1/22/2018 4:36 PM, Bjorn Helgaas wrote:

>>> Reducing MPS may be necessary if there are several devices in the
>>> hierarchy and one requires a smaller MPS than the others.  That
>>> obviously reduces the maximum read and write performance.
>>>
>>> Reducing the MRRS may be useful to prevent one device from hogging
>>> a link, but of course, it reduces read performance for that device
>>> because we need more read requests.
>>
>> Maybe, a picture could help.
>>
>>                root (MPS=256)
>>                  |
>>          ------------------
>>         /                  \
>>    bridge0 (MPS=256)      bridge1 (MPS=128)
>>       /                       \
>>     EP0 (MPS=256)            EP1 (MPS=128)
>>
>> If I understood this right, code allows the configuration above with
>> the performance mode so that MPS doesn't have to be uniform across
>> the tree. 
> 
> Yes.  In PERFORMANCE mode, we will set EP1's MRRS=128 and
> EP0's MRRS=256, just as you show.
> 
>> It just needs to be consistent between the root port and endpoints.
> 
> No, it doesn't need to be consistent.  In PERFORMANCE mode, we'll set
> the root's MPS=256 and EP1's MPS=128.
> 
> (I'm not actually 100% convinced that the PERFORMANCE mode approach of
> reducing MRRS is safe, necessary, and maintainable.  I suspect that in
> many of the interesting cases, the device we care about is the only
> one below a Root Port, and we can get the performance we need by
> maximizing MPS and MRRS for that Root Port and its children,
> independent of the rest of the system.)

Maybe, I started seeing more and more NVMe devices behind a switch every
day. That's why, I'm concerned.

> 
>> Why are we reducing MRRS in this case?
> 
> We have to set EP1's MRRS=128 so it will never receive a completion
> larger than 128 bytes.  If we set EP1's MRRS=256, it could receive
> 256-byte TLPs, which it would treat as malformed.  (We also assume no
> peer-to-peer DMA that targets EP1.)

What if we were to keep root port MPS as 128? and not touch the BIOS
configured MRRS (4k) ?

Everybody should be happy, right?

I know there is a rule to check the completions against MPS. Root port
could generate transactions that is a multiple of 128 bytes for reads.

Is there any rule against checking incoming writes?

> 
>> Are we assuming that root bus cannot handle more than 256 bytes and
>> bridge1 would be starved while root bus is passing the completions
>> to bridge0?
> 
> We don't have to assume.  Every device tells us via Dev Cap what size
> TLPs it can handle.  In your example, I assume the root's Dev Cap
> tells us it supports 256-byte TLPs.
> 
> PERFORMANCE mode reduces MRRS not because of a starvation issue, but
> because reducing EP1's MRRS allows EP0 to use a larger MPS.
> 


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Bjorn Helgaas]

On Mon, Jan 22, 2018 at 06:24:22PM -0500, Sinan Kaya wrote:
> On 1/22/2018 5:51 PM, Bjorn Helgaas wrote:
> > On Mon, Jan 22, 2018 at 05:04:03PM -0500, Sinan Kaya wrote:
> >> On 1/22/2018 4:36 PM, Bjorn Helgaas wrote:
> 
> >>> Reducing MPS may be necessary if there are several devices in the
> >>> hierarchy and one requires a smaller MPS than the others.  That
> >>> obviously reduces the maximum read and write performance.
> >>>
> >>> Reducing the MRRS may be useful to prevent one device from hogging
> >>> a link, but of course, it reduces read performance for that device
> >>> because we need more read requests.
> >>
> >> Maybe, a picture could help.
> >>
> >>                root (MPS=256)
> >>                  |
> >>          ------------------
> >>         /                  \
> >>    bridge0 (MPS=256)      bridge1 (MPS=128)
> >>       /                       \
> >>     EP0 (MPS=256)            EP1 (MPS=128)
> >>
> >> If I understood this right, code allows the configuration above with
> >> the performance mode so that MPS doesn't have to be uniform across
> >> the tree. 
> > 
> > Yes.  In PERFORMANCE mode, we will set EP1's MRRS=128 and
> > EP0's MRRS=256, just as you show.
> > 
> >> It just needs to be consistent between the root port and endpoints.
> > 
> > No, it doesn't need to be consistent.  In PERFORMANCE mode, we'll set
> > the root's MPS=256 and EP1's MPS=128.
> > 
> > (I'm not actually 100% convinced that the PERFORMANCE mode approach of
> > reducing MRRS is safe, necessary, and maintainable.  I suspect that in
> > many of the interesting cases, the device we care about is the only
> > one below a Root Port, and we can get the performance we need by
> > maximizing MPS and MRRS for that Root Port and its children,
> > independent of the rest of the system.)
> 
> Maybe, I started seeing more and more NVMe devices behind a switch every
> day. That's why, I'm concerned.

Are there a mix of devices that support large MPS and those that only
support a smaller MPS behind the same switch?

I don't have any actual data about topologies of interest.  I'm just
guessing that high-performance devices will often have their own root
port, without being mixed in with low-speed devices.  It's OK to have
switches in the path; it's the mixing high-speed with low-speed
devices that causes the problems.

> >> Why are we reducing MRRS in this case?
> > 
> > We have to set EP1's MRRS=128 so it will never receive a completion
> > larger than 128 bytes.  If we set EP1's MRRS=256, it could receive
> > 256-byte TLPs, which it would treat as malformed.  (We also assume no
> > peer-to-peer DMA that targets EP1.)
> 
> What if we were to keep root port MPS as 128? and not touch the BIOS
> configured MRRS (4k) ?
> 
> Everybody should be happy, right?

No.  In your picture (which helps a lot, thank you!), if you set the
root's MPS=128, EP1 will be fine no matter what its MRRS is because
the entire path has MPS=128.

But if EP0's MRRS is larger than 128, it's in trouble because it may
issue a 256-byte DMA write, which the root port will treat as
malformed.  The root port, as the receiver of that Memory Write
Request, is required to check the TLP against the MPS in its Device
Control (not Device Capability) register (this is in PCIe r4.0, sec
2.2.2).

> I know there is a rule to check the completions against MPS. 

Sec 2.2.2 says a receiver must check "data payloads".  I think that
includes both Completions and Memory Writes.

> Root port could generate transactions that is a multiple of 128
> bytes for reads.

If the root port generates a 256-byte Memory Read request to EP1,
that's fine because EP1 will only respond with 128-byte completions.
If it sends that 256-byte Memory Read to EP0, we have a problem
because EP0 may generate a 256-byte completion, which will cause an
error if the root port has MPS=128.

> Is there any rule against checking incoming writes?

Sec 2.2.2 says a receiver *must* check the payload size of incoming
Memory Write requests.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya]

On 1/22/2018 7:16 PM, Bjorn Helgaas wrote:
> On Mon, Jan 22, 2018 at 06:24:22PM -0500, Sinan Kaya wrote:
>> On 1/22/2018 5:51 PM, Bjorn Helgaas wrote:
>>> On Mon, Jan 22, 2018 at 05:04:03PM -0500, Sinan Kaya wrote:
>>>> On 1/22/2018 4:36 PM, Bjorn Helgaas wrote:
>>
>>>>> Reducing MPS may be necessary if there are several devices in the
>>>>> hierarchy and one requires a smaller MPS than the others.  That
>>>>> obviously reduces the maximum read and write performance.
>>>>>
>>>>> Reducing the MRRS may be useful to prevent one device from hogging
>>>>> a link, but of course, it reduces read performance for that device
>>>>> because we need more read requests.
>>>>
>>>> Maybe, a picture could help.
>>>>
>>>>                root (MPS=256)
>>>>                  |
>>>>          ------------------
>>>>         /                  \
>>>>    bridge0 (MPS=256)      bridge1 (MPS=128)
>>>>       /                       \
>>>>     EP0 (MPS=256)            EP1 (MPS=128)
>>>>
>>>> If I understood this right, code allows the configuration above with
>>>> the performance mode so that MPS doesn't have to be uniform across
>>>> the tree. 
>>>
>>> Yes.  In PERFORMANCE mode, we will set EP1's MRRS=128 and
>>> EP0's MRRS=256, just as you show.
>>>
>>>> It just needs to be consistent between the root port and endpoints.
>>>
>>> No, it doesn't need to be consistent.  In PERFORMANCE mode, we'll set
>>> the root's MPS=256 and EP1's MPS=128.
>>>
>>> (I'm not actually 100% convinced that the PERFORMANCE mode approach of
>>> reducing MRRS is safe, necessary, and maintainable.  I suspect that in
>>> many of the interesting cases, the device we care about is the only
>>> one below a Root Port, and we can get the performance we need by
>>> maximizing MPS and MRRS for that Root Port and its children,
>>> independent of the rest of the system.)
>>
>> Maybe, I started seeing more and more NVMe devices behind a switch every
>> day. That's why, I'm concerned.
> 
> Are there a mix of devices that support large MPS and those that only
> support a smaller MPS behind the same switch?

I guess we should understand the environment from Ron Yuan. Can we see
lspci -vvv output of your system?

We could maybe detect coexistence of slow and fast device condition and
put some suggestions like moving the card to another slot as a best effort
solution.

> 
> I don't have any actual data about topologies of interest.  I'm just
> guessing that high-performance devices will often have their own root
> port, without being mixed in with low-speed devices.  It's OK to have
> switches in the path; it's the mixing high-speed with low-speed
> devices that causes the problems.

Agreed. Mine was just a general rant. I was curious if we could make it better.

> 
>>>> Why are we reducing MRRS in this case?
>>>
>>> We have to set EP1's MRRS=128 so it will never receive a completion
>>> larger than 128 bytes.  If we set EP1's MRRS=256, it could receive
>>> 256-byte TLPs, which it would treat as malformed.  (We also assume no
>>> peer-to-peer DMA that targets EP1.)
>>
>> What if we were to keep root port MPS as 128? and not touch the BIOS
>> configured MRRS (4k) ?
>>
>> Everybody should be happy, right?
> 
> No.  In your picture (which helps a lot, thank you!), if you set the
> root's MPS=128, EP1 will be fine no matter what its MRRS is because
> the entire path has MPS=128.
> 
> But if EP0's MRRS is larger than 128, it's in trouble because it may
> issue a 256-byte DMA write, which the root port will treat as
> malformed.  The root port, as the receiver of that Memory Write
> Request, is required to check the TLP against the MPS in its Device
> Control (not Device Capability) register (this is in PCIe r4.0, sec
> 2.2.2).

Yeah, this breaks my theory.

> 
>> I know there is a rule to check the completions against MPS. 
> 
> Sec 2.2.2 says a receiver must check "data payloads".  I think that
> includes both Completions and Memory Writes.
> 
>> Root port could generate transactions that is a multiple of 128
>> bytes for reads.
> 
> If the root port generates a 256-byte Memory Read request to EP1,
> that's fine because EP1 will only respond with 128-byte completions.
> If it sends that 256-byte Memory Read to EP0, we have a problem
> because EP0 may generate a 256-byte completion, which will cause an
> error if the root port has MPS=128.
> 
>> Is there any rule against checking incoming writes?
> 
> Sec 2.2.2 says a receiver *must* check the payload size of incoming
> Memory Write requests.
> 


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

RE: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Ron Yuan]

[-- Attachment #1: Type: text/plain, Size: 6812 bytes --]

Bjorn and Sinan,
Thanks for the discussion.

> PERFORMANCE mode reduces MRRS not because of a starvation issue, but 
> because reducing EP1's MRRS allows EP0 to use a larger MPS.
Looks like this case is talking about EP1 requests data directly from EP0, using MRRS to control the return data payload, while still keeping the traffic from EP0 to RC in 256B. This is reasonable. 
However, is this common in the real world? We can see kernel driver usually control the whole system, set MPS well to align all EPs. Only in rare case where different EP has no other sync method, then setting MRRS for EP1 can provide more "safe" for itself. This does not match with "perf" name, so I proposed earlier move the MRRS setting to "safe" mode to make it clear. 

In my situation, data center customer is using many NVMe SSDs, they care most about best performance from each SSD, while there is no out of control EP to EP access. 

I will provide a full lspci log tomorrow. First attach the screen shoot in "safe" mode for different devices. You can see MPS are set to 256B while MRRS default remain differently. In Perf mode, all MRRS would be limited to 256B and loss some performance. 

Thanks
Ron

-----Original Message-----
From: Sinan Kaya [mailto:okaya@codeaurora.org] 
Sent: 2018年1月23日 10:27
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Ron Yuan <ron.yuan@memblaze.com>; Bjorn Helgaas <bhelgaas@google.com>; Bo Chen <bo.chen@memblaze.com>; William Huang <william.huang@memblaze.com>; Fengming Wu <fengming.wu@memblaze.com>; Jason Jiang <jason.jiang@microsemi.com>; Radjendirane Codandaramane <radjendirane.codanda@microsemi.com>; Ramyakanth Edupuganti <Ramyakanth.Edupuganti@microsemi.com>; William Cheng <william.cheng@microsemi.com>; Kim Helper (khelper) <khelper@micron.com>; Linux PCI <linux-pci@vger.kernel.org>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

On 1/22/2018 7:16 PM, Bjorn Helgaas wrote:
> On Mon, Jan 22, 2018 at 06:24:22PM -0500, Sinan Kaya wrote:
>> On 1/22/2018 5:51 PM, Bjorn Helgaas wrote:
>>> On Mon, Jan 22, 2018 at 05:04:03PM -0500, Sinan Kaya wrote:
>>>> On 1/22/2018 4:36 PM, Bjorn Helgaas wrote:
>>
>>>>> Reducing MPS may be necessary if there are several devices in the 
>>>>> hierarchy and one requires a smaller MPS than the others.  That 
>>>>> obviously reduces the maximum read and write performance.
>>>>>
>>>>> Reducing the MRRS may be useful to prevent one device from hogging 
>>>>> a link, but of course, it reduces read performance for that device 
>>>>> because we need more read requests.
>>>>
>>>> Maybe, a picture could help.
>>>>
>>>>                root (MPS=256)
>>>>                  |
>>>>          ------------------
>>>>         /                  \
>>>>    bridge0 (MPS=256)      bridge1 (MPS=128)
>>>>       /                       \
>>>>     EP0 (MPS=256)            EP1 (MPS=128)
>>>>
>>>> If I understood this right, code allows the configuration above 
>>>> with the performance mode so that MPS doesn't have to be uniform 
>>>> across the tree.
>>>
>>> Yes.  In PERFORMANCE mode, we will set EP1's MRRS=128 and EP0's 
>>> MRRS=256, just as you show.
>>>
>>>> It just needs to be consistent between the root port and endpoints.
>>>
>>> No, it doesn't need to be consistent.  In PERFORMANCE mode, we'll 
>>> set the root's MPS=256 and EP1's MPS=128.
>>>
>>> (I'm not actually 100% convinced that the PERFORMANCE mode approach 
>>> of reducing MRRS is safe, necessary, and maintainable.  I suspect 
>>> that in many of the interesting cases, the device we care about is 
>>> the only one below a Root Port, and we can get the performance we 
>>> need by maximizing MPS and MRRS for that Root Port and its children, 
>>> independent of the rest of the system.)
>>
>> Maybe, I started seeing more and more NVMe devices behind a switch 
>> every day. That's why, I'm concerned.
> 
> Are there a mix of devices that support large MPS and those that only 
> support a smaller MPS behind the same switch?

I guess we should understand the environment from Ron Yuan. Can we see lspci -vvv output of your system?

We could maybe detect coexistence of slow and fast device condition and put some suggestions like moving the card to another slot as a best effort solution.

> 
> I don't have any actual data about topologies of interest.  I'm just 
> guessing that high-performance devices will often have their own root 
> port, without being mixed in with low-speed devices.  It's OK to have 
> switches in the path; it's the mixing high-speed with low-speed 
> devices that causes the problems.

Agreed. Mine was just a general rant. I was curious if we could make it better.

> 
>>>> Why are we reducing MRRS in this case?
>>>
>>> We have to set EP1's MRRS=128 so it will never receive a completion 
>>> larger than 128 bytes.  If we set EP1's MRRS=256, it could receive 
>>> 256-byte TLPs, which it would treat as malformed.  (We also assume 
>>> no peer-to-peer DMA that targets EP1.)
>>
>> What if we were to keep root port MPS as 128? and not touch the BIOS 
>> configured MRRS (4k) ?
>>
>> Everybody should be happy, right?
> 
> No.  In your picture (which helps a lot, thank you!), if you set the 
> root's MPS=128, EP1 will be fine no matter what its MRRS is because 
> the entire path has MPS=128.
> 
> But if EP0's MRRS is larger than 128, it's in trouble because it may 
> issue a 256-byte DMA write, which the root port will treat as 
> malformed.  The root port, as the receiver of that Memory Write 
> Request, is required to check the TLP against the MPS in its Device 
> Control (not Device Capability) register (this is in PCIe r4.0, sec 
> 2.2.2).

Yeah, this breaks my theory.

> 
>> I know there is a rule to check the completions against MPS. 
> 
> Sec 2.2.2 says a receiver must check "data payloads".  I think that 
> includes both Completions and Memory Writes.
> 
>> Root port could generate transactions that is a multiple of 128 bytes 
>> for reads.
> 
> If the root port generates a 256-byte Memory Read request to EP1, 
> that's fine because EP1 will only respond with 128-byte completions.
> If it sends that 256-byte Memory Read to EP0, we have a problem 
> because EP0 may generate a 256-byte completion, which will cause an 
> error if the root port has MPS=128.
> 
>> Is there any rule against checking incoming writes?
> 
> Sec 2.2.2 says a receiver *must* check the payload size of incoming 
> Memory Write requests.
> 


--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

[-- Attachment #2: D3559A46-A52F-407a-909A-3BCD16FFDC7B.PNG --]
[-- Type: image/png, Size: 63193 bytes --]

RE: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Ron Yuan]

[-- Attachment #1: Type: text/plain, Size: 7644 bytes --]

Just got the log, see attachment. Kernel is under "perf" mode.
SSD and Ethernet controller are both set to 256B MRRS. 

-----Original Message-----
From: Ron Yuan 
Sent: 2018年1月23日 21:26
To: 'Sinan Kaya' <okaya@codeaurora.org>; Bjorn Helgaas <helgaas@kernel.org>
Cc: Bjorn Helgaas <bhelgaas@google.com>; Bo Chen <bo.chen@memblaze.com>; William Huang <william.huang@memblaze.com>; Fengming Wu <fengming.wu@memblaze.com>; Jason Jiang <jason.jiang@microsemi.com>; Radjendirane Codandaramane <radjendirane.codanda@microsemi.com>; Ramyakanth Edupuganti <Ramyakanth.Edupuganti@microsemi.com>; William Cheng <william.cheng@microsemi.com>; Kim Helper (khelper) <khelper@micron.com>; Linux PCI <linux-pci@vger.kernel.org>
Subject: RE: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

Bjorn and Sinan,
Thanks for the discussion.

> PERFORMANCE mode reduces MRRS not because of a starvation issue, but 
> because reducing EP1's MRRS allows EP0 to use a larger MPS.
Looks like this case is talking about EP1 requests data directly from EP0, using MRRS to control the return data payload, while still keeping the traffic from EP0 to RC in 256B. This is reasonable. 
However, is this common in the real world? We can see kernel driver usually control the whole system, set MPS well to align all EPs. Only in rare case where different EP has no other sync method, then setting MRRS for EP1 can provide more "safe" for itself. This does not match with "perf" name, so I proposed earlier move the MRRS setting to "safe" mode to make it clear. 

In my situation, data center customer is using many NVMe SSDs, they care most about best performance from each SSD, while there is no out of control EP to EP access. 

I will provide a full lspci log tomorrow. First attach the screen shoot in "safe" mode for different devices. You can see MPS are set to 256B while MRRS default remain differently. In Perf mode, all MRRS would be limited to 256B and loss some performance. 

Thanks
Ron

-----Original Message-----
From: Sinan Kaya [mailto:okaya@codeaurora.org]
Sent: 2018年1月23日 10:27
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Ron Yuan <ron.yuan@memblaze.com>; Bjorn Helgaas <bhelgaas@google.com>; Bo Chen <bo.chen@memblaze.com>; William Huang <william.huang@memblaze.com>; Fengming Wu <fengming.wu@memblaze.com>; Jason Jiang <jason.jiang@microsemi.com>; Radjendirane Codandaramane <radjendirane.codanda@microsemi.com>; Ramyakanth Edupuganti <Ramyakanth.Edupuganti@microsemi.com>; William Cheng <william.cheng@microsemi.com>; Kim Helper (khelper) <khelper@micron.com>; Linux PCI <linux-pci@vger.kernel.org>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

On 1/22/2018 7:16 PM, Bjorn Helgaas wrote:
> On Mon, Jan 22, 2018 at 06:24:22PM -0500, Sinan Kaya wrote:
>> On 1/22/2018 5:51 PM, Bjorn Helgaas wrote:
>>> On Mon, Jan 22, 2018 at 05:04:03PM -0500, Sinan Kaya wrote:
>>>> On 1/22/2018 4:36 PM, Bjorn Helgaas wrote:
>>
>>>>> Reducing MPS may be necessary if there are several devices in the 
>>>>> hierarchy and one requires a smaller MPS than the others.  That 
>>>>> obviously reduces the maximum read and write performance.
>>>>>
>>>>> Reducing the MRRS may be useful to prevent one device from hogging 
>>>>> a link, but of course, it reduces read performance for that device 
>>>>> because we need more read requests.
>>>>
>>>> Maybe, a picture could help.
>>>>
>>>>                root (MPS=256)
>>>>                  |
>>>>          ------------------
>>>>         /                  \
>>>>    bridge0 (MPS=256)      bridge1 (MPS=128)
>>>>       /                       \
>>>>     EP0 (MPS=256)            EP1 (MPS=128)
>>>>
>>>> If I understood this right, code allows the configuration above 
>>>> with the performance mode so that MPS doesn't have to be uniform 
>>>> across the tree.
>>>
>>> Yes.  In PERFORMANCE mode, we will set EP1's MRRS=128 and EP0's 
>>> MRRS=256, just as you show.
>>>
>>>> It just needs to be consistent between the root port and endpoints.
>>>
>>> No, it doesn't need to be consistent.  In PERFORMANCE mode, we'll 
>>> set the root's MPS=256 and EP1's MPS=128.
>>>
>>> (I'm not actually 100% convinced that the PERFORMANCE mode approach 
>>> of reducing MRRS is safe, necessary, and maintainable.  I suspect 
>>> that in many of the interesting cases, the device we care about is 
>>> the only one below a Root Port, and we can get the performance we 
>>> need by maximizing MPS and MRRS for that Root Port and its children, 
>>> independent of the rest of the system.)
>>
>> Maybe, I started seeing more and more NVMe devices behind a switch 
>> every day. That's why, I'm concerned.
> 
> Are there a mix of devices that support large MPS and those that only 
> support a smaller MPS behind the same switch?

I guess we should understand the environment from Ron Yuan. Can we see lspci -vvv output of your system?

We could maybe detect coexistence of slow and fast device condition and put some suggestions like moving the card to another slot as a best effort solution.

> 
> I don't have any actual data about topologies of interest.  I'm just 
> guessing that high-performance devices will often have their own root 
> port, without being mixed in with low-speed devices.  It's OK to have 
> switches in the path; it's the mixing high-speed with low-speed 
> devices that causes the problems.

Agreed. Mine was just a general rant. I was curious if we could make it better.

> 
>>>> Why are we reducing MRRS in this case?
>>>
>>> We have to set EP1's MRRS=128 so it will never receive a completion 
>>> larger than 128 bytes.  If we set EP1's MRRS=256, it could receive 
>>> 256-byte TLPs, which it would treat as malformed.  (We also assume 
>>> no peer-to-peer DMA that targets EP1.)
>>
>> What if we were to keep root port MPS as 128? and not touch the BIOS 
>> configured MRRS (4k) ?
>>
>> Everybody should be happy, right?
> 
> No.  In your picture (which helps a lot, thank you!), if you set the 
> root's MPS=128, EP1 will be fine no matter what its MRRS is because 
> the entire path has MPS=128.
> 
> But if EP0's MRRS is larger than 128, it's in trouble because it may 
> issue a 256-byte DMA write, which the root port will treat as 
> malformed.  The root port, as the receiver of that Memory Write 
> Request, is required to check the TLP against the MPS in its Device 
> Control (not Device Capability) register (this is in PCIe r4.0, sec 
> 2.2.2).

Yeah, this breaks my theory.

> 
>> I know there is a rule to check the completions against MPS. 
> 
> Sec 2.2.2 says a receiver must check "data payloads".  I think that 
> includes both Completions and Memory Writes.
> 
>> Root port could generate transactions that is a multiple of 128 bytes 
>> for reads.
> 
> If the root port generates a 256-byte Memory Read request to EP1, 
> that's fine because EP1 will only respond with 128-byte completions.
> If it sends that 256-byte Memory Read to EP0, we have a problem 
> because EP0 may generate a 256-byte completion, which will cause an 
> error if the root port has MPS=128.
> 
>> Is there any rule against checking incoming writes?
> 
> Sec 2.2.2 says a receiver *must* check the payload size of incoming 
> Memory Write requests.
> 


--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

[-- Attachment #2: lspci.log --]
[-- Type: application/octet-stream, Size: 233436 bytes --]

00:00.0 Host bridge: Intel Corporation Device 2020 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 0
	Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [144 v1] Vendor Specific Information: ID=0004 Rev=1 Len=03c <?>
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>

00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at 383ffff2c000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 10
	Region 0: Memory at 383ffff28000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 11
	Region 0: Memory at 383ffff24000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 11
	Region 0: Memory at 383ffff20000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at 383ffff1c000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 10
	Region 0: Memory at 383ffff18000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 11
	Region 0: Memory at 383ffff14000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 11
	Region 0: Memory at 383ffff10000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

00:05.2 System peripheral: Intel Corporation Device 2025 (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

00:05.4 PIC: Intel Corporation Device 2026 (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Device 2026
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at 9d20a000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

00:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

00:11.0 Unassigned class [ff00]: Intel Corporation Device a1ec (rev 08)
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

00:11.5 SATA controller: Intel Corporation Lewisburg SSATA Controller [AHCI mode] (rev 08) (prog-if 01 [AHCI 1.0])
	Subsystem: Intel Corporation Device 7270
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 47
	Region 0: Memory at 9d206000 (32-bit, non-prefetchable) [size=8K]
	Region 1: Memory at 9d209000 (32-bit, non-prefetchable) [size=256]
	Region 2: I/O ports at 3070 [size=8]
	Region 3: I/O ports at 3060 [size=4]
	Region 4: I/O ports at 3020 [size=32]
	Region 5: Memory at 9d180000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00318  Data: 0000
	Capabilities: [70] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
	Kernel driver in use: ahci

00:14.0 USB controller: Intel Corporation Lewisburg USB 3.0 xHCI Controller (rev 08) (prog-if 30 [XHCI])
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 46
	Region 0: Memory at 383ffff00000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
		Address: 00000000fee002f8  Data: 0000
	Kernel driver in use: xhci_hcd

00:14.2 Signal processing controller: Intel Corporation Device a1b1 (rev 08)
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin C routed to IRQ 11
	Region 0: Memory at 383ffff33000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000

00:16.0 Communication controller: Intel Corporation Lewisburg CSME: HECI #1 (rev 08)
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at 383ffff32000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000

00:16.1 Communication controller: Intel Corporation Lewisburg CSME: HECI #2 (rev 08)
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at 383ffff31000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000

00:16.4 Communication controller: Intel Corporation Lewisburg CSME: HECI #3 (rev 08)
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at 383ffff30000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000

00:17.0 SATA controller: Intel Corporation Lewisburg SATA Controller [AHCI mode] (rev 08) (prog-if 01 [AHCI 1.0])
	Subsystem: Intel Corporation Device 7270
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 48
	Region 0: Memory at 9d204000 (32-bit, non-prefetchable) [size=8K]
	Region 1: Memory at 9d208000 (32-bit, non-prefetchable) [size=256]
	Region 2: I/O ports at 3050 [size=8]
	Region 3: I/O ports at 3040 [size=4]
	Region 4: I/O ports at 3000 [size=32]
	Region 5: Memory at 9d100000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00338  Data: 0000
	Capabilities: [70] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
	Kernel driver in use: ahci

00:1c.0 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #1 (rev f8) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 00001000-00001fff
	Memory behind bridge: 90000000-901fffff
	Prefetchable memory behind bridge: 0000380000000000-00003800001fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #1, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt- HPIrq+ LinkChg+
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee002b8  Data: 0000
	Capabilities: [90] Subsystem: Intel Corporation Device 7270
	Capabilities: [a0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: pcieport

00:1c.4 PCI bridge: Intel Corporation Lewisburg PCI Express Root Port #5 (rev f8) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=02, subordinate=03, sec-latency=0
	I/O behind bridge: 00002000-00002fff
	Memory behind bridge: 9c000000-9d0fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #5, Speed 8GT/s, Width x1, ASPM not supported, Exit Latency L0s <1us, L1 <16us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 10.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee002d8  Data: 0000
	Capabilities: [90] Subsystem: Intel Corporation Device 7270
	Capabilities: [a0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [220 v1] #19
	Kernel driver in use: pcieport

00:1f.0 ISA bridge: Intel Corporation Lewisburg LPC Controller (rev 08)
	Subsystem: Intel Corporation Device 7270
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

00:1f.2 Memory controller: Intel Corporation Lewisburg PMC (rev 08)
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Region 0: Memory at 9d200000 (32-bit, non-prefetchable) [size=16K]

00:1f.4 SMBus: Intel Corporation Lewisburg SMBus (rev 08)
	Subsystem: Intel Corporation Device 7270
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at 380000200000 (64-bit, non-prefetchable) [size=256]
	Region 4: I/O ports at 0780 [size=32]
	Kernel driver in use: i801_smbus

00:1f.5 Serial bus controller [0c80]: Intel Corporation Lewisburg SPI Controller (rev 08)
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Region 0: Memory at fe010000 (32-bit, non-prefetchable) [size=4K]

02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=02, secondary=03, subordinate=03, sec-latency=32
	I/O behind bridge: 00002000-00002fff
	Memory behind bridge: 9c000000-9d0fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] Express (v2) PCI-Express to PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <2us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [c0] Subsystem: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [800 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41) (prog-if 00 [VGA controller])
	Subsystem: ASPEED Technology, Inc. ASPEED Graphics Family
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at 9c000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at 9d000000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at 2000 [size=128]
	Expansion ROM at <unassigned> [disabled]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Kernel driver in use: ast

17:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 0
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=17, secondary=18, subordinate=19, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: aad00000-aaefffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00038  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

17:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 1
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=17, secondary=1a, subordinate=1a, sec-latency=0
	I/O behind bridge: 00004000-00004fff
	Memory behind bridge: 9d800000-9d9fffff
	Prefetchable memory behind bridge: 0000384000000000-00003840001fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00058  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #3, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #1, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

17:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

17:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

17:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Device 2036
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at aaf00000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

17:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

17:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

18:00.0 PCI bridge: Intel Corporation Device 37c0 (rev 08) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at aae00000 (64-bit, non-prefetchable) [size=128K]
	Bus: primary=18, secondary=19, subordinate=19, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	Expansion ROM at aad00000 [disabled] [size=1M]
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Express (v2) Upstream Port, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 256 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <1us, L1 <32us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [88] Subsystem: Device beef:dead
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [200 v1] #19
	Kernel driver in use: pcieport

3a:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=3a, secondary=3b, subordinate=3b, sec-latency=0
	I/O behind bridge: 00006000-00006fff
	Memory behind bridge: b8500000-b86fffff
	Prefetchable memory behind bridge: 000038bffa000000-000038bffeffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00038  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #5, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #2, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

3a:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 3
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=3a, secondary=3c, subordinate=3c, sec-latency=0
	I/O behind bridge: 00007000-00007fff
	Memory behind bridge: ab000000-ab1fffff
	Prefetchable memory behind bridge: 0000388000000000-00003880001fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00058  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #7, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #3, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

3a:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

3a:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

3a:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Device 2036
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at b8700000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

3a:08.0 System peripheral: Intel Corporation Device 2066 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:09.0 System peripheral: Intel Corporation Device 2066 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.0 System peripheral: Intel Corporation Device 2040 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.1 System peripheral: Intel Corporation Device 2041 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.2 System peripheral: Intel Corporation Device 2042 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.3 System peripheral: Intel Corporation Device 2043 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.4 System peripheral: Intel Corporation Device 2044 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.5 System peripheral: Intel Corporation Device 2045 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.6 System peripheral: Intel Corporation Device 2046 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0a.7 System peripheral: Intel Corporation Device 2047 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0b.0 System peripheral: Intel Corporation Device 2048 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0b.1 System peripheral: Intel Corporation Device 2049 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0b.2 System peripheral: Intel Corporation Device 204a (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0b.3 System peripheral: Intel Corporation Device 204b (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.0 System peripheral: Intel Corporation Device 2040 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.1 System peripheral: Intel Corporation Device 2041 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.2 System peripheral: Intel Corporation Device 2042 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.3 System peripheral: Intel Corporation Device 2043 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.4 System peripheral: Intel Corporation Device 2044 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.5 System peripheral: Intel Corporation Device 2045 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.6 System peripheral: Intel Corporation Device 2046 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0c.7 System peripheral: Intel Corporation Device 2047 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0d.0 System peripheral: Intel Corporation Device 2048 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0d.1 System peripheral: Intel Corporation Device 2049 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0d.2 System peripheral: Intel Corporation Device 204a (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3a:0d.3 System peripheral: Intel Corporation Device 204b (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

3b:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Subsystem: Mellanox Technologies Device 0021
	Physical Slot: 2-1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 51
	Region 0: Memory at 38bffc000000 (64-bit, prefetchable) [size=32M]
	Expansion ROM at b8600000 [disabled] [size=1M]
	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 256 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [48] Vital Product Data
		Product Name: CX4421A- ConnectX-4 LX SFP28
		Read-only fields:
			[PN] Part number: MCX4421A-ACQN        
			[EC] Engineering changes: A6
			[SN] Serial number: MT1719X02671            
			[V0] Vendor specific: PCIeGen3 x8     
			[RV] Reserved: checksum good, 0 byte(s) reserved
		End
	Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [c0] Vendor Specific Information: Len=18 <?>
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 2, stride: 1, Device ID: 1016
		Supported Page Size: 000007ff, System Page Size: 00000001
		Region 0: Memory at 000038bffe800000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [1c0 v1] #19
	Capabilities: [230 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Kernel driver in use: mlx5_core

3b:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Subsystem: Mellanox Technologies Device 0021
	Physical Slot: 2-1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 168
	Region 0: Memory at 38bffa000000 (64-bit, prefetchable) [size=32M]
	Expansion ROM at b8500000 [disabled] [size=1M]
	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 256 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [48] Vital Product Data
		Product Name: CX4421A- ConnectX-4 LX SFP28
		Read-only fields:
			[PN] Part number: MCX4421A-ACQN        
			[EC] Engineering changes: A6
			[SN] Serial number: MT1719X02671            
			[V0] Vendor specific: PCIeGen3 x8     
			[RV] Reserved: checksum good, 0 byte(s) reserved
		End
	Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [c0] Vendor Specific Information: Len=18 <?>
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 9, stride: 1, Device ID: 1016
		Supported Page Size: 000007ff, System Page Size: 00000001
		Region 0: Memory at 000038bffe000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [230 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Kernel driver in use: mlx5_core

5d:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 4
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=5d, secondary=5e, subordinate=5e, sec-latency=0
	I/O behind bridge: 00008000-00008fff
	Memory behind bridge: b8800000-b89fffff
	Prefetchable memory behind bridge: 000038c000000000-000038c0001fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00038  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #9, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #4, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

5d:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 6
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=5d, secondary=5f, subordinate=5f, sec-latency=0
	I/O behind bridge: 00009000-00009fff
	Memory behind bridge: b8a00000-b8bfffff
	Prefetchable memory behind bridge: 000038c000200000-000038c0003fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00058  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #11, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #6, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

5d:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Device 2036
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at c5f00000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

5d:0e.0 Performance counters: Intel Corporation Device 2058 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:0e.1 System peripheral: Intel Corporation Device 2059 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

5d:0f.0 Performance counters: Intel Corporation Device 2058 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:0f.1 System peripheral: Intel Corporation Device 2059 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

5d:10.0 Performance counters: Intel Corporation Device 2058 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:10.1 System peripheral: Intel Corporation Device 2059 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

5d:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

5d:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

5d:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

80:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at 393ffff1c000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 10
	Region 0: Memory at 393ffff18000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 11
	Region 0: Memory at 393ffff14000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 11
	Region 0: Memory at 393ffff10000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at 393ffff0c000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 10
	Region 0: Memory at 393ffff08000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 11
	Region 0: Memory at 393ffff04000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 11
	Region 0: Memory at 393ffff00000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [90] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

80:05.2 System peripheral: Intel Corporation Device 2025 (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

80:05.4 PIC: Intel Corporation Device 2026 (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Device 2026
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at d3700000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

80:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

80:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

80:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

85:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 8
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=85, secondary=86, subordinate=86, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: d3800000-d39fffff
	Prefetchable memory behind bridge: 0000394000000000-00003940001fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00038  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #8, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

85:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

85:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

85:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Device 2036
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at e0f00000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

85:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

85:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ae:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 12
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=ae, secondary=af, subordinate=af, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: ee600000-ee6fffff
	Prefetchable memory behind bridge: 0000398000000000-00003980001fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00038  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #5, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #12, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

ae:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1B (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 13
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=ae, secondary=b0, subordinate=b0, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: ee500000-ee5fffff
	Prefetchable memory behind bridge: 0000398000200000-00003980003fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00058  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #6, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #13, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

ae:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 14
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=ae, secondary=b1, subordinate=b1, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: e1000000-e11fffff
	Prefetchable memory behind bridge: 0000398000400000-00003980005fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00078  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #7, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #14, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

ae:03.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1D (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 15
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=ae, secondary=b2, subordinate=b2, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: e1200000-e13fffff
	Prefetchable memory behind bridge: 0000398000600000-00003980007fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00098  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #8, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #15, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

ae:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

ae:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

ae:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Device 2036
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at ee700000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

ae:08.0 System peripheral: Intel Corporation Device 2066 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:09.0 System peripheral: Intel Corporation Device 2066 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.0 System peripheral: Intel Corporation Device 2040 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.1 System peripheral: Intel Corporation Device 2041 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.2 System peripheral: Intel Corporation Device 2042 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.3 System peripheral: Intel Corporation Device 2043 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.4 System peripheral: Intel Corporation Device 2044 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.5 System peripheral: Intel Corporation Device 2045 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.6 System peripheral: Intel Corporation Device 2046 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0a.7 System peripheral: Intel Corporation Device 2047 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0b.0 System peripheral: Intel Corporation Device 2048 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0b.1 System peripheral: Intel Corporation Device 2049 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0b.2 System peripheral: Intel Corporation Device 204a (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0b.3 System peripheral: Intel Corporation Device 204b (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.0 System peripheral: Intel Corporation Device 2040 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.1 System peripheral: Intel Corporation Device 2041 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.2 System peripheral: Intel Corporation Device 2042 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.3 System peripheral: Intel Corporation Device 2043 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.4 System peripheral: Intel Corporation Device 2044 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.5 System peripheral: Intel Corporation Device 2045 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.6 System peripheral: Intel Corporation Device 2046 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0c.7 System peripheral: Intel Corporation Device 2047 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0d.0 System peripheral: Intel Corporation Device 2048 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0d.1 System peripheral: Intel Corporation Device 2049 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0d.2 System peripheral: Intel Corporation Device 204a (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

ae:0d.3 System peripheral: Intel Corporation Device 204b (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

af:00.0 Non-Volatile memory controller: Device 1c5f:0550 (rev 02) (prog-if 02 [NVM Express])
	Subsystem: Device 1c5f:0552
	Physical Slot: 12-1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 49
	Region 0: Memory at ee610000 (64-bit, non-prefetchable) [size=16K]
	Expansion ROM at ee600000 [disabled] [size=64K]
	Capabilities: [c0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000001
	Capabilities: [e0] MSI-X: Enable+ Count=129 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 256 bytes, MaxReadReq 256 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [40] Vendor Specific Information: Len=24 <?>
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [190 v1] #19
	Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
	Kernel driver in use: nvme

b0:00.0 Non-Volatile memory controller: Device 1c5f:0550 (rev 02) (prog-if 02 [NVM Express])
	Subsystem: Device 1c5f:0552
	Physical Slot: 13-1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 50
	Region 0: Memory at ee510000 (64-bit, non-prefetchable) [size=16K]
	Expansion ROM at ee500000 [disabled] [size=64K]
	Capabilities: [c0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000001
	Capabilities: [e0] MSI-X: Enable+ Count=129 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 256 bytes, MaxReadReq 256 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [40] Vendor Specific Information: Len=24 <?>
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [190 v1] #19
	Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
	Kernel driver in use: nvme

d7:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 16
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=d7, secondary=d8, subordinate=d8, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: ee800000-ee9fffff
	Prefetchable memory behind bridge: 000039c000000000-000039c0001fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00038  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #9, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #16, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

d7:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 18
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=d7, secondary=d9, subordinate=d9, sec-latency=0
	I/O behind bridge: 0000f000-0000ffff
	Memory behind bridge: eea00000-eebfffff
	Prefetchable memory behind bridge: 000039c000200000-000039c0003fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00058  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #11, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #18, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

d7:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 04)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:05.4 PIC: Intel Corporation Device 2036 (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Intel Corporation Device 2036
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at fbf00000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [44] Express (v1) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

d7:0e.0 Performance counters: Intel Corporation Device 2058 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:0e.1 System peripheral: Intel Corporation Device 2059 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

d7:0f.0 Performance counters: Intel Corporation Device 2058 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:0f.1 System peripheral: Intel Corporation Device 2059 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

d7:10.0 Performance counters: Intel Corporation Device 2058 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:10.1 System peripheral: Intel Corporation Device 2059 (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [100 v0] Vendor Specific Information: ID=0001 Rev=0 Len=0f0 <?>

d7:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

d7:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

d7:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 04)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Bjorn Helgaas]

On Tue, Jan 23, 2018 at 01:25:56PM +0000, Ron Yuan wrote:

I'm reproducing Sinan's picture here so we can see what you're talking
about:

> >>>>                root (MPS=256)
> >>>>                  |
> >>>>          ------------------
> >>>>         /                  \
> >>>>    bridge0 (MPS=256)      bridge1 (MPS=128)
> >>>>       /                       \
> >>>>     EP0 (MPS=256)            EP1 (MPS=128)
> >>>>

> > PERFORMANCE mode reduces MRRS not because of a starvation issue,
> > but because reducing EP1's MRRS allows EP0 to use a larger MPS.

> Looks like this case is talking about EP1 requests data directly
> from EP0, using MRRS to control the return data payload, while still
> keeping the traffic from EP0 to RC in 256B. 

No, this is not talking about EP1 requesting data from EP0.  That
would be peer-to-peer DMA, and PERFORMANCE mode explicitly assumes
there is no peer-to-peer DMA.  It reduces MRRS to allow EP0 to use a
larger MPS.

We must guarantee that no device receives a TLP larger than its MPS
setting.  The simple and obvious configuration is to set MPS=128 for
everything in Sinan's picture.  That works correctly but limits EP0's
performance.

What PERFORMANCE mode does is set MPS as shown in the picture and set
EP1's MRRS=128.  We're assuming no peer-to-peer DMA, but of course EP1
may still need to do DMA reads from system memory, and setting its
MRRS=128 means those reads will be of 128 bytes or less.

If we set EP1's MRRS=256, it could do a 256-byte DMA read from system
memory, the root port could send a 256-byte completion, and bridge1
would treat that as a malformed TLP because its MPS=128.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Bjorn Helgaas]

On Tue, Jan 23, 2018 at 02:01:27PM +0000, Ron Yuan wrote:
> Just got the log, see attachment. Kernel is under "perf" mode.
> SSD and Ethernet controller are both set to 256B MRRS. 

Here's what I see from lspci:

  3a:00.0 Root Port to 3b     MPS_cap=256 MPS=256 MRRS=128
  3b:00.0 NIC Endpoint        MPS_cap=512 MPS=256 MRRS=256
  3b:00.1 NIC Endpoint        MPS_cap=512 MPS=256 MRRS=256

Here, the NICs support up to MPS=512 but the Root Port only supports
MPS=256.  We must set MPS=256 for the NICs.  Otherwise, the NICs could
do 512-byte DMA writes to system memory, and the Root Port would treat
those TLPs as malformed.

There is no need to limit MRRS because there are no other devices
under this Root Port.  We can't tell from lspci what the maximum MRRS
is, but if the NICs support MRRS=4096, we could use that.

  ae:00.0 Root Port to af     MPS_cap=256 MPS=256 MRRS=128
  ae:01.0 Root Port to b0     MPS_cap=256 MPS=256 MRRS=128
  af:00.0 SSD Endpoint        MPS_cap=256 MPS=256 MRRS=256
  b0:00.0 SSD Endpoint        MPS_cap=256 MPS=256 MRRS=256

Everything here supports MPS=256, so that's what we should use.

As with the NICs, there's no need to limit MRRS here.  We don't know
what MRRS the endpoints support, but PERFORMANCE mode is being
unnecessarily conservative when it limits MRRS to the MPS (256 in this
case).

It's not as simple as just removing the "set MRRS=MPS" part because we
do rely on that in some topologies.  There are interesting topologies
where we *don't* need it, like both of the ones above, and we need to
make Linux smart enough to recognize them.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Bjorn Helgaas]

On Tue, Jan 23, 2018 at 11:48:21AM -0600, Bjorn Helgaas wrote:
> On Tue, Jan 23, 2018 at 02:01:27PM +0000, Ron Yuan wrote:
> > Just got the log, see attachment. Kernel is under "perf" mode.
> > SSD and Ethernet controller are both set to 256B MRRS. 
> 
> Here's what I see from lspci:
> 
>   3a:00.0 Root Port to 3b     MPS_cap=256 MPS=256 MRRS=128
>   3b:00.0 NIC Endpoint        MPS_cap=512 MPS=256 MRRS=256
>   3b:00.1 NIC Endpoint        MPS_cap=512 MPS=256 MRRS=256
> 
> Here, the NICs support up to MPS=512 but the Root Port only supports
> MPS=256.  We must set MPS=256 for the NICs.  Otherwise, the NICs could
> do 512-byte DMA writes to system memory, and the Root Port would treat
> those TLPs as malformed.
> 
> There is no need to limit MRRS because there are no other devices
> under this Root Port.  We can't tell from lspci what the maximum MRRS
> is, but if the NICs support MRRS=4096, we could use that.
> 
>   ae:00.0 Root Port to af     MPS_cap=256 MPS=256 MRRS=128
>   ae:01.0 Root Port to b0     MPS_cap=256 MPS=256 MRRS=128
>   af:00.0 SSD Endpoint        MPS_cap=256 MPS=256 MRRS=256
>   b0:00.0 SSD Endpoint        MPS_cap=256 MPS=256 MRRS=256
> 
> Everything here supports MPS=256, so that's what we should use.
> 
> As with the NICs, there's no need to limit MRRS here.  We don't know
> what MRRS the endpoints support, but PERFORMANCE mode is being
> unnecessarily conservative when it limits MRRS to the MPS (256 in this
> case).
> 
> It's not as simple as just removing the "set MRRS=MPS" part because we
> do rely on that in some topologies.  There are interesting topologies
> where we *don't* need it, like both of the ones above, and we need to
> make Linux smart enough to recognize them.

The mailing list dropped Ron's email with the lspci log because it was
a multipart base64-encoded message.  The lists accept plain text only
(see http://vger.kernel.org/majordomo-info.html).  

Here's Ron's message:

> Just got the log, see attachment. Kernel is under "perf" mode.
> SSD and Ethernet controller are both set to 256B MRRS.

And here are the relevant parts of the lspci.log that he attached (I
didn't include it all because it's over 100K and the mailing list
wouldn't accept that either):

3a:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=3a, secondary=3b, subordinate=3b, sec-latency=0
	I/O behind bridge: 00006000-00006fff
	Memory behind bridge: b8500000-b86fffff
	Prefetchable memory behind bridge: 000038bffa000000-000038bffeffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00038  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #5, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
			Slot #2, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

3b:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Subsystem: Mellanox Technologies Device 0021
	Physical Slot: 2-1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 51
	Region 0: Memory at 38bffc000000 (64-bit, prefetchable) [size=32M]
	Expansion ROM at b8600000 [disabled] [size=1M]
	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 256 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [48] Vital Product Data
		Product Name: CX4421A- ConnectX-4 LX SFP28
		Read-only fields:
			[PN] Part number: MCX4421A-ACQN        
			[EC] Engineering changes: A6
			[SN] Serial number: MT1719X02671            
			[V0] Vendor specific: PCIeGen3 x8     
			[RV] Reserved: checksum good, 0 byte(s) reserved
		End
	Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [c0] Vendor Specific Information: Len=18 <?>
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 2, stride: 1, Device ID: 1016
		Supported Page Size: 000007ff, System Page Size: 00000001
		Region 0: Memory at 000038bffe800000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [1c0 v1] #19
	Capabilities: [230 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Kernel driver in use: mlx5_core

3b:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Subsystem: Mellanox Technologies Device 0021
	Physical Slot: 2-1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 168
	Region 0: Memory at 38bffa000000 (64-bit, prefetchable) [size=32M]
	Expansion ROM at b8500000 [disabled] [size=1M]
	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 256 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [48] Vital Product Data
		Product Name: CX4421A- ConnectX-4 LX SFP28
		Read-only fields:
			[PN] Part number: MCX4421A-ACQN        
			[EC] Engineering changes: A6
			[SN] Serial number: MT1719X02671            
			[V0] Vendor specific: PCIeGen3 x8     
			[RV] Reserved: checksum good, 0 byte(s) reserved
		End
	Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [c0] Vendor Specific Information: Len=18 <?>
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 9, stride: 1, Device ID: 1016
		Supported Page Size: 000007ff, System Page Size: 00000001
		Region 0: Memory at 000038bffe000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [230 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Kernel driver in use: mlx5_core

ae:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 12
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=ae, secondary=af, subordinate=af, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: ee600000-ee6fffff
	Prefetchable memory behind bridge: 0000398000000000-00003980001fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00038  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #5, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #12, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

ae:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1B (rev 04) (prog-if 00 [Normal decode])
	Physical Slot: 13
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=ae, secondary=b0, subordinate=b0, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: ee500000-ee5fffff
	Prefetchable memory behind bridge: 0000398000200000-00003980003fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
		Address: fee00058  Data: 0000
		Masking: 00000002  Pending: 00000000
	Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #6, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <512ns, L1 <16us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl+ MRL- AttnInd+ PwrInd+ HotPlug+ Surprise+
			Slot #13, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
			Control: AttnInd Off, PwrInd Off, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal+ ErrFatal+ PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
		DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [e0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
	Capabilities: [110 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [148 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [1d0 v1] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
	Capabilities: [250 v1] #19
	Capabilities: [280 v1] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
	Capabilities: [300 v1] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
	Kernel driver in use: pcieport

af:00.0 Non-Volatile memory controller: Device 1c5f:0550 (rev 02) (prog-if 02 [NVM Express])
	Subsystem: Device 1c5f:0552
	Physical Slot: 12-1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 49
	Region 0: Memory at ee610000 (64-bit, non-prefetchable) [size=16K]
	Expansion ROM at ee600000 [disabled] [size=64K]
	Capabilities: [c0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000001
	Capabilities: [e0] MSI-X: Enable+ Count=129 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 256 bytes, MaxReadReq 256 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [40] Vendor Specific Information: Len=24 <?>
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [190 v1] #19
	Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
	Kernel driver in use: nvme

b0:00.0 Non-Volatile memory controller: Device 1c5f:0550 (rev 02) (prog-if 02 [NVM Express])
	Subsystem: Device 1c5f:0552
	Physical Slot: 13-1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 50
	Region 0: Memory at ee510000 (64-bit, non-prefetchable) [size=16K]
	Expansion ROM at ee500000 [disabled] [size=64K]
	Capabilities: [c0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000001
	Capabilities: [e0] MSI-X: Enable+ Count=129 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 256 bytes, MaxReadReq 256 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [40] Vendor Specific Information: Len=24 <?>
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [190 v1] #19
	Capabilities: [150 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
	Kernel driver in use: nvme

RE: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Radjendirane Codandaramane]

Hi Bjorne,

Ceiling the MRRS to the MPS value in order to guarantee the interoperabilit=
y in pcie_bus_perf mode does not make sense. A device can make a memrd requ=
est according to the MRRS setting (which can be higher than its MPS), but t=
he completer has to respect the MPS and send completions accordingly. As an=
 example, system can configure MPS=3D128B and MRRS=3D4K, where an endpoint =
can a make 4K MemRd request, but the completer has to send completions as 1=
28B TLPs, by respecting the MPS setting. MRRS does not force a device to us=
e higher MPS value than it is configured to.

Another factor that need to be considered for storage devices is that suppo=
rt of T10 Protection Information (DIF). For every 512B or 4KB, a 8B PI is c=
omputed and inserted or verified, which require the 512B of data to arrive =
in sequence. If the MRRS is < 512B, this might pose out of order completion=
s to the storage device, if the EP has to submit multiple outstanding read =
requests in order to achieve higher performance. This would be a challenge =
for the storage endpoints that process the T10 PI inline with the transfer,=
 now they have to store and process the 512B sectors once they receive all =
the TLPs for that sector.

So, it is better to decouple the MRRS and MPS in pcie_bus_perf mode. Like s=
tated earlier in the thread, provide an option to configure MRRS separately=
 in pcie_bus_perf mode.

Regards,
Radj.

-----Original Message-----
From: Bjorn Helgaas [mailto:helgaas@kernel.org]=20
Sent: Tuesday, January 23, 2018 6:39 AM
To: Ron Yuan <ron.yuan@memblaze.com>
Cc: Sinan Kaya <okaya@codeaurora.org>; Bjorn Helgaas <bhelgaas@google.com>;=
 Bo Chen <bo.chen@memblaze.com>; William Huang <william.huang@memblaze.com>=
; Fengming Wu <fengming.wu@memblaze.com>; Jason Jiang <jason.jiang@microsem=
i.com>; Radjendirane Codandaramane <radjendirane.codanda@microsemi.com>; Ra=
myakanth Edupuganti <Ramyakanth.Edupuganti@microsemi.com>; William Cheng <w=
illiam.cheng@microsemi.com>; Kim Helper (khelper) <khelper@micron.com>; Lin=
ux PCI <linux-pci@vger.kernel.org>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or =
pcie_bus_perf On NVMe Device

EXTERNAL EMAIL


On Tue, Jan 23, 2018 at 01:25:56PM +0000, Ron Yuan wrote:

I'm reproducing Sinan's picture here so we can see what you're talking
about:

> >>>>                root (MPS=3D256)
> >>>>                  |
> >>>>          ------------------
> >>>>         /                  \
> >>>>    bridge0 (MPS=3D256)      bridge1 (MPS=3D128)
> >>>>       /                       \
> >>>>     EP0 (MPS=3D256)            EP1 (MPS=3D128)
> >>>>

> > PERFORMANCE mode reduces MRRS not because of a starvation issue, but=20
> > because reducing EP1's MRRS allows EP0 to use a larger MPS.

> Looks like this case is talking about EP1 requests data directly from=20
> EP0, using MRRS to control the return data payload, while still=20
> keeping the traffic from EP0 to RC in 256B.

No, this is not talking about EP1 requesting data from EP0.  That would be =
peer-to-peer DMA, and PERFORMANCE mode explicitly assumes there is no peer-=
to-peer DMA.  It reduces MRRS to allow EP0 to use a larger MPS.

We must guarantee that no device receives a TLP larger than its MPS setting=
.  The simple and obvious configuration is to set MPS=3D128 for everything =
in Sinan's picture.  That works correctly but limits EP0's performance.

What PERFORMANCE mode does is set MPS as shown in the picture and set EP1's=
 MRRS=3D128.  We're assuming no peer-to-peer DMA, but of course EP1 may sti=
ll need to do DMA reads from system memory, and setting its
MRRS=3D128 means those reads will be of 128 bytes or less.

If we set EP1's MRRS=3D256, it could do a 256-byte DMA read from system mem=
ory, the root port could send a 256-byte completion, and bridge1 would trea=
t that as a malformed TLP because its MPS=3D128.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Myron Stowe]

Radjendirane,

I've struggled with a response to your latest posting all
night.  I don't want to come off as offensive or terse as happens all too
often on linux mail lists, all that does is shut things down unnecessarly
without relaying any information that is being sought.

Bjorn is just one person trying to keep up with this entire list.  On this
particular topic he has taken considerable time explaining, and answering
peoples specific questions, as to how Linux currently handles PCIe MPS and
MRRS settings.  As such, it was quite surprising to see your
latest posting as the majority of the content had already
been covered - the exact same questions/points - in great
detail.

We should respect Bjorn's time as much as possible and "do our
homework"; in this specific case, take the time to read the entire thread,
carefully, as that would have circumvented this awkward and frustrating
situation.  If there are still questions on covered topics, then ask the
questions at that point (i.e. use proper response techniques to preserve
context for all; don't just repeat a question much later in the thread that
has already been discussed).

Again, I'm not trying to shut you down.


As Sinan pointed out in the thread's inception:
  "Please use mailing list email syntax moving forward. (inline and 75
  characters per line)".

On Tue, Jan 23, 2018 at 4:50 PM, Radjendirane Codandaramane
<radjendirane.codanda@microsemi.com> wrote:
> Hi Bjorne,

Bjorn

>
> Ceiling the MRRS to the MPS value in order to guarantee the interoperabil=
ity in pcie_bus_perf mode does not make sense. A device can make a memrd re=
quest according to the MRRS setting (which can be higher than its MPS), but=
 the completer has to respect the MPS and send completions accordingly. As =
an example, system can configure MPS=3D128B and MRRS=3D4K, where an endpoin=
t can a make 4K MemRd request, but the completer has to send completions as=
 128B TLPs, by respecting the MPS setting. MRRS does not force a device to =
use higher MPS value than it is configured to.

This was covered by the very first topic in Bjorn's first reply within the
thread...

>
> Another factor that need to be considered for storage devices is that sup=
port of T10 Protection Information (DIF). For every 512B or 4KB, a 8B PI is=
 computed and inserted or verified, which require the 512B of data to arriv=
e in sequence. If the MRRS is < 512B, this might pose out of order completi=
ons to the storage device, if the EP has to submit multiple outstanding rea=
d requests in order to achieve higher performance. This would be a challeng=
e for the storage endpoints that process the T10 PI inline with the transfe=
r, now they have to store and process the 512B sectors once they receive al=
l the TLPs for that sector.

The "T10" aspects are new and I've not heard about them before.  On the
surface they seem to be storage device specific.  If that is indeed the
case then there seems to be some mixing of two, distinctly different,
things.  PCIe TLPs, MPS, MRRS, ..., are all PCIe defined items that are
covered by its specification.  Expecting that T10 specifics can be
intermixed within PCIe's protocol doesn't make any sense and sounds much
more like something that will have to be taken care of at the controller's
level.   Perhaps I'm  way off base here, we'll have to hear more about this
to come to some understanding.

>
> So, it is better to decouple the MRRS and MPS in pcie_bus_perf mode. Like=
 stated earlier in the thread, provide an option to configure MRRS separate=
ly in pcie_bus_perf mode.

This also has been brought up twice already, and covered in the prior
responses...

>
> Regards,
> Radj.

RE: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Ron Yuan]

Qmpvcm4sIA0KDQpJIGdvIGJhY2sgcmVhZGluZyB5b3VyIGVhcmxpZXIgcmVwbHkgY2FyZWZ1bGx5
IGFuZCBmaW5hbGx5IHVuZGVyc3RhbmQgdGhhdCB5b3VyIGV4YW1wbGUgdGhlIFJQIG1heSByZXBs
eSAyNTZCIFRMUCBiZWNhdXNlIG9mIGl0cyBvd24gbGFyZ2VyIE1QUyBzZXR0aW5nLiBUaGlzIGlz
IGVhc2lseSBvdmVybG9va2VkIHNpbmNlIHVzdWFsbHkgTVBTIGlzIHN5bmMgb24gdGhlIHdob2xl
IHBhdGggYXQgdGhlIGJlZ2lubmluZywgSSBhbG1vc3QgdGFrZSBpdCBmb3IgZ3JhbnRlZC4gVGhh
bmsgeW91ISAgDQoNCkhlcmUgaXMgbXkgdW5kZXJzdGFuZGluZzogDQpJZiBNUFMgaXMgc2FtZSBv
biBhIHBhdGgsIE1SUlMgaXMgbm90IG5lY2Vzc2FyeSB0byBiZSBwYXJ0IG9mIE1QUyBjb250cm9s
LiAtPiB3aGljaCBoZWxwIG1lIGluIE5WTWUgY2FzZSwgTVBTPTI1NkIgYXJlIHN1cHBvcnRlZCBm
b3IgYm90aCBSUCBhbmQgRVAuDQpPbmx5IGlmIE1QUyBpcyBub3QgdGhlIHNhbWUgb24gYSBwYXRo
IChpbiB0aGUgZXhhbXBsZSBSUCBhbmQgRVAxKSwgTVJSUyBzaG91bGQgYmUgc2V0IHRvIHNhbWUg
YXMgTVBTIHRvIGF2b2lkIHJlY2VpdmluZyBsYXJnZXIgY29tcGxldGlvbiBUTFAgdGhhbiBpdHMg
TVBTLiBQZXJoYXBzIHRoZSBjb2RlIGNhbiBjaGVjayBhbGwgdXBzdHJlYW0gZGV2aWNlcycgTVBT
IGFuZCBkZXRlcm1pbmUgaWYgaXQgbmVlZHMgdG8gYXBwbHkgYSBNUlJTIGxpbWl0YXRpb24uIA0K
DQpCdHcsIEkgYWxzbyBmb3VuZCBmb2xsb3dpbmcgc3RhdGVtZW50IGZyb20gUENJRSBzcGVjLCB3
aGljaCBtaWdodCBiZSB0aGUgb3JpZ2luYWwgcmVhc29uIG9mIHRoZSBjb2RlIGF0IHRoZSBiZWdp
bm5pbmcuIA0KUGFnZSA1MDAsIFNlY3Rpb24gNi4zLjQuMSBSdWxlcyBmb3IgU29mdHdhcmUgQ29u
ZmlndXJhdGlvbg0KIlNvZnR3YXJlIG11c3Qgc2V0IE1heF9SZWFkX1JlcXVlc3RfU2l6ZSBvZiBh
biBpc29jaHJvbm91cy1jb25maWd1cmVkIGRldmljZSB3aXRoIGEgdmFsdWUgdGhhdCBkb2VzIG5v
dCBleGNlZWQgdGhlIE1heF9QYXlsb2FkX1NpemUgc2V0IGZvciB0aGUgZGV2aWNlLiINCg0KDQpS
b24NCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IE15cm9uIFN0b3dlIFttYWls
dG86bXlyb24uc3Rvd2VAZ21haWwuY29tXSANClNlbnQ6IDIwMTjlubQx5pyIMjXml6UgMDozMA0K
VG86IFJhZGplbmRpcmFuZSBDb2RhbmRhcmFtYW5lIDxyYWRqZW5kaXJhbmUuY29kYW5kYUBtaWNy
b3NlbWkuY29tPg0KQ2M6IEJqb3JuIEhlbGdhYXMgPGhlbGdhYXNAa2VybmVsLm9yZz47IFJvbiBZ
dWFuIDxyb24ueXVhbkBtZW1ibGF6ZS5jb20+OyBTaW5hbiBLYXlhIDxva2F5YUBjb2RlYXVyb3Jh
Lm9yZz47IEJqb3JuIEhlbGdhYXMgPGJoZWxnYWFzQGdvb2dsZS5jb20+OyBCbyBDaGVuIDxiby5j
aGVuQG1lbWJsYXplLmNvbT47IFdpbGxpYW0gSHVhbmcgPHdpbGxpYW0uaHVhbmdAbWVtYmxhemUu
Y29tPjsgRmVuZ21pbmcgV3UgPGZlbmdtaW5nLnd1QG1lbWJsYXplLmNvbT47IEphc29uIEppYW5n
IDxqYXNvbi5qaWFuZ0BtaWNyb3NlbWkuY29tPjsgUmFteWFrYW50aCBFZHVwdWdhbnRpIDxSYW15
YWthbnRoLkVkdXB1Z2FudGlAbWljcm9zZW1pLmNvbT47IFdpbGxpYW0gQ2hlbmcgPHdpbGxpYW0u
Y2hlbmdAbWljcm9zZW1pLmNvbT47IEtpbSBIZWxwZXIgKGtoZWxwZXIpIDxraGVscGVyQG1pY3Jv
bi5jb20+OyBMaW51eCBQQ0kgPGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc+DQpTdWJqZWN0OiBS
ZTogT25lIFF1ZXN0aW9uIEFib3V0IFBDSWUgQlVTIENvbmZpZyBUeXBlIHdpdGggcGNpZV9idXNf
c2FmZSBvciBwY2llX2J1c19wZXJmIE9uIE5WTWUgRGV2aWNlDQoNClJhZGplbmRpcmFuZSwNCg0K
SSd2ZSBzdHJ1Z2dsZWQgd2l0aCBhIHJlc3BvbnNlIHRvIHlvdXIgbGF0ZXN0IHBvc3RpbmcgYWxs
IG5pZ2h0LiAgSSBkb24ndCB3YW50IHRvIGNvbWUgb2ZmIGFzIG9mZmVuc2l2ZSBvciB0ZXJzZSBh
cyBoYXBwZW5zIGFsbCB0b28gb2Z0ZW4gb24gbGludXggbWFpbCBsaXN0cywgYWxsIHRoYXQgZG9l
cyBpcyBzaHV0IHRoaW5ncyBkb3duIHVubmVjZXNzYXJseSB3aXRob3V0IHJlbGF5aW5nIGFueSBp
bmZvcm1hdGlvbiB0aGF0IGlzIGJlaW5nIHNvdWdodC4NCg0KQmpvcm4gaXMganVzdCBvbmUgcGVy
c29uIHRyeWluZyB0byBrZWVwIHVwIHdpdGggdGhpcyBlbnRpcmUgbGlzdC4gIE9uIHRoaXMgcGFy
dGljdWxhciB0b3BpYyBoZSBoYXMgdGFrZW4gY29uc2lkZXJhYmxlIHRpbWUgZXhwbGFpbmluZywg
YW5kIGFuc3dlcmluZyBwZW9wbGVzIHNwZWNpZmljIHF1ZXN0aW9ucywgYXMgdG8gaG93IExpbnV4
IGN1cnJlbnRseSBoYW5kbGVzIFBDSWUgTVBTIGFuZCBNUlJTIHNldHRpbmdzLiAgQXMgc3VjaCwg
aXQgd2FzIHF1aXRlIHN1cnByaXNpbmcgdG8gc2VlIHlvdXIgbGF0ZXN0IHBvc3RpbmcgYXMgdGhl
IG1ham9yaXR5IG9mIHRoZSBjb250ZW50IGhhZCBhbHJlYWR5IGJlZW4gY292ZXJlZCAtIHRoZSBl
eGFjdCBzYW1lIHF1ZXN0aW9ucy9wb2ludHMgLSBpbiBncmVhdCBkZXRhaWwuDQoNCldlIHNob3Vs
ZCByZXNwZWN0IEJqb3JuJ3MgdGltZSBhcyBtdWNoIGFzIHBvc3NpYmxlIGFuZCAiZG8gb3VyIGhv
bWV3b3JrIjsgaW4gdGhpcyBzcGVjaWZpYyBjYXNlLCB0YWtlIHRoZSB0aW1lIHRvIHJlYWQgdGhl
IGVudGlyZSB0aHJlYWQsIGNhcmVmdWxseSwgYXMgdGhhdCB3b3VsZCBoYXZlIGNpcmN1bXZlbnRl
ZCB0aGlzIGF3a3dhcmQgYW5kIGZydXN0cmF0aW5nIHNpdHVhdGlvbi4gIElmIHRoZXJlIGFyZSBz
dGlsbCBxdWVzdGlvbnMgb24gY292ZXJlZCB0b3BpY3MsIHRoZW4gYXNrIHRoZSBxdWVzdGlvbnMg
YXQgdGhhdCBwb2ludCAoaS5lLiB1c2UgcHJvcGVyIHJlc3BvbnNlIHRlY2huaXF1ZXMgdG8gcHJl
c2VydmUgY29udGV4dCBmb3IgYWxsOyBkb24ndCBqdXN0IHJlcGVhdCBhIHF1ZXN0aW9uIG11Y2gg
bGF0ZXIgaW4gdGhlIHRocmVhZCB0aGF0IGhhcyBhbHJlYWR5IGJlZW4gZGlzY3Vzc2VkKS4NCg0K
QWdhaW4sIEknbSBub3QgdHJ5aW5nIHRvIHNodXQgeW91IGRvd24uDQoNCg0KQXMgU2luYW4gcG9p
bnRlZCBvdXQgaW4gdGhlIHRocmVhZCdzIGluY2VwdGlvbjoNCiAgIlBsZWFzZSB1c2UgbWFpbGlu
ZyBsaXN0IGVtYWlsIHN5bnRheCBtb3ZpbmcgZm9yd2FyZC4gKGlubGluZSBhbmQgNzUNCiAgY2hh
cmFjdGVycyBwZXIgbGluZSkiLg0KDQpPbiBUdWUsIEphbiAyMywgMjAxOCBhdCA0OjUwIFBNLCBS
YWRqZW5kaXJhbmUgQ29kYW5kYXJhbWFuZSA8cmFkamVuZGlyYW5lLmNvZGFuZGFAbWljcm9zZW1p
LmNvbT4gd3JvdGU6DQo+IEhpIEJqb3JuZSwNCg0KQmpvcm4NCg0KPg0KPiBDZWlsaW5nIHRoZSBN
UlJTIHRvIHRoZSBNUFMgdmFsdWUgaW4gb3JkZXIgdG8gZ3VhcmFudGVlIHRoZSBpbnRlcm9wZXJh
YmlsaXR5IGluIHBjaWVfYnVzX3BlcmYgbW9kZSBkb2VzIG5vdCBtYWtlIHNlbnNlLiBBIGRldmlj
ZSBjYW4gbWFrZSBhIG1lbXJkIHJlcXVlc3QgYWNjb3JkaW5nIHRvIHRoZSBNUlJTIHNldHRpbmcg
KHdoaWNoIGNhbiBiZSBoaWdoZXIgdGhhbiBpdHMgTVBTKSwgYnV0IHRoZSBjb21wbGV0ZXIgaGFz
IHRvIHJlc3BlY3QgdGhlIE1QUyBhbmQgc2VuZCBjb21wbGV0aW9ucyBhY2NvcmRpbmdseS4gQXMg
YW4gZXhhbXBsZSwgc3lzdGVtIGNhbiBjb25maWd1cmUgTVBTPTEyOEIgYW5kIE1SUlM9NEssIHdo
ZXJlIGFuIGVuZHBvaW50IGNhbiBhIG1ha2UgNEsgTWVtUmQgcmVxdWVzdCwgYnV0IHRoZSBjb21w
bGV0ZXIgaGFzIHRvIHNlbmQgY29tcGxldGlvbnMgYXMgMTI4QiBUTFBzLCBieSByZXNwZWN0aW5n
IHRoZSBNUFMgc2V0dGluZy4gTVJSUyBkb2VzIG5vdCBmb3JjZSBhIGRldmljZSB0byB1c2UgaGln
aGVyIE1QUyB2YWx1ZSB0aGFuIGl0IGlzIGNvbmZpZ3VyZWQgdG8uDQoNClRoaXMgd2FzIGNvdmVy
ZWQgYnkgdGhlIHZlcnkgZmlyc3QgdG9waWMgaW4gQmpvcm4ncyBmaXJzdCByZXBseSB3aXRoaW4g
dGhlIHRocmVhZC4uLg0KDQo+DQo+IEFub3RoZXIgZmFjdG9yIHRoYXQgbmVlZCB0byBiZSBjb25z
aWRlcmVkIGZvciBzdG9yYWdlIGRldmljZXMgaXMgdGhhdCBzdXBwb3J0IG9mIFQxMCBQcm90ZWN0
aW9uIEluZm9ybWF0aW9uIChESUYpLiBGb3IgZXZlcnkgNTEyQiBvciA0S0IsIGEgOEIgUEkgaXMg
Y29tcHV0ZWQgYW5kIGluc2VydGVkIG9yIHZlcmlmaWVkLCB3aGljaCByZXF1aXJlIHRoZSA1MTJC
IG9mIGRhdGEgdG8gYXJyaXZlIGluIHNlcXVlbmNlLiBJZiB0aGUgTVJSUyBpcyA8IDUxMkIsIHRo
aXMgbWlnaHQgcG9zZSBvdXQgb2Ygb3JkZXIgY29tcGxldGlvbnMgdG8gdGhlIHN0b3JhZ2UgZGV2
aWNlLCBpZiB0aGUgRVAgaGFzIHRvIHN1Ym1pdCBtdWx0aXBsZSBvdXRzdGFuZGluZyByZWFkIHJl
cXVlc3RzIGluIG9yZGVyIHRvIGFjaGlldmUgaGlnaGVyIHBlcmZvcm1hbmNlLiBUaGlzIHdvdWxk
IGJlIGEgY2hhbGxlbmdlIGZvciB0aGUgc3RvcmFnZSBlbmRwb2ludHMgdGhhdCBwcm9jZXNzIHRo
ZSBUMTAgUEkgaW5saW5lIHdpdGggdGhlIHRyYW5zZmVyLCBub3cgdGhleSBoYXZlIHRvIHN0b3Jl
IGFuZCBwcm9jZXNzIHRoZSA1MTJCIHNlY3RvcnMgb25jZSB0aGV5IHJlY2VpdmUgYWxsIHRoZSBU
TFBzIGZvciB0aGF0IHNlY3Rvci4NCg0KVGhlICJUMTAiIGFzcGVjdHMgYXJlIG5ldyBhbmQgSSd2
ZSBub3QgaGVhcmQgYWJvdXQgdGhlbSBiZWZvcmUuICBPbiB0aGUgc3VyZmFjZSB0aGV5IHNlZW0g
dG8gYmUgc3RvcmFnZSBkZXZpY2Ugc3BlY2lmaWMuICBJZiB0aGF0IGlzIGluZGVlZCB0aGUgY2Fz
ZSB0aGVuIHRoZXJlIHNlZW1zIHRvIGJlIHNvbWUgbWl4aW5nIG9mIHR3bywgZGlzdGluY3RseSBk
aWZmZXJlbnQsIHRoaW5ncy4gIFBDSWUgVExQcywgTVBTLCBNUlJTLCAuLi4sIGFyZSBhbGwgUENJ
ZSBkZWZpbmVkIGl0ZW1zIHRoYXQgYXJlIGNvdmVyZWQgYnkgaXRzIHNwZWNpZmljYXRpb24uICBF
eHBlY3RpbmcgdGhhdCBUMTAgc3BlY2lmaWNzIGNhbiBiZSBpbnRlcm1peGVkIHdpdGhpbiBQQ0ll
J3MgcHJvdG9jb2wgZG9lc24ndCBtYWtlIGFueSBzZW5zZSBhbmQgc291bmRzIG11Y2ggbW9yZSBs
aWtlIHNvbWV0aGluZyB0aGF0IHdpbGwgaGF2ZSB0byBiZSB0YWtlbiBjYXJlIG9mIGF0IHRoZSBj
b250cm9sbGVyJ3MNCmxldmVsLiAgIFBlcmhhcHMgSSdtICB3YXkgb2ZmIGJhc2UgaGVyZSwgd2Un
bGwgaGF2ZSB0byBoZWFyIG1vcmUgYWJvdXQgdGhpcw0KdG8gY29tZSB0byBzb21lIHVuZGVyc3Rh
bmRpbmcuDQoNCj4NCj4gU28sIGl0IGlzIGJldHRlciB0byBkZWNvdXBsZSB0aGUgTVJSUyBhbmQg
TVBTIGluIHBjaWVfYnVzX3BlcmYgbW9kZS4gTGlrZSBzdGF0ZWQgZWFybGllciBpbiB0aGUgdGhy
ZWFkLCBwcm92aWRlIGFuIG9wdGlvbiB0byBjb25maWd1cmUgTVJSUyBzZXBhcmF0ZWx5IGluIHBj
aWVfYnVzX3BlcmYgbW9kZS4NCg0KVGhpcyBhbHNvIGhhcyBiZWVuIGJyb3VnaHQgdXAgdHdpY2Ug
YWxyZWFkeSwgYW5kIGNvdmVyZWQgaW4gdGhlIHByaW9yIHJlc3BvbnNlcy4uLg0KDQo+DQo+IFJl
Z2FyZHMsDQo+IFJhZGouDQo=

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Bjorn Helgaas]

On Tue, Jan 23, 2018 at 11:50:30PM +0000, Radjendirane Codandaramane wrote:
> Hi Bjorne,

s/Bjorne/Bjorn/

Also, the convention on Linux mailing lists is to use interleaved
reply style [1] because it makes it easier to follow the conversation.

> Ceiling the MRRS to the MPS value in order to guarantee the
> interoperability in pcie_bus_perf mode does not make sense.

I disagree with this part.  The Linux pcie_bus_perf mode isn't my
favorite approach, but given the assumptions it makes, it does make
sense.  I laid out the explanation in [2]; if you disagree with
something there, please be specific about what's wrong.

> A device can make a memrd request according to the MRRS setting
> (which can be higher than its MPS), but the completer has to respect
> the MPS and send completions accordingly. As an example, system can
> configure MPS=128B and MRRS=4K, where an endpoint can a make 4K
> MemRd request, but the completer has to send completions as 128B
> TLPs, by respecting the MPS setting. MRRS does not force a device to
> use higher MPS value than it is configured to.

This part is certainly true.  I used a very similar example in [2]:

  Consider the case where a function has MPS=256 and MRRS=1024: if the
  function generates a 1024-byte read request, it should receive 4
  completions, each with a 256-byte data payload.

> Another factor that need to be considered for storage devices is
> that support of T10 Protection Information (DIF). For every 512B or
> 4KB, a 8B PI is computed and inserted or verified, which require the
> 512B of data to arrive in sequence. 

MPS and MRRS are parts of the PCIe TLP protocol.  I'm not aware of any
connection between the PCIe protocol and the T10 Protection
Information/DIF stuff.  I assume the T10/DIF stuff is content being
carried by PCIe, and PCIe doesn't care at all about the content it
carries.  Can you clarify what the connection is here?

> If the MRRS is < 512B, this might pose out of order completions to
> the storage device, if the EP has to submit multiple outstanding
> read requests in order to achieve higher performance. 

If I understand correctly, this is exactly the situation described in
the implementation note in PCIe r4.0, sec 2.4.1: completions for a
single memory read request are always returned in order (Table 2-39,
entry D5b), but completions for multiple read requests may be returned
in any order because completions may pass each other (entry D5a).  The
note doesn't mention this, but I think even the multiple outstanding
read requests themselves can pass each other (entry B3).

So I agree that if the function generates multiple outstanding read
requests, the completions for those requests may come back out of
order.  For example, if the function generates req-A and req-B,
completions for req-A will be in order and completions for req-B will
be in order, but it might receive cmpl-B1, cmpl-A1, cmpl-B2, cmpl-B3,
cmpl-A2, cmpl-A3, cmpl-A4, cmpl-B4.

This reordering may happen no matter what the MRRS setting is.  The
only way to avoid it is to avoid multiple outstanding read requests.

> This would be a challenge for the storage endpoints that process the
> T10 PI inline with the transfer, now they have to store and process
> the 512B sectors once they receive all the TLPs for that sector.

Are you saying these endpoints depend on MRRS >= 512 for correct
operation?

I don't think think that would be legal, per sec 7.5.3.4, which says
"the Function must not generate Read Requests with a size exceeding
the set [MRRS] value."  That doesn't leave the function any wiggle
room -- if the OS programs MRRS=128, the function has to deal with it.
There's no provision for the function to say "I can only support
MRRS > X".

But you probably mean the endpoints will function correctly even with
MRRS=128, but performance will be poor.  That's certainly a concern
for all devices; reducing MRRS increases overhead, so we should use
the largest possible MRRS, subject to the constraints of any MPS/MRRS
design (like pcie_bus_perf) and concerns about bandwidth allocation
(see the implementation note in sec 7.5.3.4).

> So, it is better to decouple the MRRS and MPS in pcie_bus_perf mode.
> Like stated earlier in the thread, provide an option to configure
> MRRS separately in pcie_bus_perf mode.

If you read [2] again, it will be obvious why the Linux pcie_bus_perf
mode requires the use of MRRS in addition to MPS.  At the very end of
that email and in [3], I pointed out that the current Linux
implementation is unnecessarily conservative in some cases, and there
is room for fixing that, which should improve performance in some
cases.  But given the design of pcie_bus_perf, it's impossible to
completely decouple MRRS and MPS in that mode.

[1] https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
[2] https://lkml.kernel.org/r/20180119205153.GB160618@bhelgaas-glaptop.roam.corp.google.com
[3] https://lkml.kernel.org/r/20180123174821.GF5317@bhelgaas-glaptop.roam.corp.google.com

RE: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Ron Yuan]

SGksIEkgd291bGQgbGlrZSB0byBwcm92aWRlIG1vcmUgaW5mb3JtYXRpb24ganVzdCBmb3IgYW55
b25lIHdobyBtaWdodCBiZSBpbnRlcmVzdGVkLiANCldlIG1vZGlmeSBGVyB0byBzaW11bGF0ZSBh
IE1QUyAxMjggY2FwYWJpbGl0eSBTU0QsIGFuZCBleHBlcmltZW50IHdpdGggZGlmZmVyZW50IHBj
aXNfYnVzIG1vZGUgb24gRGVsbCBSNzMwWEQsIGhlbmNlIHdlIGNhbiBoYXZlIGEgYmV0dGVyIGxv
b2sgYXQgdGhlIHdob2xlIHBpY3R1cmUuDQoNCkZpcnN0LCBjb2xkIGJvb3Qgd2l0aCBzaW5nbGUg
U1NEIA0KZGV2aWNlOiAgU2xvdCAoQyAyNTYpICAgICBTU0QgKEMgMTI4QikJU2xvdCAoQyAyNTZC
KSAgIFNTRCAoQyAyNTZCKQkNCiAgICAgICAgTVBTICBNUlJTICAgIE1QUyAgTVJSUyAgIE1QUyBN
UlJTICAgICAgIE1QUyBNUlJTIA0KTm9ybWFsCTEyOAkxMjgJICAgICAxMjgJNDA5NiANCk5vcm1h
bAkgICAgICAgICAgIAkgICAgICAgICAgICAgICAJMjU2CTEyOAkgICAgICAgMjU2CTQwOTYNClBl
cmYJICAgIDI1NgkxMjgJICAgICAxMjgJMTI4CSAgIA0KUGVyZgkgICAgICAgICAgCSAgICAgICAg
ICAgIAkgICAgICAgIDI1NgkxMjggICAgICAgIDI1NgkyNTYNClNhZmUgICAgCTEyOAkxMjgJICAg
ICAxMjgJNDA5NgkNClNhZmUJICAgICAgICAgIAkgICAgICAgICAgICAJICAgICAgICAyNTYJMTI4
CSAgICAgICAyNTYJNDA5Ng0KDQpUaGVuIGNvbGQgYm9vdCB3aXRoIHR3byBkZXZpY2VzLCBkaWZm
ZXJlbnQgTVBTIGNhcGFiaWxpdHmjrCBib3RoIHNsb3RzIGFyZSBkaXJlY3RseSBjb25uZWN0IHRv
IENQVQ0KZGV2aWNlOiAgc2xvdCAoQyAyNTYpICAgICBTU0QgKEMgMTI4QikJICAgU2xvdCAoQyAy
NTZCKSAgICAgU1NEIChDIDI1NkIpCQ0KICAgICAgICBNUFMgIE1SUlMgICAgTVBTICBNUlJTICAg
ICAgTVBTIE1SUlMgICAgICAgTVBTIE1SUlMgICAgICANCk5vcm1hbAkxMjgJMTI4CSAgICAgMTI4
CTQwOTYJMjU2ICAxMjggICAgICAgMjU2ICA0MDk2ICAgIA0KUGVyZiAgICAgMjU2CTEyOAkgICAg
IDEyOAkxMjgJICAgIDI1NiAgMTI4ICAgICAgIDI1NiAgMjU2ICAgIA0KU2FmZSAgCTEyOAkxMjgJ
ICAgICAxMjgJNDA5NgkyNTYgIDEyOCAgICAgICAyNTYgIDQwOTYgICANCg0KRmluYWxseSwgdG8g
bWF0Y2ggU2luYW4ncyBleGFtcGxlLCB3ZSB1c2UgYSBQQ0llIHN3aXRjaCBmb3IgdHdvIFUuMiBT
U0Q6DQpcLVswMDAwOjAwXS0rLTAwLjANCiAgICAgICAgICAgICArLTAxLjAtWzAzXS0tLS0wMC4w
DQogICAgICAgICAgICAgKy0wMi4wLVswNF0tLQ0KICAgICAgICAgICAgICstMDMuMC1bMDJdLS0r
LTAwLjANCiAgICAgICAgICAgICB8ICAgICAgICAgICAgXC0wMC4xDQogICAgICAgICAgICAgKy0w
My4xLVswMV0tLSstMDAuMA0KICAgICAgICAgICAgIHwgICAgICAgICAgICBcLTAwLjENCiAgICAg
ICAgICAgICArLTAzLjItWzA1LTBhXS0tLS0wMC4wLVswNi0wYV0tLSstMDQuMC1bMDddLS0NCiAg
ICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICstMDUuMC1bMDhdLS0t
LTAwLjAgICAgLT4gY29ubmVjdCBhIDI1NkIgc3NkDQogICAgICAgICAgICAgfCAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICArLTA2LjAtWzA5XS0tLS0wMC4wICAgIC0+IGNvbm5lY3QgYSAx
MjhCIHNzZA0KDQoJCTAwLjAzLjIgKEMgMjU2KSAgMDU6MDAuMCAoQzUxMikJMDY6MDUuMCAoQzUx
MikJICAwODowMC4wIChTU0QgQzI1NqOpMDY6MDYuMCAoQzUxMikJMDk6MDAuMCAoU1NEIEMxMjgp
CQ0KCQlNUFMJTVJSUwkgTVBTCU1SUlMJTVBTCU1SUlMJIE1QUwlNUlJTCU1QUyBNUlJTCU1QUwlN
UlJTDQpOb3JtYWwJMTI4CTEyOAkgICAgIDEyOAkxMjgJICAgIDEyOAkxMjgJICAgICAxMjgJNDA5
NgkxMjgJMTI4CSAgICAxMjgJNDA5Ng0KUGVyZgkJMjU2CTEyOCAgICAgIDI1NgkxMjgJICAgIDI1
NgkxMjgJICAgICAyNTYJMjU2CSAgICAyNTYJMTI4CSAgICAxMjgJMTI4DQpTYWZlCQkxMjgJMTI4
CSAgICAgMTI4CTEyOAkgICAgMTI4CTEyOAkgICAgIDEyOAk0MDk2CTEyOAkxMjgJICAgIDEyOAk0
MDk2DQoNCkkgdGhpbmsgZnJvbSBhYm92ZSBleGFtcGxlczoNCjEuIHBlcmYgbW9kZSBpcyBtb3Zp
bmcgZGV2aWNlcyB0byAyNTYgTVBTIGFzIGl0IGNhbi4NCjIuIHNhZmUgbW9kZSBpcyBzZXR0aW5n
IHRvIDEyOCBNUFMgDQozLiBwZXJmIG1vZGUgc2V0IE1SUlM9TVBTIGlzIGEgQ09SUkVDVCBjYWxs
IGZvciBkZXZpY2Ugd2l0aCBNUFNDIGxvd2VyIHRoYW4gaXRzIHBhcmVudHMuDQo0LiBwZXJmIG1v
ZGUgc2V0IE1SUlM9TVBTIGlzIG5vdCBuZWNlc3NhcnkgZm9yIGEgZGV2aWNlIHdpdGggU0FNRSBN
UFNDIGFzIGl0cyBwYXJlbnRzPyANCjUuIGl0IGlzIGFuIGludGVyZXN0ZWQgcG9pbnQgdG8gbWUg
dGhhdCBzbG90L3N3aXRjaC9yb290IE1SUlMgYXJlIGFsd2F5cyBzZXQgdG8gMTI4QiwgSSBoYXZl
IG5vdCBmb3VuZCBvdXQgd2h5Lg0KDQpBZ2FpbiwgdGhhbmtzIGZvciBldmVyeW9uZSdzIHRpbWUg
b24gdGhpcyBzdWJqZWN0LiBXZSBoYXZlIGxlYXJudCBhIGxvdC4NCg0KUm9uDQoNCi0tLS0tT3Jp
Z2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBCam9ybiBIZWxnYWFzIFttYWlsdG86aGVsZ2Fhc0Br
ZXJuZWwub3JnXSANClNlbnQ6IDIwMTjE6jHUwjI1yNUgMjowMQ0KVG86IFJhZGplbmRpcmFuZSBD
b2RhbmRhcmFtYW5lIDxyYWRqZW5kaXJhbmUuY29kYW5kYUBtaWNyb3NlbWkuY29tPg0KQ2M6IFJv
biBZdWFuIDxyb24ueXVhbkBtZW1ibGF6ZS5jb20+OyBTaW5hbiBLYXlhIDxva2F5YUBjb2RlYXVy
b3JhLm9yZz47IEJqb3JuIEhlbGdhYXMgPGJoZWxnYWFzQGdvb2dsZS5jb20+OyBCbyBDaGVuIDxi
by5jaGVuQG1lbWJsYXplLmNvbT47IFdpbGxpYW0gSHVhbmcgPHdpbGxpYW0uaHVhbmdAbWVtYmxh
emUuY29tPjsgRmVuZ21pbmcgV3UgPGZlbmdtaW5nLnd1QG1lbWJsYXplLmNvbT47IEphc29uIEpp
YW5nIDxqYXNvbi5qaWFuZ0BtaWNyb3NlbWkuY29tPjsgUmFteWFrYW50aCBFZHVwdWdhbnRpIDxS
YW15YWthbnRoLkVkdXB1Z2FudGlAbWljcm9zZW1pLmNvbT47IFdpbGxpYW0gQ2hlbmcgPHdpbGxp
YW0uY2hlbmdAbWljcm9zZW1pLmNvbT47IEtpbSBIZWxwZXIgKGtoZWxwZXIpIDxraGVscGVyQG1p
Y3Jvbi5jb20+OyBMaW51eCBQQ0kgPGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc+DQpTdWJqZWN0
OiBSZTogT25lIFF1ZXN0aW9uIEFib3V0IFBDSWUgQlVTIENvbmZpZyBUeXBlIHdpdGggcGNpZV9i
dXNfc2FmZSBvciBwY2llX2J1c19wZXJmIE9uIE5WTWUgRGV2aWNlDQoNCk9uIFR1ZSwgSmFuIDIz
LCAyMDE4IGF0IDExOjUwOjMwUE0gKzAwMDAsIFJhZGplbmRpcmFuZSBDb2RhbmRhcmFtYW5lIHdy
b3RlOg0KPiBIaSBCam9ybmUsDQoNCnMvQmpvcm5lL0Jqb3JuLw0KDQpBbHNvLCB0aGUgY29udmVu
dGlvbiBvbiBMaW51eCBtYWlsaW5nIGxpc3RzIGlzIHRvIHVzZSBpbnRlcmxlYXZlZCByZXBseSBz
dHlsZSBbMV0gYmVjYXVzZSBpdCBtYWtlcyBpdCBlYXNpZXIgdG8gZm9sbG93IHRoZSBjb252ZXJz
YXRpb24uDQoNCj4gQ2VpbGluZyB0aGUgTVJSUyB0byB0aGUgTVBTIHZhbHVlIGluIG9yZGVyIHRv
IGd1YXJhbnRlZSB0aGUgDQo+IGludGVyb3BlcmFiaWxpdHkgaW4gcGNpZV9idXNfcGVyZiBtb2Rl
IGRvZXMgbm90IG1ha2Ugc2Vuc2UuDQoNCkkgZGlzYWdyZWUgd2l0aCB0aGlzIHBhcnQuICBUaGUg
TGludXggcGNpZV9idXNfcGVyZiBtb2RlIGlzbid0IG15IGZhdm9yaXRlIGFwcHJvYWNoLCBidXQg
Z2l2ZW4gdGhlIGFzc3VtcHRpb25zIGl0IG1ha2VzLCBpdCBkb2VzIG1ha2Ugc2Vuc2UuICBJIGxh
aWQgb3V0IHRoZSBleHBsYW5hdGlvbiBpbiBbMl07IGlmIHlvdSBkaXNhZ3JlZSB3aXRoIHNvbWV0
aGluZyB0aGVyZSwgcGxlYXNlIGJlIHNwZWNpZmljIGFib3V0IHdoYXQncyB3cm9uZy4NCg0KPiBB
IGRldmljZSBjYW4gbWFrZSBhIG1lbXJkIHJlcXVlc3QgYWNjb3JkaW5nIHRvIHRoZSBNUlJTIHNl
dHRpbmcgKHdoaWNoIA0KPiBjYW4gYmUgaGlnaGVyIHRoYW4gaXRzIE1QUyksIGJ1dCB0aGUgY29t
cGxldGVyIGhhcyB0byByZXNwZWN0IHRoZSBNUFMgDQo+IGFuZCBzZW5kIGNvbXBsZXRpb25zIGFj
Y29yZGluZ2x5LiBBcyBhbiBleGFtcGxlLCBzeXN0ZW0gY2FuIGNvbmZpZ3VyZSANCj4gTVBTPTEy
OEIgYW5kIE1SUlM9NEssIHdoZXJlIGFuIGVuZHBvaW50IGNhbiBhIG1ha2UgNEsgTWVtUmQgcmVx
dWVzdCwgDQo+IGJ1dCB0aGUgY29tcGxldGVyIGhhcyB0byBzZW5kIGNvbXBsZXRpb25zIGFzIDEy
OEIgVExQcywgYnkgcmVzcGVjdGluZyANCj4gdGhlIE1QUyBzZXR0aW5nLiBNUlJTIGRvZXMgbm90
IGZvcmNlIGEgZGV2aWNlIHRvIHVzZSBoaWdoZXIgTVBTIHZhbHVlIA0KPiB0aGFuIGl0IGlzIGNv
bmZpZ3VyZWQgdG8uDQoNClRoaXMgcGFydCBpcyBjZXJ0YWlubHkgdHJ1ZS4gIEkgdXNlZCBhIHZl
cnkgc2ltaWxhciBleGFtcGxlIGluIFsyXToNCg0KICBDb25zaWRlciB0aGUgY2FzZSB3aGVyZSBh
IGZ1bmN0aW9uIGhhcyBNUFM9MjU2IGFuZCBNUlJTPTEwMjQ6IGlmIHRoZQ0KICBmdW5jdGlvbiBn
ZW5lcmF0ZXMgYSAxMDI0LWJ5dGUgcmVhZCByZXF1ZXN0LCBpdCBzaG91bGQgcmVjZWl2ZSA0DQog
IGNvbXBsZXRpb25zLCBlYWNoIHdpdGggYSAyNTYtYnl0ZSBkYXRhIHBheWxvYWQuDQoNCj4gQW5v
dGhlciBmYWN0b3IgdGhhdCBuZWVkIHRvIGJlIGNvbnNpZGVyZWQgZm9yIHN0b3JhZ2UgZGV2aWNl
cyBpcyB0aGF0IA0KPiBzdXBwb3J0IG9mIFQxMCBQcm90ZWN0aW9uIEluZm9ybWF0aW9uIChESUYp
LiBGb3IgZXZlcnkgNTEyQiBvciA0S0IsIGEgDQo+IDhCIFBJIGlzIGNvbXB1dGVkIGFuZCBpbnNl
cnRlZCBvciB2ZXJpZmllZCwgd2hpY2ggcmVxdWlyZSB0aGUgNTEyQiBvZiANCj4gZGF0YSB0byBh
cnJpdmUgaW4gc2VxdWVuY2UuDQoNCk1QUyBhbmQgTVJSUyBhcmUgcGFydHMgb2YgdGhlIFBDSWUg
VExQIHByb3RvY29sLiAgSSdtIG5vdCBhd2FyZSBvZiBhbnkgY29ubmVjdGlvbiBiZXR3ZWVuIHRo
ZSBQQ0llIHByb3RvY29sIGFuZCB0aGUgVDEwIFByb3RlY3Rpb24gSW5mb3JtYXRpb24vRElGIHN0
dWZmLiAgSSBhc3N1bWUgdGhlIFQxMC9ESUYgc3R1ZmYgaXMgY29udGVudCBiZWluZyBjYXJyaWVk
IGJ5IFBDSWUsIGFuZCBQQ0llIGRvZXNuJ3QgY2FyZSBhdCBhbGwgYWJvdXQgdGhlIGNvbnRlbnQg
aXQgY2Fycmllcy4gIENhbiB5b3UgY2xhcmlmeSB3aGF0IHRoZSBjb25uZWN0aW9uIGlzIGhlcmU/
DQoNCj4gSWYgdGhlIE1SUlMgaXMgPCA1MTJCLCB0aGlzIG1pZ2h0IHBvc2Ugb3V0IG9mIG9yZGVy
IGNvbXBsZXRpb25zIHRvIHRoZSANCj4gc3RvcmFnZSBkZXZpY2UsIGlmIHRoZSBFUCBoYXMgdG8g
c3VibWl0IG11bHRpcGxlIG91dHN0YW5kaW5nIHJlYWQgDQo+IHJlcXVlc3RzIGluIG9yZGVyIHRv
IGFjaGlldmUgaGlnaGVyIHBlcmZvcm1hbmNlLg0KDQpJZiBJIHVuZGVyc3RhbmQgY29ycmVjdGx5
LCB0aGlzIGlzIGV4YWN0bHkgdGhlIHNpdHVhdGlvbiBkZXNjcmliZWQgaW4gdGhlIGltcGxlbWVu
dGF0aW9uIG5vdGUgaW4gUENJZSByNC4wLCBzZWMgMi40LjE6IGNvbXBsZXRpb25zIGZvciBhIHNp
bmdsZSBtZW1vcnkgcmVhZCByZXF1ZXN0IGFyZSBhbHdheXMgcmV0dXJuZWQgaW4gb3JkZXIgKFRh
YmxlIDItMzksIGVudHJ5IEQ1YiksIGJ1dCBjb21wbGV0aW9ucyBmb3IgbXVsdGlwbGUgcmVhZCBy
ZXF1ZXN0cyBtYXkgYmUgcmV0dXJuZWQgaW4gYW55IG9yZGVyIGJlY2F1c2UgY29tcGxldGlvbnMg
bWF5IHBhc3MgZWFjaCBvdGhlciAoZW50cnkgRDVhKS4gIFRoZSBub3RlIGRvZXNuJ3QgbWVudGlv
biB0aGlzLCBidXQgSSB0aGluayBldmVuIHRoZSBtdWx0aXBsZSBvdXRzdGFuZGluZyByZWFkIHJl
cXVlc3RzIHRoZW1zZWx2ZXMgY2FuIHBhc3MgZWFjaCBvdGhlciAoZW50cnkgQjMpLg0KDQpTbyBJ
IGFncmVlIHRoYXQgaWYgdGhlIGZ1bmN0aW9uIGdlbmVyYXRlcyBtdWx0aXBsZSBvdXRzdGFuZGlu
ZyByZWFkIHJlcXVlc3RzLCB0aGUgY29tcGxldGlvbnMgZm9yIHRob3NlIHJlcXVlc3RzIG1heSBj
b21lIGJhY2sgb3V0IG9mIG9yZGVyLiAgRm9yIGV4YW1wbGUsIGlmIHRoZSBmdW5jdGlvbiBnZW5l
cmF0ZXMgcmVxLUEgYW5kIHJlcS1CLCBjb21wbGV0aW9ucyBmb3IgcmVxLUEgd2lsbCBiZSBpbiBv
cmRlciBhbmQgY29tcGxldGlvbnMgZm9yIHJlcS1CIHdpbGwgYmUgaW4gb3JkZXIsIGJ1dCBpdCBt
aWdodCByZWNlaXZlIGNtcGwtQjEsIGNtcGwtQTEsIGNtcGwtQjIsIGNtcGwtQjMsIGNtcGwtQTIs
IGNtcGwtQTMsIGNtcGwtQTQsIGNtcGwtQjQuDQoNClRoaXMgcmVvcmRlcmluZyBtYXkgaGFwcGVu
IG5vIG1hdHRlciB3aGF0IHRoZSBNUlJTIHNldHRpbmcgaXMuICBUaGUgb25seSB3YXkgdG8gYXZv
aWQgaXQgaXMgdG8gYXZvaWQgbXVsdGlwbGUgb3V0c3RhbmRpbmcgcmVhZCByZXF1ZXN0cy4NCg0K
PiBUaGlzIHdvdWxkIGJlIGEgY2hhbGxlbmdlIGZvciB0aGUgc3RvcmFnZSBlbmRwb2ludHMgdGhh
dCBwcm9jZXNzIHRoZQ0KPiBUMTAgUEkgaW5saW5lIHdpdGggdGhlIHRyYW5zZmVyLCBub3cgdGhl
eSBoYXZlIHRvIHN0b3JlIGFuZCBwcm9jZXNzIA0KPiB0aGUgNTEyQiBzZWN0b3JzIG9uY2UgdGhl
eSByZWNlaXZlIGFsbCB0aGUgVExQcyBmb3IgdGhhdCBzZWN0b3IuDQoNCkFyZSB5b3Ugc2F5aW5n
IHRoZXNlIGVuZHBvaW50cyBkZXBlbmQgb24gTVJSUyA+PSA1MTIgZm9yIGNvcnJlY3Qgb3BlcmF0
aW9uPw0KDQpJIGRvbid0IHRoaW5rIHRoaW5rIHRoYXQgd291bGQgYmUgbGVnYWwsIHBlciBzZWMg
Ny41LjMuNCwgd2hpY2ggc2F5cyAidGhlIEZ1bmN0aW9uIG11c3Qgbm90IGdlbmVyYXRlIFJlYWQg
UmVxdWVzdHMgd2l0aCBhIHNpemUgZXhjZWVkaW5nIHRoZSBzZXQgW01SUlNdIHZhbHVlLiIgIFRo
YXQgZG9lc24ndCBsZWF2ZSB0aGUgZnVuY3Rpb24gYW55IHdpZ2dsZSByb29tIC0tIGlmIHRoZSBP
UyBwcm9ncmFtcyBNUlJTPTEyOCwgdGhlIGZ1bmN0aW9uIGhhcyB0byBkZWFsIHdpdGggaXQuDQpU
aGVyZSdzIG5vIHByb3Zpc2lvbiBmb3IgdGhlIGZ1bmN0aW9uIHRvIHNheSAiSSBjYW4gb25seSBz
dXBwb3J0IE1SUlMgPiBYIi4NCg0KQnV0IHlvdSBwcm9iYWJseSBtZWFuIHRoZSBlbmRwb2ludHMg
d2lsbCBmdW5jdGlvbiBjb3JyZWN0bHkgZXZlbiB3aXRoIE1SUlM9MTI4LCBidXQgcGVyZm9ybWFu
Y2Ugd2lsbCBiZSBwb29yLiAgVGhhdCdzIGNlcnRhaW5seSBhIGNvbmNlcm4gZm9yIGFsbCBkZXZp
Y2VzOyByZWR1Y2luZyBNUlJTIGluY3JlYXNlcyBvdmVyaGVhZCwgc28gd2Ugc2hvdWxkIHVzZSB0
aGUgbGFyZ2VzdCBwb3NzaWJsZSBNUlJTLCBzdWJqZWN0IHRvIHRoZSBjb25zdHJhaW50cyBvZiBh
bnkgTVBTL01SUlMgZGVzaWduIChsaWtlIHBjaWVfYnVzX3BlcmYpIGFuZCBjb25jZXJucyBhYm91
dCBiYW5kd2lkdGggYWxsb2NhdGlvbiAoc2VlIHRoZSBpbXBsZW1lbnRhdGlvbiBub3RlIGluIHNl
YyA3LjUuMy40KS4NCg0KPiBTbywgaXQgaXMgYmV0dGVyIHRvIGRlY291cGxlIHRoZSBNUlJTIGFu
ZCBNUFMgaW4gcGNpZV9idXNfcGVyZiBtb2RlLg0KPiBMaWtlIHN0YXRlZCBlYXJsaWVyIGluIHRo
ZSB0aHJlYWQsIHByb3ZpZGUgYW4gb3B0aW9uIHRvIGNvbmZpZ3VyZSBNUlJTIA0KPiBzZXBhcmF0
ZWx5IGluIHBjaWVfYnVzX3BlcmYgbW9kZS4NCg0KSWYgeW91IHJlYWQgWzJdIGFnYWluLCBpdCB3
aWxsIGJlIG9idmlvdXMgd2h5IHRoZSBMaW51eCBwY2llX2J1c19wZXJmIG1vZGUgcmVxdWlyZXMg
dGhlIHVzZSBvZiBNUlJTIGluIGFkZGl0aW9uIHRvIE1QUy4gIEF0IHRoZSB2ZXJ5IGVuZCBvZiB0
aGF0IGVtYWlsIGFuZCBpbiBbM10sIEkgcG9pbnRlZCBvdXQgdGhhdCB0aGUgY3VycmVudCBMaW51
eCBpbXBsZW1lbnRhdGlvbiBpcyB1bm5lY2Vzc2FyaWx5IGNvbnNlcnZhdGl2ZSBpbiBzb21lIGNh
c2VzLCBhbmQgdGhlcmUgaXMgcm9vbSBmb3IgZml4aW5nIHRoYXQsIHdoaWNoIHNob3VsZCBpbXBy
b3ZlIHBlcmZvcm1hbmNlIGluIHNvbWUgY2FzZXMuICBCdXQgZ2l2ZW4gdGhlIGRlc2lnbiBvZiBw
Y2llX2J1c19wZXJmLCBpdCdzIGltcG9zc2libGUgdG8gY29tcGxldGVseSBkZWNvdXBsZSBNUlJT
IGFuZCBNUFMgaW4gdGhhdCBtb2RlLg0KDQpbMV0gaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dp
a2kvUG9zdGluZ19zdHlsZSNJbnRlcmxlYXZlZF9zdHlsZQ0KWzJdIGh0dHBzOi8vbGttbC5rZXJu
ZWwub3JnL3IvMjAxODAxMTkyMDUxNTMuR0IxNjA2MThAYmhlbGdhYXMtZ2xhcHRvcC5yb2FtLmNv
cnAuZ29vZ2xlLmNvbQ0KWzNdIGh0dHBzOi8vbGttbC5rZXJuZWwub3JnL3IvMjAxODAxMjMxNzQ4
MjEuR0Y1MzE3QGJoZWxnYWFzLWdsYXB0b3Aucm9hbS5jb3JwLmdvb2dsZS5jb20NCg==

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Myron Stowe]

On Wed, Jan 31, 2018 at 1:40 AM, Ron Yuan <ron.yuan@memblaze.com> wrote:
> Hi, I would like to provide more information just for anyone who might be=
 interested.
> We modify FW to simulate a MPS 128 capability SSD, and experiment with di=
fferent pcis_bus mode on Dell R730XD, hence we can have a better look at th=
e whole picture.
>
> First, cold boot with single SSD
> device:  Slot (C 256)     SSD (C 128B)  Slot (C 256B)   SSD (C 256B)
>         MPS  MRRS    MPS  MRRS   MPS MRRS       MPS MRRS
> Normal  128     128          128        4096
> Normal                                  256     128            256      4=
096
> Perf        256 128          128        128
> Perf                                            256     128        256  2=
56
> Safe            128     128          128        4096
> Safe                                            256     128            25=
6      4096
>
> Then cold boot with two devices, different MPS capability=EF=BC=8C both s=
lots are directly connect to CPU
> device:  slot (C 256)     SSD (C 128B)     Slot (C 256B)     SSD (C 256B)
>         MPS  MRRS    MPS  MRRS      MPS MRRS       MPS MRRS
> Normal  128     128          128        4096    256  128       256  4096
> Perf     256    128          128        128         256  128       256  2=
56
> Safe    128     128          128        4096    256  128       256  4096
>
> Finally, to match Sinan's example, we use a PCIe switch for two U.2 SSD:
> \-[0000:00]-+-00.0
>              +-01.0-[03]----00.0
>              +-02.0-[04]--
>              +-03.0-[02]--+-00.0
>              |            \-00.1
>              +-03.1-[01]--+-00.0
>              |            \-00.1
>              +-03.2-[05-0a]----00.0-[06-0a]--+-04.0-[07]--
>              |                               +-05.0-[08]----00.0    -> co=
nnect a 256B ssd
>              |                               +-06.0-[09]----00.0    -> co=
nnect a 128B ssd
>
>                 00.03.2 (C 256)  05:00.0 (C512) 06:05.0 (C512)    08:00.0=
 (SSD C256=EF=BC=8906:06.0 (C512)      09:00.0 (SSD C128)
>                 MPS     MRRS     MPS    MRRS    MPS     MRRS     MPS    M=
RRS    MPS MRRS        MPS     MRRS
> Normal  128     128          128        128         128 128          128 =
       4096    128     128         128 4096
> Perf            256     128      256    128         256 128          256 =
       256         256 128         128 128
> Safe            128     128          128        128         128 128      =
    128        4096    128     128         128 4096
>
> I think from above examples:
> 1. perf mode is moving devices to 256 MPS as it can.
> 2. safe mode is setting to 128 MPS
> 3. perf mode set MRRS=3DMPS is a CORRECT call for device with MPSC lower =
than its parents.
> 4. perf mode set MRRS=3DMPS is not necessary for a device with SAME MPSC =
as its parents?
> 5. it is an interested point to me that slot/switch/root MRRS are always =
set to 128B, I have not found out why.

In Sinan's original posting, a reference to
https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
was provided.  When I read that paper and got to the "Read Completion
Boundary" section I thought to myself: "If RCB can only be 64 or 128
bytes then what's the point of MPS (or MRRS) as all TLP completions
would be limited to 64 or 128 bytes? (see also the paper's 'Read
Completions with the RCB Set to 64 Bytes' figure)".  I brought this up
to a colleague and they surmised that possibly only _lower end_
(a.k.a. lazy) chipset implementations would truly have RCB limited
sized completions; higher end chipsets would of course have to comply
with RCB when communicating with the memory controller but could then
aggregate data into larger MPS (or MRRS) sized TLP completion packets.
Perhaps this might explain why you always saw slot/switch/root values
set at 128B?

>
> Again, thanks for everyone's time on this subject. We have learnt a lot.
>
> Ron

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya]

On 1/31/2018 7:01 PM, Myron Stowe wrote:
>> I think from above examples:
>> 1. perf mode is moving devices to 256 MPS as it can.
>> 2. safe mode is setting to 128 MPS
>> 3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.
>> 4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
>> 5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.
> In Sinan's original posting, a reference to
> https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
> was provided.  When I read that paper and got to the "Read Completion
> Boundary" section I thought to myself: "If RCB can only be 64 or 128
> bytes then what's the point of MPS (or MRRS) as all TLP completions
> would be limited to 64 or 128 bytes? (see also the paper's 'Read
> Completions with the RCB Set to 64 Bytes' figure)".  I brought this up
> to a colleague and they surmised that possibly only _lower end_
> (a.k.a. lazy) chipset implementations would truly have RCB limited
> sized completions; higher end chipsets would of course have to comply
> with RCB when communicating with the memory controller but could then
> aggregate data into larger MPS (or MRRS) sized TLP completion packets.
> Perhaps this might explain why you always saw slot/switch/root values
> set at 128B?
> 

I have looked at that paper before. It is plain wrong. Read Completion
Boundary is about the alignment of addresses that an endpoint is sending
in memory read packets according to the spec. It has nothing to do with
the packet size.

The limitation on MRRS could be that device (switch) doesn't support it.
A device is allowed to reject an MRRS set value.

-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Bjorn Helgaas]

On Wed, Jan 31, 2018 at 07:13:56PM -0500, Sinan Kaya wrote:
> On 1/31/2018 7:01 PM, Myron Stowe wrote:
> >> I think from above examples:
> >> 1. perf mode is moving devices to 256 MPS as it can.
> >> 2. safe mode is setting to 128 MPS
> >> 3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.
> >> 4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
> >> 5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.

> > In Sinan's original posting, a reference to
> > https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
> > was provided.  When I read that paper and got to the "Read
> > Completion Boundary" section I thought to myself: "If RCB can only
> > be 64 or 128 bytes then what's the point of MPS (or MRRS) as all
> > TLP completions would be limited to 64 or 128 bytes? (see also the
> > paper's 'Read Completions with the RCB Set to 64 Bytes' figure)".
> > I brought this up to a colleague and they surmised that possibly
> > only _lower end_ (a.k.a. lazy) chipset implementations would truly
> > have RCB limited sized completions; higher end chipsets would of
> > course have to comply with RCB when communicating with the memory
> > controller but could then aggregate data into larger MPS (or MRRS)
> > sized TLP completion packets.  Perhaps this might explain why you
> > always saw slot/switch/root values set at 128B?
> 
> I have looked at that paper before. It is plain wrong. Read
> Completion Boundary is about the alignment of addresses that an
> endpoint is sending in memory read packets according to the spec. It
> has nothing to do with the packet size.

I hope this isn't being too pedantic, but I think RCB applies to the
*completer*, not the requester.  Typically an endpoint generates a
memory read and a root port supplies data to complete it.

I didn't see anything in the "Read Completion Boundary" section of the
paper that I thought *contradicted* the spec.  I did think this was
speculation:

  Typically, most root complexes set the RCB at 64 bytes and return
  data in 64-byte completions instead of what might be allowed by the
  MPS.

That would be legal per spec, but I don't have direct knowledge of
whether root complexes actually do that.  I tend to doubt it because
it seems like people do observe performance differences based on the
MPS setting.

I agree that the "Read Transaction Throughput" section is inaccurate
when it says:

  The size of the completion is determined by a completer's read
  completion boundary.

RCB may limit the size of the first completion to less than MPS, but
subsequent completions (except the final one) only need to be a
multiple of RCB.

RCB determines the boundaries at which a completer is allowed to end a
TLP (PCIe r4.0, sec 2.3.1.1 has the rules).  Let's work through an
example:

  MPS=128
  RCB=64

  Endpoint generates a 256-byte read request for addresses 0x2-0x101

In the absence of RCB, a root port could satisfy this request with two
completions:

  completion 0: bytes   0x2- 0x81  (128 bytes)  <-- illegal
  completion 0: bytes  0x82-0x101  (128 bytes)

But with RCB, the first completion is illegal because it doesn't
satisfy the entire request and it doesn't end on a multiple of RCB.
Because of RCB the completer must respond with at least three
completions.  The best it can do is this:

  completion 0: bytes   0x2- 0x7f  (126 bytes)
  completion 1: bytes  0x80- 0xff  (128 bytes)
  completion 2: bytes 0x100-0x101  (2 bytes)

It is *allowed* but not required to respond with more completions,
e.g.,

  completion 0: bytes   0x2- 0x3f  (62 bytes)
  completion 1: bytes  0x40- 0x7f  (64 bytes)
  completion 2: bytes  0x80- 0xbf  (64 bytes)
  completion 3: bytes  0xc0- 0xff  (64 bytes)
  completion 4: bytes 0x100-0x101  (2 bytes)

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya]

On 1/31/2018 10:37 PM, Bjorn Helgaas wrote:
> On Wed, Jan 31, 2018 at 07:13:56PM -0500, Sinan Kaya wrote:
>> On 1/31/2018 7:01 PM, Myron Stowe wrote:
>>>> I think from above examples:
>>>> 1. perf mode is moving devices to 256 MPS as it can.
>>>> 2. safe mode is setting to 128 MPS
>>>> 3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.
>>>> 4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
>>>> 5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.
> 
>>> In Sinan's original posting, a reference to
>>> https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
>>> was provided.  When I read that paper and got to the "Read
>>> Completion Boundary" section I thought to myself: "If RCB can only
>>> be 64 or 128 bytes then what's the point of MPS (or MRRS) as all
>>> TLP completions would be limited to 64 or 128 bytes? (see also the
>>> paper's 'Read Completions with the RCB Set to 64 Bytes' figure)".
>>> I brought this up to a colleague and they surmised that possibly
>>> only _lower end_ (a.k.a. lazy) chipset implementations would truly
>>> have RCB limited sized completions; higher end chipsets would of
>>> course have to comply with RCB when communicating with the memory
>>> controller but could then aggregate data into larger MPS (or MRRS)
>>> sized TLP completion packets.  Perhaps this might explain why you
>>> always saw slot/switch/root values set at 128B?
>>
>> I have looked at that paper before. It is plain wrong. Read
>> Completion Boundary is about the alignment of addresses that an
>> endpoint is sending in memory read packets according to the spec. It
>> has nothing to do with the packet size.
> 
> I hope this isn't being too pedantic, but I think RCB applies to the
> *completer*, not the requester.  Typically an endpoint generates a
> memory read and a root port supplies data to complete it.
> 
> I didn't see anything in the "Read Completion Boundary" section of the
> paper that I thought *contradicted* the spec.  I did think this was
> speculation:
> 
>   Typically, most root complexes set the RCB at 64 bytes and return
>   data in 64-byte completions instead of what might be allowed by the
>   MPS.
> 
> That would be legal per spec, but I don't have direct knowledge of
> whether root complexes actually do that.  I tend to doubt it because
> it seems like people do observe performance differences based on the
> MPS setting.
> 
> I agree that the "Read Transaction Throughput" section is inaccurate
> when it says:
> 
>   The size of the completion is determined by a completer's read
>   completion boundary.
> 
> RCB may limit the size of the first completion to less than MPS, but
> subsequent completions (except the final one) only need to be a
> multiple of RCB.
> 
> RCB determines the boundaries at which a completer is allowed to end a
> TLP (PCIe r4.0, sec 2.3.1.1 has the rules).  Let's work through an
> example:
> 
>   MPS=128
>   RCB=64
> 
>   Endpoint generates a 256-byte read request for addresses 0x2-0x101
> 
> In the absence of RCB, a root port could satisfy this request with two
> completions:
> 
>   completion 0: bytes   0x2- 0x81  (128 bytes)  <-- illegal
>   completion 0: bytes  0x82-0x101  (128 bytes)
> 
> But with RCB, the first completion is illegal because it doesn't
> satisfy the entire request and it doesn't end on a multiple of RCB.
> Because of RCB the completer must respond with at least three
> completions.  The best it can do is this:
> 
>   completion 0: bytes   0x2- 0x7f  (126 bytes)
>   completion 1: bytes  0x80- 0xff  (128 bytes)
>   completion 2: bytes 0x100-0x101  (2 bytes)
> 
> It is *allowed* but not required to respond with more completions,
> e.g.,
> 
>   completion 0: bytes   0x2- 0x3f  (62 bytes)
>   completion 1: bytes  0x40- 0x7f  (64 bytes)
>   completion 2: bytes  0x80- 0xbf  (64 bytes)
>   completion 3: bytes  0xc0- 0xff  (64 bytes)
>   completion 4: bytes 0x100-0x101  (2 bytes)
> 

Very good summary. Thanks for doing a detailed analysis.


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya]

On 2/1/2018 10:14 AM, Sinan Kaya wrote:
> I think from above examples:
> 1. perf mode is moving devices to 256 MPS as it can.
> 2. safe mode is setting to 128 MPS
> 3. perf mode set MRRS=MPS is a CORRECT call for device with MPSC lower than its parents.

Thinking more about this...

The only way for a transaction of size that's greater than MRRS can happen if there
is a device specific driver attached that sends commands to the device and device issues
a large sized memory read transaction.

Let's say that we find the slow device in tree.

If no driver is attached to a slow device in the tree and assume it is safe and do not
reduce MRRS?

> 4. perf mode set MRRS=MPS is not necessary for a device with SAME MPSC as its parents?
> 5. it is an interested point to me that slot/switch/root MRRS are always set to 128B, I have not found out why.


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.