Re: [Bug 205701] New: Can't access RAM from PCIe

Re: [Bug 205701] New: Can't access RAM from PCIe

[Bjorn Helgaas]

On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > https://bugzilla.kernel.org/show_bug.cgi?id=205701
> > > ...
> > >
> > > Using Intel Xeon computer with linux kernel 4.18.0 centos8.
> > > Trying to access RAM (with DMA) using FPGA  fails in this computer.
> > >
> > > 1. I tried to add intel_iommu=off - it did not help.
> > >
> > > 2. Installing windows on same PC - FPGA can access RAM using DMA without
> > > issues.
> > >
> > > 3. using another PC (Intel Duo) with same linux and OS - FPGA access works.
> > >
> > > FPGA access the RAM using a physical address provided by a kernel module which
> > > allocates physical continuous memory in PC. (the module works perfectly with
> > > Intel Duo on exactly same OS and kernel).
> >
> > Hi, thanks for the report!  Can you please attach the complete dmesg
> > and "sudo lspci -vv" output for the working and non-working v4.18
> > kernels to the bugzilla?
> >
> > Then please try to reproduce the problem on the current v5.4 kernel
> > and attach the v5.4 dmesg log.  If v5.4 fails, we'll have to debug it.
> > If v5.4 works, figure out what fixed it (by comparing dmesg logs or by
> > bisection) and backport it to v4.18.
> >
> > Bjorn
> 
> Hi,
> I've attached 2 files:
> 1. dmesg.log - is the dmesg you've requested.
> 2. dmesg_intel_iommu_off.log - dmesg when I added intel_iommu=off
> kernel parameter.

Thanks, I attached these to the bugzilla.  I think the linux-pci
mailing list rejected your mail since it wasn't plain-text.

Please also attach the "sudo lspci -vv" output to the bugzilla and
indicate which device is your FPGA.  I guess it might be 0000:20:00.0,
since it looks like it's being claimed by an out-of-tree module in
your dmesg_intel_iommu_off.log (but not dmesg.log).

Please also attach the driver source so we can see how it is obtaining
and using the DMA buffer address.

> I might try the new kernel, yet since we are required to use the
> installation of centos8  (centos8 was just published about 2 month ago
> and it comes with kernel 4.18.0), updating kernel might be
> problematic.

Even if you can't use the v5.4 kernel for your project, if you can
establish that it works, then you have a clear path to finding the
fix.  If v5.4 still *doesn't* work, then we'll be much more interested
in helping to fix that.

> I would please like to ask if there is some workaround you can think of ?
> For example, might it help if I disable iommu (VT-d) in BIOS ?

Usually when an IOMMU blocks a DMA, it seems like there's a note in
dmesg.  I don't see that in either of your logs, but I'm not an IOMMU
expert, so it does seem reasonable to try disabling the IOMMU.

Bjorn

Re: [Bug 205701] New: Can't access RAM from PCIe

[Ranran]

On Fri, Nov 29, 2019 at 8:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> > On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=205701
> > > > ...
> > > >
> > > > Using Intel Xeon computer with linux kernel 4.18.0 centos8.
> > > > Trying to access RAM (with DMA) using FPGA  fails in this computer.
> > > >
> > > > 1. I tried to add intel_iommu=off - it did not help.
> > > >
> > > > 2. Installing windows on same PC - FPGA can access RAM using DMA without
> > > > issues.
> > > >
> > > > 3. using another PC (Intel Duo) with same linux and OS - FPGA access works.
> > > >
> > > > FPGA access the RAM using a physical address provided by a kernel module which
> > > > allocates physical continuous memory in PC. (the module works perfectly with
> > > > Intel Duo on exactly same OS and kernel).
> > >
> > > Hi, thanks for the report!  Can you please attach the complete dmesg
> > > and "sudo lspci -vv" output for the working and non-working v4.18
> > > kernels to the bugzilla?
> > >
> > > Then please try to reproduce the problem on the current v5.4 kernel
> > > and attach the v5.4 dmesg log.  If v5.4 fails, we'll have to debug it.
> > > If v5.4 works, figure out what fixed it (by comparing dmesg logs or by
> > > bisection) and backport it to v4.18.
> > >
> > > Bjorn
> >
> > Hi,
> > I've attached 2 files:
> > 1. dmesg.log - is the dmesg you've requested.
> > 2. dmesg_intel_iommu_off.log - dmesg when I added intel_iommu=off
> > kernel parameter.
>
> Thanks, I attached these to the bugzilla.  I think the linux-pci
> mailing list rejected your mail since it wasn't plain-text.
>
> Please also attach the "sudo lspci -vv" output to the bugzilla and
> indicate which device is your FPGA.  I guess it might be 0000:20:00.0,
> since it looks like it's being claimed by an out-of-tree module in
> your dmesg_intel_iommu_off.log (but not dmesg.log).
>
> Please also attach the driver source so we can see how it is obtaining
> and using the DMA buffer address.
>

I've added the module in bugzilla,

I will try to fetch the other information you requested and update in bugzilla.

Thank you

> > I might try the new kernel, yet since we are required to use the
> > installation of centos8  (centos8 was just published about 2 month ago
> > and it comes with kernel 4.18.0), updating kernel might be
> > problematic.
>
> Even if you can't use the v5.4 kernel for your project, if you can
> establish that it works, then you have a clear path to finding the
> fix.  If v5.4 still *doesn't* work, then we'll be much more interested
> in helping to fix that.
>
> > I would please like to ask if there is some workaround you can think of ?
> > For example, might it help if I disable iommu (VT-d) in BIOS ?
>
> Usually when an IOMMU blocks a DMA, it seems like there's a note in
> dmesg.  I don't see that in either of your logs, but I'm not an IOMMU
> expert, so it does seem reasonable to try disabling the IOMMU.
>
> Bjorn

Re: [Bug 205701] New: Can't access RAM from PCIe

[Ranran]

On Fri, Nov 29, 2019 at 8:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> > On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=205701
> > > > ...
> > > >
> > > > Using Intel Xeon computer with linux kernel 4.18.0 centos8.
> > > > Trying to access RAM (with DMA) using FPGA  fails in this computer.
> > > >
> > > > 1. I tried to add intel_iommu=off - it did not help.
> > > >
> > > > 2. Installing windows on same PC - FPGA can access RAM using DMA without
> > > > issues.
> > > >
> > > > 3. using another PC (Intel Duo) with same linux and OS - FPGA access works.
> > > >
> > > > FPGA access the RAM using a physical address provided by a kernel module which
> > > > allocates physical continuous memory in PC. (the module works perfectly with
> > > > Intel Duo on exactly same OS and kernel).
> > >
> > > Hi, thanks for the report!  Can you please attach the complete dmesg
> > > and "sudo lspci -vv" output for the working and non-working v4.18
> > > kernels to the bugzilla?
> > >
> > > Then please try to reproduce the problem on the current v5.4 kernel
> > > and attach the v5.4 dmesg log.  If v5.4 fails, we'll have to debug it.
> > > If v5.4 works, figure out what fixed it (by comparing dmesg logs or by
> > > bisection) and backport it to v4.18.
> > >
> > > Bjorn
> >
> > Hi,
> > I've attached 2 files:
> > 1. dmesg.log - is the dmesg you've requested.
> > 2. dmesg_intel_iommu_off.log - dmesg when I added intel_iommu=off
> > kernel parameter.
>
> Thanks, I attached these to the bugzilla.  I think the linux-pci
> mailing list rejected your mail since it wasn't plain-text.
>
> Please also attach the "sudo lspci -vv" output to the bugzilla and
> indicate which device is your FPGA.  I guess it might be 0000:20:00.0,
> since it looks like it's being claimed by an out-of-tree module in
> your dmesg_intel_iommu_off.log (but not dmesg.log).
>
> Please also attach the driver source so we can see how it is obtaining
> and using the DMA buffer address.
>
> > I might try the new kernel, yet since we are required to use the
> > installation of centos8  (centos8 was just published about 2 month ago
> > and it comes with kernel 4.18.0), updating kernel might be
> > problematic.
>
> Even if you can't use the v5.4 kernel for your project, if you can
> establish that it works, then you have a clear path to finding the
> fix.  If v5.4 still *doesn't* work, then we'll be much more interested
> in helping to fix that.
>
> > I would please like to ask if there is some workaround you can think of ?
> > For example, might it help if I disable iommu (VT-d) in BIOS ?
>
> Usually when an IOMMU blocks a DMA, it seems like there's a note in
> dmesg.  I don't see that in either of your logs, but I'm not an IOMMU
> expert, so it does seem reasonable to try disabling the IOMMU.
>
> Bjorn


Hello,

I have tried to upgrade to latest kernel 5.4 (elrepo in centos), but
with this processor/board (system x3650, Xeon), it get hang during
kernel boot, without any error in dmesg, just keeps waiting for
nothing for couple of minutes and than drops to dracut.
I am really stuck with this, is there any way to get progress with this bug ?
Please help,
Ran

Re: [Bug 205701] New: Can't access RAM from PCIe

[Bjorn Helgaas]

On Fri, Dec 06, 2019 at 08:09:48AM +0200, Ranran wrote:
> On Fri, Nov 29, 2019 at 8:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> > > On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > > > https://bugzilla.kernel.org/show_bug.cgi?id=205701

> I have tried to upgrade to latest kernel 5.4 (elrepo in centos), but
> with this processor/board (system x3650, Xeon), it get hang during
> kernel boot, without any error in dmesg, just keeps waiting for
> nothing for couple of minutes and than drops to dracut.

- I don't think you ever said exactly what the original failure mode
  was.  You said DMA from an FPGA failed.  What is the specific
  device?  How do you know the DMA fails?

- Re your v5.4 kernel testing, dracut is a user-space distro thing, so
  it sounds like your hang is some sort of installation problem that I
  can't really help you with.  Maybe there are troubleshooting hints
  at https://www.kernel.org/pub/linux/utils/boot/dracut/dracut.html.
  You may also be able to just drop a v5.4 kernel on your v4.18
  system, at least for testing purposes.

- Your comment #3 in bugzilla is a link to a Google Doc containing a
  test module.  In the future, please attach things as plain text
  attachments directly to the bugzilla.  There's an "Add attachment"
  link immediately before the "Description" comment in bugzilla.  I
  did it for you this time.

- It looks like your test_module.c is a kernel module, and frankly
  it's a mess.  Global variables that should be per-device, unused
  variables (dma_get_mask() called for no reason), confused usage
  (e.g., using both pci_dev_s and pPciDev), whitespace that appears
  random, etc.  I suggest starting with Documentation/PCI/pci.rst and,
  at least for this debugging effort, making it a self-contained
  driver instead of splitting things between a kernel module and
  user-space.

- Your comment #4 is a link to a Google Doc containing lspci output.
  I attached it to bugzilla directly for you.

- You apparently didn't run lspci as root ("sudo lspci -vv"), so it
  is missing a lot of information.

- Your lspci doesn't match either of the dmesg logs.  Please make sure
  all your logs are from the same machine in the same configuration.
  For example, the first devices found by the kernel (from both
  comments #1 and #2) are:

    pci 0000:00:00.0: [8086:3c00] type 00 class 0x060000
    pci 0000:00:01.0: [8086:3c02] type 01 class 0x060400
    pci 0000:00:02.0: [8086:3c04] type 01 class 0x060400
    pci 0000:00:02.2: [8086:3c06] type 01 class 0x060400
    ...

  But the lspci doesn't include 00:01.0, 00:02.0, or 00:02.2.  It
  shows:

    00:00.0 Host bridge: Intel Corporation Device 2020 (rev 04)
    00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
    ...

Re: [Bug 205701] New: Can't access RAM from PCIe

[Ranran]

[-- Attachment #1: Type: text/plain, Size: 3964 bytes --]

On Fri, Dec 6, 2019 at 5:08 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Fri, Dec 06, 2019 at 08:09:48AM +0200, Ranran wrote:
> > On Fri, Nov 29, 2019 at 8:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > >
> > > On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> > > > On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > > > > https://bugzilla.kernel.org/show_bug.cgi?id=205701
>
> > I have tried to upgrade to latest kernel 5.4 (elrepo in centos), but
> > with this processor/board (system x3650, Xeon), it get hang during
> > kernel boot, without any error in dmesg, just keeps waiting for
> > nothing for couple of minutes and than drops to dracut.
>
> - I don't think you ever said exactly what the original failure mode
>   was.  You said DMA from an FPGA failed.  What is the specific
>   device?  How do you know the DMA fails?
>

Hi,
FPGA is Intel's Arria 10 device.
We know that DMA fails because on using signaltap/probing the DMA
transaction from FPGA to CPU's RAM we see that it stall, i.e. keep
waiting for the access to finish.
We don't observe any error in dmesg.


> - Re your v5.4 kernel testing, dracut is a user-space distro thing, so
>   it sounds like your hang is some sort of installation problem that I
>   can't really help you with.  Maybe there are troubleshooting hints
>   at https://www.kernel.org/pub/linux/utils/boot/dracut/dracut.html.

I know, that's quite frustrating. I tried to disable features using
kernel arguments noacpi, noapic, but it still freeze somewhere without
giving any error,

>   You may also be able to just drop a v5.4 kernel on your v4.18
>   system, at least for testing purposes.
>
What does it mean to drop 5.4 kernel on 4.18 kernel ?


> - Your comment #3 in bugzilla is a link to a Google Doc containing a
>   test module.  In the future, please attach things as plain text
>   attachments directly to the bugzilla.  There's an "Add attachment"
>   link immediately before the "Description" comment in bugzilla.  I
>   did it for you this time.
>
> - It looks like your test_module.c is a kernel module, and frankly
>   it's a mess.  Global variables that should be per-device, unused
>   variables (dma_get_mask() called for no reason), confused usage
>   (e.g., using both pci_dev_s and pPciDev), whitespace that appears
>   random, etc.  I suggest starting with Documentation/PCI/pci.rst and,
>   at least for this debugging effort, making it a self-contained
>   driver instead of splitting things between a kernel module and
>   user-space.
>

I've attached latest kernel module, which I hope will make it more
clear, I will try to make it a standalone test next time I'm in lab.

> - Your comment #4 is a link to a Google Doc containing lspci output.
>   I attached it to bugzilla directly for you.
>
> - You apparently didn't run lspci as root ("sudo lspci -vv"), so it
>   is missing a lot of information.
>
> - Your lspci doesn't match either of the dmesg logs.  Please make sure
>   all your logs are from the same machine in the same configuration.
>   For example, the first devices found by the kernel (from both
>   comments #1 and #2) are:
>
>     pci 0000:00:00.0: [8086:3c00] type 00 class 0x060000
>     pci 0000:00:01.0: [8086:3c02] type 01 class 0x060400
>     pci 0000:00:02.0: [8086:3c04] type 01 class 0x060400
>     pci 0000:00:02.2: [8086:3c06] type 01 class 0x060400
>     ...
>
>   But the lspci doesn't include 00:01.0, 00:02.0, or 00:02.2.  It
>   shows:
>
>     00:00.0 Host bridge: Intel Corporation Device 2020 (rev 04)
>     00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
>     00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
>     00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
>     ...

 I will do it in lab tomorrow. Thanks.

[-- Attachment #2: Fio.c.txt --]
[-- Type: text/plain, Size: 8166 bytes --]

#include <linux/init.h>
#include <linux/module.h>
#include <linux/pci.h>
#include <linux/fs.h>
#include <linux/poll.h>
#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/wait.h>
#include <linux/sched.h>
#include <linux/cdev.h>
#include <linux/pci.h>
#include <linux/platform_device.h>

#include "../../common/FioIoctl.h"
#include "../../common/FioIn.h"

int DeviceInit(struct pci_dev *pPciDev);

static DECLARE_WAIT_QUEUE_HEAD(sWaitQueuePacket);

static u32 sInterruptFlagPacket = 0;

//static u32 sInterruptCounterPacket = 0;

static	void*		sVirtualKernelCommonBuffer;
static  u64			sPhysicalKernelCommonBuffer;
static	u32     	sPhysicalBar1Address;
static	u32			sBar1Length;
static	void*		sVirtualBar1Address;
static  dma_addr_t 	sDmaHandle;
struct  pci_dev		*pci_dev_s;

MODULE_AUTHOR("Z.V");
MODULE_DESCRIPTION("rcm");
MODULE_LICENSE("GPL");

/**********************************************************************************/
u64 ReadWord64(u32 Offset)
{
	return ioread64((void*)sVirtualBar1Address + Offset);
}

/**********************************************************************************/
void WriteWord64(u32 Offset, u64 Data)
{
	iowrite64(Data, (void*)sVirtualBar1Address + Offset);
}

/**********************************************************************************/
static long FioIoctl (struct file *file,unsigned int IoctlCode,unsigned long IoctlParam)
{
	WRITE_WORD64_REQUEST WriteWord64Request;
	READ_WORD64_REQUEST ReadWord64Request;
	GET_PHYSICAL_BUFFER_REQUEST GetPhysicalBufferRequest;

	int rc;

	//printk("--> FioIoctl. Minor=%x\n", Minor);

	switch (IoctlCode)
	{
	case WRITE_WORD64_REQUEST_CODE:
		//copy request from user space
		rc = copy_from_user(&WriteWord64Request, (void*)IoctlParam, sizeof(WRITE_WORD64_REQUEST));
		if (rc)
			printk("FioIoctl: copy_from_user failed. rc=0x%x\n", rc);
		WriteWord64 (WriteWord64Request.Offset, WriteWord64Request.Data);
		printk ("WR: 0x%08x to   0x%08x\n", WriteWord64Request.Data, WriteWord64Request.Offset);
		break;

	case READ_WORD64_REQUEST_CODE:
		//copy request from user space
		rc = copy_from_user(&ReadWord64Request, (void*)IoctlParam, sizeof(READ_WORD64_REQUEST));
		if (rc)
			printk("rcm_ioctl: copy_from_user failed. rc=0x%x\n", rc);
		ReadWord64Request.Data = ReadWord64 (ReadWord64Request.Offset);
		printk ("RD: 0x%08x from 0x%08x\n", ReadWord64Request.Data, ReadWord64Request.Offset);
		rc = copy_to_user((void*)IoctlParam, &ReadWord64Request, sizeof(READ_WORD64_REQUEST));
		if (rc)
			printk("FioIoctl: copy_to_user failed. rc=0x%x\n", rc);
		break;

	case INTERRUPT_REQUEST_CODE:
		wait_event_interruptible(sWaitQueuePacket, sInterruptFlagPacket != 0);
		sInterruptFlagPacket = 0;
		break;

	case GET_PHYSICAL_BUFFER_REQUEST_CODE:
		rc = copy_from_user(&GetPhysicalBufferRequest, (void*)IoctlParam, sizeof(GET_PHYSICAL_BUFFER_REQUEST));
		if (rc)
			printk("rcm_ioctl: copy_from_user failed. rc=0x%x\n", rc);

		GetPhysicalBufferRequest.Address = sPhysicalKernelCommonBuffer;

		rc = copy_to_user((void*)IoctlParam, &GetPhysicalBufferRequest, sizeof(GET_PHYSICAL_BUFFER_REQUEST));
		if (rc)
			printk("FioIoctl: copy_to_user failed. rc=0x%x\n", rc);
		break;

	default:
		printk ("FioIoctl: invalid ioctl code(0x%x)\n", IoctlCode);
		break;
	}
	//printk("<-- FioIoctl\n");

	return 0;
}

/**********************************************************************************/
static int FioMmap(struct file* flip, struct vm_area_struct *vma)
{
	int rc;
	
	printk("-->FioMmap\n");
	rc = remap_pfn_range  (vma, 
							vma->vm_start, 
							sPhysicalKernelCommonBuffer >> PAGE_SHIFT,
							vma->vm_end - vma->vm_start,
							vma->vm_page_prot);

	if (rc)
	{
		printk ("rcm_mmap: remap_page_range failed. rc=%d\n",rc);
		return -1;
	}
	printk("<--FioMmap\n");

	return 0;
}

/**********************************************************************************/
static irqreturn_t IrqHandlerPacket (int irq, void *data)
{
	u32 Status;

	printk ("-->IrqHandler\n");
	Status = ReadWord64(CARD_STATUS_OFFSET);
	if (Status ==0)
		return IRQ_NONE;

	WriteWord64(CARD_STATUS_OFFSET, Status);

	sInterruptFlagPacket = 1;
	wake_up_interruptible(&sWaitQueuePacket);
	printk ("<--IrqHandler\n");

	return IRQ_HANDLED;
}

/**********************************************************************************/
static int FioOpen (struct inode *inode, struct file *fl)
{
	printk("-->FioOpen\n");

	printk("<--FioOpen\n");
	return 0;
}

/**********************************************************************************/
static int FioRelease (struct inode *inode, struct file *fl)
{
	printk("-->FioRelease\n");
	printk("<--FioRelease\n");
	return 0;
}

/**********************************************************************************/
static struct file_operations sDrvOperations =
{
	unlocked_ioctl: FioIoctl,
	open	: FioOpen,
	release : FioRelease,
	mmap	: FioMmap,
	owner	: THIS_MODULE
};

/**********************************************************************************/
int AllocateCommonBuffer(struct pci_dev *pPciDev)
{
	int rc;
	u64 mask;

	/*rc=dma_set_mask_and_coherent (&pPciDev->dev, DMA_BIT_MASK(64));
	if (rc!=0)
	{
		printk ("dma_set_mask failed. rc=%d",rc);
		return -1;
	}*/
	mask = dma_get_mask(&pPciDev->dev);
	printk("maks=0x%llx\n", mask);

	//Loop for allocating common buffer. 
	sVirtualKernelCommonBuffer = dma_alloc_coherent (
							&pPciDev->dev, 
							COMMON_BUFFER_SIZE,
							&sDmaHandle,
							GFP_KERNEL | GFP_DMA);
	if (sVirtualKernelCommonBuffer == 0x0)
	{
		printk("FioInit: phys_to_virt failed\n");
		return -1;
	}

	sPhysicalKernelCommonBuffer = (u64)sDmaHandle;

	printk("sPhysicalKernelCommonBuffer=0x%llx\n", sPhysicalKernelCommonBuffer);
	*(u32*)sVirtualKernelCommonBuffer = 0xCAFE2DAD;

	return 0;
}


/**********************************************************************************/
int DeviceInit (struct pci_dev *pPciDev)
{
	int err;
	u32 Pattern;

	//enable device
	if (pci_enable_device(pPciDev))
	{
		printk("FioInit: pci_enable_device failed\n");
		return -EIO;
	}

	//find base address of BAR1
	sPhysicalBar1Address = pci_resource_start(pPciDev, 1);
	printk ("FioInit: Bar1Address=0x%x\n", sPhysicalBar1Address);

	//find length of BAR1
	sBar1Length = pci_resource_len (pPciDev, 1);
	printk ("FioInit: bar1_len_s=0x%x\n", sBar1Length);

	if (request_mem_region(sPhysicalBar1Address, sBar1Length,"fio")==NULL)
	{
		printk ("FioInit: request_mem_region failed\n");
		return -1;
	}
	
	//find virtual address of bar1
	sVirtualBar1Address = ioremap_nocache(sPhysicalBar1Address, sBar1Length);
	printk ("virtual_bar1_base_s=0x%p\n", sVirtualBar1Address);

	//Settings
	err = request_irq(pci_dev_s->irq, IrqHandlerPacket, IRQF_SHARED,"fio", pPciDev);
	if (err!=0)
	{
		printk("request_irq 0 failed.\n");
		return EBUSY;
	}

	Pattern=ReadWord64 (FPGA_VERSION_OFFSET);
	printk ("Pattern=0x%08x\n",Pattern);

	if (AllocateCommonBuffer(pPciDev) != 0)
		return -1;

	return 0;
}
/**********************************************************************************/
static int __init FioInit(void)
{
	int rc;

	printk("-->FioInit\n");

	rc=register_chrdev(0, "fio", &sDrvOperations);
	if (rc < 0)
	{
		printk("FioInit: register_chrdev failed\n");
		return -1;
	}

	printk("Major=%d\n", rc);

	//get device according to vendor,device
	pci_dev_s = pci_get_device(VENDOR_ID, DEVICE_ID, NULL);
	if (pci_dev_s == NULL)
	{
		printk("rcm_init: pci_get_device failed\n");
		return -ENODEV;
	}

	DeviceInit(pci_dev_s);

	printk("<--FioInit\n");
	return 0;
}

/**********************************************************************************/
static void __exit FioExit (void)
{
	printk(KERN_ALERT "-->FioExit\n");

	/*dma_free_coherent(&pci_dev_s->dev,
						COMMON_BUFFER_SIZE,
						sVirtualKernelCommonBuffer,
						sDmaHandle);*/
						
	printk(KERN_ALERT "<--FioExit\n");
}

module_init(FioInit);
module_exit(FioExit);

Re: [Bug 205701] New: Can't access RAM from PCIe

[Ranran]

On Fri, Dec 6, 2019 at 6:48 PM Ranran <ranshalit@gmail.com> wrote:
>
> On Fri, Dec 6, 2019 at 5:08 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > On Fri, Dec 06, 2019 at 08:09:48AM +0200, Ranran wrote:
> > > On Fri, Nov 29, 2019 at 8:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > >
> > > > On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> > > > > On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > > > > > https://bugzilla.kernel.org/show_bug.cgi?id=205701
> >
> > > I have tried to upgrade to latest kernel 5.4 (elrepo in centos), but
> > > with this processor/board (system x3650, Xeon), it get hang during
> > > kernel boot, without any error in dmesg, just keeps waiting for
> > > nothing for couple of minutes and than drops to dracut.
> >
> > - I don't think you ever said exactly what the original failure mode
> >   was.  You said DMA from an FPGA failed.  What is the specific
> >   device?  How do you know the DMA fails?
> >
>
> Hi,
> FPGA is Intel's Arria 10 device.
> We know that DMA fails because on using signaltap/probing the DMA
> transaction from FPGA to CPU's RAM we see that it stall, i.e. keep
> waiting for the access to finish.
> We don't observe any error in dmesg.
>

Two more notes about this:
1. We know that on same computer (Intel's Xeon, system x3650) the FPGA
can do the transaction without any issues.
2. Using the exact same test module in older compute/cpu (Intel's
DUO), we observe no issues in the dma transaction from FPGA.
The DMA transaction is always from FPGA to CPU's RAM.


>
> > - Re your v5.4 kernel testing, dracut is a user-space distro thing, so
> >   it sounds like your hang is some sort of installation problem that I
> >   can't really help you with.  Maybe there are troubleshooting hints
> >   at https://www.kernel.org/pub/linux/utils/boot/dracut/dracut.html.
>
> I know, that's quite frustrating. I tried to disable features using
> kernel arguments noacpi, noapic, but it still freeze somewhere without
> giving any error,
>
> >   You may also be able to just drop a v5.4 kernel on your v4.18
> >   system, at least for testing purposes.
> >
> What does it mean to drop 5.4 kernel on 4.18 kernel ?
>
>
> > - Your comment #3 in bugzilla is a link to a Google Doc containing a
> >   test module.  In the future, please attach things as plain text
> >   attachments directly to the bugzilla.  There's an "Add attachment"
> >   link immediately before the "Description" comment in bugzilla.  I
> >   did it for you this time.
> >
> > - It looks like your test_module.c is a kernel module, and frankly
> >   it's a mess.  Global variables that should be per-device, unused
> >   variables (dma_get_mask() called for no reason), confused usage
> >   (e.g., using both pci_dev_s and pPciDev), whitespace that appears
> >   random, etc.  I suggest starting with Documentation/PCI/pci.rst and,
> >   at least for this debugging effort, making it a self-contained
> >   driver instead of splitting things between a kernel module and
> >   user-space.
> >
>
> I've attached latest kernel module, which I hope will make it more
> clear, I will try to make it a standalone test next time I'm in lab.
>
> > - Your comment #4 is a link to a Google Doc containing lspci output.
> >   I attached it to bugzilla directly for you.
> >
> > - You apparently didn't run lspci as root ("sudo lspci -vv"), so it
> >   is missing a lot of information.
> >
> > - Your lspci doesn't match either of the dmesg logs.  Please make sure
> >   all your logs are from the same machine in the same configuration.
> >   For example, the first devices found by the kernel (from both
> >   comments #1 and #2) are:
> >
> >     pci 0000:00:00.0: [8086:3c00] type 00 class 0x060000
> >     pci 0000:00:01.0: [8086:3c02] type 01 class 0x060400
> >     pci 0000:00:02.0: [8086:3c04] type 01 class 0x060400
> >     pci 0000:00:02.2: [8086:3c06] type 01 class 0x060400
> >     ...
> >
> >   But the lspci doesn't include 00:01.0, 00:02.0, or 00:02.2.  It
> >   shows:
> >
> >     00:00.0 Host bridge: Intel Corporation Device 2020 (rev 04)
> >     00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
> >     00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
> >     00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04)
> >     ...
>
>  I will do it in lab tomorrow. Thanks.

Re: [Bug 205701] New: Can't access RAM from PCIe

[Bjorn Helgaas]

On Fri, Dec 06, 2019 at 06:48:24PM +0200, Ranran wrote:
> On Fri, Dec 6, 2019 at 5:08 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Fri, Dec 06, 2019 at 08:09:48AM +0200, Ranran wrote:
> > > On Fri, Nov 29, 2019 at 8:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> > > > > On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > > > > > https://bugzilla.kernel.org/show_bug.cgi?id=205701
> >
> > > I have tried to upgrade to latest kernel 5.4 (elrepo in centos), but
> > > with this processor/board (system x3650, Xeon), it get hang during
> > > kernel boot, without any error in dmesg, just keeps waiting for
> > > nothing for couple of minutes and than drops to dracut.
> >
> > - I don't think you ever said exactly what the original failure mode
> >   was.  You said DMA from an FPGA failed.  What is the specific
> >   device?  How do you know the DMA fails?
> 
> FPGA is Intel's Arria 10 device.

I really meant which bus/device/function it is so we can correlate it
with the dmesg log and lspci output.

> We know that DMA fails because on using signaltap/probing the DMA
> transaction from FPGA to CPU's RAM we see that it stall, i.e. keep
> waiting for the access to finish.
> We don't observe any error in dmesg.

I'm not familiar with Signal Tap, but Google suggests that it's
basically an embedded logic analyzer on the FPGA itself.  So I assume
that:

  - On the working system (Intel DUO?) Signal Tap shows the PCIe
    Memory Read TLP from the FPGA and the matching Completion.

  - On the non-working system Signal Tap shows the PCIe Memory Read
    TLP from the FPGA but the Completion never arrives.  I assume the
    FPGA eventually logs a Completion Timeout error?

My guess would be something's wrong with the address the FPGA is
generating.  So please collect the complete dmesg log and /proc/iomem
contents and the address used in the FPGA DMA TLP from both the
working and non-working systems.  There should be some clue if we
look at the differences between the systems.

> >   You may also be able to just drop a v5.4 kernel on your v4.18
> >   system, at least for testing purposes.
> >
> What does it mean to drop 5.4 kernel on 4.18 kernel ?

Not on a v4.18 *kernel*; on the CentOS *file system* that was
installed along with your v4.18-based kernel.  If you take a v5.4
kernel built with the right config options/modules/etc, it should work
on the same root filesystem as the v4.18 kernel.

Bjorn

Re: [Bug 205701] New: Can't access RAM from PCIe

[Ranran]

[-- Attachment #1: Type: text/plain, Size: 2937 bytes --]

On Fri, Dec 6, 2019 at 7:57 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Fri, Dec 06, 2019 at 06:48:24PM +0200, Ranran wrote:
> > On Fri, Dec 6, 2019 at 5:08 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > On Fri, Dec 06, 2019 at 08:09:48AM +0200, Ranran wrote:
> > > > On Fri, Nov 29, 2019 at 8:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> > > > > > On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > > > > > > https://bugzilla.kernel.org/show_bug.cgi?id=205701
> > >
> > > > I have tried to upgrade to latest kernel 5.4 (elrepo in centos), but
> > > > with this processor/board (system x3650, Xeon), it get hang during
> > > > kernel boot, without any error in dmesg, just keeps waiting for
> > > > nothing for couple of minutes and than drops to dracut.
> > >
> > > - I don't think you ever said exactly what the original failure mode
> > >   was.  You said DMA from an FPGA failed.  What is the specific
> > >   device?  How do you know the DMA fails?
> >
> > FPGA is Intel's Arria 10 device.
>
> I really meant which bus/device/function it is so we can correlate it
> with the dmesg log and lspci output.
>
> > We know that DMA fails because on using signaltap/probing the DMA
> > transaction from FPGA to CPU's RAM we see that it stall, i.e. keep
> > waiting for the access to finish.
> > We don't observe any error in dmesg.
>
> I'm not familiar with Signal Tap, but Google suggests that it's
> basically an embedded logic analyzer on the FPGA itself.  So I assume
> that:
>
>   - On the working system (Intel DUO?) Signal Tap shows the PCIe
>     Memory Read TLP from the FPGA and the matching Completion.
>
>   - On the non-working system Signal Tap shows the PCIe Memory Read
>     TLP from the FPGA but the Completion never arrives.  I assume the
>     FPGA eventually logs a Completion Timeout error?
>
> My guess would be something's wrong with the address the FPGA is
> generating.  So please collect the complete dmesg log and /proc/iomem
> contents and the address used in the FPGA DMA TLP from both the
> working and non-working systems.  There should be some clue if we
> look at the differences between the systems.
>
> > >   You may also be able to just drop a v5.4 kernel on your v4.18
> > >   system, at least for testing purposes.
> > >
> > What does it mean to drop 5.4 kernel on 4.18 kernel ?
>
> Not on a v4.18 *kernel*; on the CentOS *file system* that was
> installed along with your v4.18-based kernel.  If you take a v5.4
> kernel built with the right config options/modules/etc, it should work
> on the same root filesystem as the v4.18 kernel.
>
> Bjorn

Hello,

I've installed ubuntu 19.10 with kernel 5.3, and I still see same
issue with Xeon.
I've attached result of lspci -vv

Thank you,
Ran

[-- Attachment #2: lspci_vv --]
[-- Type: application/octet-stream, Size: 69665 bytes --]

00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMI2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	Capabilities: <access denied>

00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 25
	NUMA node: 0
	Bus: primary=00, secondary=0c, subordinate=10, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: None
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 27
	NUMA node: 0
	Bus: primary=00, secondary=11, subordinate=15, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: None
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 28
	NUMA node: 0
	Bus: primary=00, secondary=16, subordinate=1a, sec-latency=0
	I/O behind bridge: 00002000-00002fff [size=4K]
	Memory behind bridge: c4300000-c43fffff [size=1M]
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 30
	NUMA node: 0
	Bus: primary=00, secondary=1b, subordinate=1f, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: None
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 07) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 31
	NUMA node: 0
	Bus: primary=00, secondary=20, subordinate=24, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: c4400000-c44fffff [size=1M]
	Prefetchable memory behind bridge: 00000000c6200000-00000000c62fffff [size=1M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 0
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 92
	NUMA node: 0
	Region 0: Memory at c60e0000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 94
	NUMA node: 0
	Region 0: Memory at c60e4000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 2
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 92
	NUMA node: 0
	Region 0: Memory at c60e8000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 3
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 94
	NUMA node: 0
	Region 0: Memory at c60ec000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 4
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 92
	NUMA node: 0
	Region 0: Memory at c60f0000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 5
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 94
	NUMA node: 0
	Region 0: Memory at c60f4000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 6
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 92
	NUMA node: 0
	Region 0: Memory at c60f8000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 7
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 94
	NUMA node: 0
	Region 0: Memory at c60fc000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
	Subsystem: IBM Xeon E5/Core i7 Address Map, VTd_Misc, System Management
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	Capabilities: <access denied>

00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
	Subsystem: IBM Xeon E5/Core i7 Control Status and Global Errors
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	Capabilities: <access denied>

00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 32
	NUMA node: 0
	Bus: primary=00, secondary=25, subordinate=25, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: None
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06) (prog-if 20 [EHCI])
	Subsystem: IBM C600/X79 series chipset USB2 Enhanced Host Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin D routed to IRQ 19
	NUMA node: 0
	Region 0: Memory at c42fd000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <access denied>
	Kernel driver in use: ehci-pci

00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 33
	NUMA node: 0
	Bus: primary=00, secondary=06, subordinate=08, sec-latency=0
	I/O behind bridge: 00003000-00003fff [size=4K]
	Memory behind bridge: c4500000-c45fffff [size=1M]
	Prefetchable memory behind bridge: 00000000c6100000-00000000c61fffff [size=1M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b6) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 34
	NUMA node: 0
	Bus: primary=00, secondary=01, subordinate=05, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: c4600000-c4ffffff [size=10M]
	Prefetchable memory behind bridge: 00000000c5000000-00000000c5ffffff [size=16M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA+ VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06) (prog-if 20 [EHCI])
	Subsystem: IBM C600/X79 series chipset USB2 Enhanced Host Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 23
	NUMA node: 0
	Region 0: Memory at c42fe000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <access denied>
	Kernel driver in use: ehci-pci

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6) (prog-if 01 [Subtractive decode])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 0
	Bus: primary=00, secondary=26, subordinate=26, sec-latency=32
	I/O behind bridge: None
	Memory behind bridge: None
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>

00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
	Subsystem: IBM C600/X79 series chipset LPC Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	NUMA node: 0
	Capabilities: <access denied>
	Kernel driver in use: lpc_ich
	Kernel modules: lpc_ich

00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06) (prog-if 01 [AHCI 1.0])
	Subsystem: IBM C600/X79 series chipset 6-Port SATA AHCI Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 62
	NUMA node: 0
	Region 0: I/O ports at 1fb0 [size=8]
	Region 1: I/O ports at 1fa8 [size=4]
	Region 2: I/O ports at 1fb8 [size=8]
	Region 3: I/O ports at 1fac [size=4]
	Region 4: I/O ports at 1fc0 [size=32]
	Region 5: Memory at c42ff000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: <access denied>
	Kernel driver in use: ahci
	Kernel modules: ahci

01:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS] (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	NUMA node: 0
	BIST result: 00
	Bus: primary=01, secondary=02, subordinate=05, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: c4600000-c4ffffff [size=10M]
	Prefetchable memory behind bridge: 00000000c5000000-00000000c5ffffff [size=16M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA+ VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

02:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS] (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 35
	NUMA node: 0
	BIST result: 00
	Bus: primary=02, secondary=03, subordinate=04, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: c4700000-c4ffffff [size=9M]
	Prefetchable memory behind bridge: 00000000c5000000-00000000c5ffffff [size=16M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA+ VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

02:01.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch [PS] (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 36
	NUMA node: 0
	BIST result: 00
	Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: c4600000-c46fffff [size=1M]
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

03:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe-PCI Bridge [PPB] (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	NUMA node: 0
	BIST result: 00
	Bus: primary=03, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: c4700000-c4ffffff [size=9M]
	Prefetchable memory behind bridge: 00000000c5000000-00000000c5ffffff [size=16M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA+ VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>

04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2 (prog-if 00 [VGA controller])
	DeviceName: Matrox G200
	Subsystem: IBM G200eR2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64 (4000ns min, 8000ns max), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 20
	NUMA node: 0
	Region 0: Memory at c5000000 (32-bit, prefetchable) [size=16M]
	Region 1: Memory at c47fc000 (32-bit, non-prefetchable) [size=16K]
	Region 2: Memory at c4800000 (32-bit, non-prefetchable) [size=8M]
	[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: mgag200
	Kernel modules: mgag200

06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
	DeviceName: Intel i350
	Subsystem: Intel Corporation I350 Gigabit Network Connection
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	NUMA node: 0
	Region 0: Memory at c4580000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at 3f80 [size=32]
	Region 3: Memory at c4570000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: igb
	Kernel modules: igb

06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
	DeviceName: Intel i350
	Subsystem: Intel Corporation I350 Gigabit Network Connection
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 17
	NUMA node: 0
	Region 0: Memory at c45a0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at 3fa0 [size=32]
	Region 3: Memory at c4574000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: igb
	Kernel modules: igb

06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
	DeviceName: Intel i350
	Subsystem: Intel Corporation I350 Gigabit Network Connection
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 18
	NUMA node: 0
	Region 0: Memory at c45c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at 3fc0 [size=32]
	Region 3: Memory at c4578000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: igb
	Kernel modules: igb

06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
	DeviceName: Intel i350
	Subsystem: Intel Corporation I350 Gigabit Network Connection
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 19
	NUMA node: 0
	Region 0: Memory at c45e0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at 3fe0 [size=32]
	Region 3: Memory at c457c000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: igb
	Kernel modules: igb

16:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2208 [Thunderbolt] (rev 05)
	DeviceName: LSI 2208 SAS Controller
	Subsystem: IBM ServeRAID M5110e SAS/SATA Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 43
	NUMA node: 0
	Region 0: I/O ports at 2f00 [size=256]
	Region 1: Memory at c43bc000 (64-bit, non-prefetchable) [size=16K]
	Region 3: Memory at c43c0000 (64-bit, non-prefetchable) [size=256K]
	Expansion ROM at c4300000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: megaraid_sas
	Kernel modules: megaraid_sas

20:00.0 Non-VGA unclassified device: Altera Corporation Device 0000
	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 112
	NUMA node: 0
	Region 0: Memory at c62fc000 (64-bit, prefetchable) [size=16K]
	Region 2: Memory at c62fb000 (64-bit, prefetchable) [size=2K]
	Region 4: Memory at c44fe000 (32-bit, non-prefetchable) [size=16]
	Region 5: Memory at c44ff000 (32-bit, non-prefetchable) [size=16]
	Capabilities: <access denied>
	Kernel modules: altera_cvp

7f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:08.2 Performance counters: Intel Corporation Device 3c41 (rev 07)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

7f:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:09.2 Performance counters: Intel Corporation Device 3c42 (rev 07)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

7f:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Power Control Unit 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Power Control Unit 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Power Control Unit 2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Power Control Unit 3
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Unicast Register 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Unicast Register 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 System Address Decoder
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Unicast Register 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Unicast Register 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

7f:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>
	Kernel driver in use: snbep_uncore

7f:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>
	Kernel driver in use: snbep_uncore

7f:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>
	Kernel driver in use: snbep_uncore

7f:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>
	Kernel driver in use: snbep_uncore

7f:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

7f:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 DDRIO
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 R2PCIe
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

7f:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

7f:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

7f:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

80:00.0 PCI bridge: Intel Corporation Xeon E5/Core i7 DMI2 in PCI Express Mode (rev 07) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 38
	NUMA node: 1
	Bus: primary=80, secondary=81, subordinate=85, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: None
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 40
	NUMA node: 1
	Bus: primary=80, secondary=86, subordinate=8a, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: None
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 42
	NUMA node: 1
	Bus: primary=80, secondary=8b, subordinate=8f, sec-latency=0
	I/O behind bridge: None
	Memory behind bridge: 90100000-901fffff [size=1M]
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

80:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 0
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 102
	NUMA node: 1
	Region 0: Memory at 90000000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

80:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 1
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 104
	NUMA node: 1
	Region 0: Memory at 90004000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

80:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 2
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 102
	NUMA node: 1
	Region 0: Memory at 90008000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

80:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 3
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 104
	NUMA node: 1
	Region 0: Memory at 9000c000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

80:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 4
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 102
	NUMA node: 1
	Region 0: Memory at 90010000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

80:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 5
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 104
	NUMA node: 1
	Region 0: Memory at 90014000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

80:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 6
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 102
	NUMA node: 1
	Region 0: Memory at 90018000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

80:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07)
	Subsystem: IBM Xeon E5/Core i7 DMA Channel 7
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 104
	NUMA node: 1
	Region 0: Memory at 9001c000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: ioatdma
	Kernel modules: ioatdma

80:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
	Subsystem: IBM Xeon E5/Core i7 Address Map, VTd_Misc, System Management
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 1
	Capabilities: <access denied>

80:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
	Subsystem: IBM Xeon E5/Core i7 Control Status and Global Errors
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	NUMA node: 1
	Capabilities: <access denied>

8b:00.0 PCI bridge: PLX Technology, Inc. PEX 8114 PCI Express-to-PCI/PCI-X Bridge (rev bd) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 255
	NUMA node: 1
	Region 0: Memory at 901fe000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Bus: primary=8b, secondary=8c, subordinate=8c, sec-latency=64
	I/O behind bridge: None
	Memory behind bridge: None
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>

ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:08.2 Performance counters: Intel Corporation Device 3c41 (rev 07)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

ff:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:09.2 Performance counters: Intel Corporation Device 3c42 (rev 07)
	Subsystem: Intel Corporation Device 0000
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

ff:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Power Control Unit 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Power Control Unit 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Power Control Unit 2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Power Control Unit 3
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Unicast Register 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Unicast Register 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 System Address Decoder
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Unicast Register 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Unicast Register 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07)
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

ff:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>
	Kernel driver in use: snbep_uncore

ff:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>
	Kernel driver in use: snbep_uncore

ff:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>
	Kernel driver in use: snbep_uncore

ff:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>
	Kernel driver in use: snbep_uncore

ff:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

ff:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 DDRIO
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 R2PCIe
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

ff:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

ff:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

ff:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)
	Subsystem: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: snbep_uncore

Re: [Bug 205701] New: Can't access RAM from PCIe

[Bjorn Helgaas]

On Sun, Dec 15, 2019 at 07:29:58PM +0200, Ranran wrote:
> On Fri, Dec 6, 2019 at 7:57 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Fri, Dec 06, 2019 at 06:48:24PM +0200, Ranran wrote:
> > > On Fri, Dec 6, 2019 at 5:08 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > On Fri, Dec 06, 2019 at 08:09:48AM +0200, Ranran wrote:
> > > > > On Fri, Nov 29, 2019 at 8:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> > > > > > > On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > > > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > > > > > > > https://bugzilla.kernel.org/show_bug.cgi?id=205701
> > > >
> > > > > I have tried to upgrade to latest kernel 5.4 (elrepo in centos), but
> > > > > with this processor/board (system x3650, Xeon), it get hang during
> > > > > kernel boot, without any error in dmesg, just keeps waiting for
> > > > > nothing for couple of minutes and than drops to dracut.
> > > >
> > > > - I don't think you ever said exactly what the original failure mode
> > > >   was.  You said DMA from an FPGA failed.  What is the specific
> > > >   device?  How do you know the DMA fails?
> > >
> > > FPGA is Intel's Arria 10 device.
> >
> > I really meant which bus/device/function it is so we can correlate it
> > with the dmesg log and lspci output.
> >
> > > We know that DMA fails because on using signaltap/probing the DMA
> > > transaction from FPGA to CPU's RAM we see that it stall, i.e. keep
> > > waiting for the access to finish.
> > > We don't observe any error in dmesg.
> >
> > ... So I assume that:
> >
> >   - On the working system (Intel DUO?) Signal Tap shows the PCIe
> >     Memory Read TLP from the FPGA and the matching Completion.
> >
> >   - On the non-working system Signal Tap shows the PCIe Memory Read
> >     TLP from the FPGA but the Completion never arrives.  I assume the
> >     FPGA eventually logs a Completion Timeout error?
> >
> > My guess would be something's wrong with the address the FPGA is
> > generating.  So please collect the complete dmesg log and /proc/iomem
> > contents and the address used in the FPGA DMA TLP from both the
> > working and non-working systems.  There should be some clue if we
> > look at the differences between the systems.

> I've installed ubuntu 19.10 with kernel 5.3, and I still see same
> issue with Xeon.
> I've attached result of lspci -vv

Sorry, all this information is incomplete and inconsistent, so I can't
give you any useful help.  The dmesg log contains output not produced
by the module you supplied, the lspci doesn't match the dmesg output,
the lspci was not collected as root ("sudo lspci -vv", as I requested
twice), you haven't supplied the user-space code, etc.

Bjorn

Re: [Bug 205701] New: Can't access RAM from PCIe

[Bjorn Helgaas]

On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=205701
> ...
> 
> Using Intel Xeon computer with linux kernel 4.18.0 centos8.
> Trying to access RAM (with DMA) using FPGA  fails in this computer.
> 
> 1. I tried to add intel_iommu=off - it did not help.
> 
> 2. Installing windows on same PC - FPGA can access RAM using DMA without
> issues.
> 
> 3. using another PC (Intel Duo) with same linux and OS - FPGA access works.  
> 
> FPGA access the RAM using a physical address provided by a kernel module which
> allocates physical continuous memory in PC. (the module works perfectly with
> Intel Duo on exactly same OS and kernel).

Hi, thanks for the report!  Can you please attach the complete dmesg
and "sudo lspci -vv" output for the working and non-working v4.18
kernels to the bugzilla?

Then please try to reproduce the problem on the current v5.4 kernel
and attach the v5.4 dmesg log.  If v5.4 fails, we'll have to debug it.
If v5.4 works, figure out what fixed it (by comparing dmesg logs or by
bisection) and backport it to v4.18.

Bjorn