[PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Helgaas,

Some AMD GPUs have hardware support for graphics SR-IOV.
If the SR-IOV capable GPU is plugged into the SR-IOV incapable
platform. It would cause a problem on PCI resource allocation in
current Linux kernel.

Therefore in order to allow the PF (Physical Function) device of
SR-IOV capable GPU to work on the SR-IOV incapable platform,
it is required to verify conditions for initializing BAR resources
on AMD SR-IOV capable GPUs.

If the device is an AMD graphics device and it supports
SR-IOV it will require a large amount of resources.
Before calling sriov_init() must ensure that the system
BIOS also supports SR-IOV and that system BIOS has been
able to allocate enough resources.
If the VF BARs are zero then the system BIOS does not
support SR-IOV or it could not allocate the resources
and this platform will not support AMD graphics SR-IOV.
Therefore do not call sriov_init().
If the system BIOS does support SR-IOV then the VF BARs
will be properly initialized to non-zero values.

Below is the patch against to Kernel 4.8 & 4.9. Please review.

I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in
quirks.c are for specific devices and one or more device ID are defined
for the specific devices. However my patch is for all AMD SR-IOV
capable GPUs, that includes all existing and future AMD server GPUs.
So it doesn't seem like a good fit to put the fix in quirks.c.



Signed-off-by: Collins Cheng <collins.cheng@amd.com>
---
 drivers/pci/iov.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 60 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index e30f05c..e4f1405 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
 		msleep(100);
 }
 
+/*
+ * pci_vf_bar_valid - check if VF BARs have resource allocated
+ * @dev: the PCI device
+ * @pos: register offset of SR-IOV capability in PCI config space
+ * Returns true any VF BAR has resource allocated, false
+ * if all VF BARs are empty.
+ */
+static bool pci_vf_bar_valid(struct pci_dev *dev, int pos)
+{
+	int i;
+	u32 bar_value;
+	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
+			PCI_BASE_ADDRESS_MEM_TYPE_64 |
+			PCI_BASE_ADDRESS_MEM_PREFETCH);
+
+	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
+		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
+		if (bar_value & bar_size_mask)
+			return true;
+	}
+
+	return false;
+}
+
+/*
+ * is_amd_display_adapter - check if it is an AMD/ATI GPU device
+ * @dev: the PCI device
+ *
+ * Returns true if device is an AMD/ATI display adapter,
+ * otherwise return false.
+ */
+
+static bool is_amd_display_adapter(struct pci_dev *dev)
+{
+	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
+		(dev->vendor == PCI_VENDOR_ID_ATI ||
+		dev->vendor == PCI_VENDOR_ID_AMD));
+}
+
 /**
  * pci_iov_init - initialize the IOV capability
  * @dev: the PCI device
@@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
 		return -ENODEV;
 
 	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
-	if (pos)
-		return sriov_init(dev, pos);
-
+	if (pos) {
+	/*
+	 * If the device is an AMD graphics device and it supports
+	 * SR-IOV it will require a large amount of resources.
+	 * Before calling sriov_init() must ensure that the system
+	 * BIOS also supports SR-IOV and that system BIOS has been
+	 * able to allocate enough resources.
+	 * If the VF BARs are zero then the system BIOS does not
+	 * support SR-IOV or it could not allocate the resources
+	 * and this platform will not support AMD graphics SR-IOV.
+	 * Therefore do not call sriov_init().
+	 * If the system BIOS does support SR-IOV then the VF BARs
+	 * will be properly initialized to non-zero values.
+	 */
+		if (is_amd_display_adapter(dev)) {
+			if (pci_vf_bar_valid(dev, pos))
+				return sriov_init(dev, pos);
+		} else {
+			return sriov_init(dev, pos);
+		}
+	}
 	return -ENODEV;
 }
 
-- 
1.9.1



-Collins Cheng

[PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Helgaas,

Some AMD GPUs have hardware support for graphics SR-IOV.
If the SR-IOV capable GPU is plugged into the SR-IOV incapable
platform. It would cause a problem on PCI resource allocation in
current Linux kernel.

Therefore in order to allow the PF (Physical Function) device of
SR-IOV capable GPU to work on the SR-IOV incapable platform,
it is required to verify conditions for initializing BAR resources
on AMD SR-IOV capable GPUs.

If the device is an AMD graphics device and it supports
SR-IOV it will require a large amount of resources.
Before calling sriov_init() must ensure that the system
BIOS also supports SR-IOV and that system BIOS has been
able to allocate enough resources.
If the VF BARs are zero then the system BIOS does not
support SR-IOV or it could not allocate the resources
and this platform will not support AMD graphics SR-IOV.
Therefore do not call sriov_init().
If the system BIOS does support SR-IOV then the VF BARs
will be properly initialized to non-zero values.

Below is the patch against to Kernel 4.8 & 4.9. Please review.

I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in
quirks.c are for specific devices and one or more device ID are defined
for the specific devices. However my patch is for all AMD SR-IOV
capable GPUs, that includes all existing and future AMD server GPUs.
So it doesn't seem like a good fit to put the fix in quirks.c.



Signed-off-by: Collins Cheng <collins.cheng@amd.com>
---
 drivers/pci/iov.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++=
+---
 1 file changed, 60 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index e30f05c..e4f1405 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
 		msleep(100);
 }
=20
+/*
+ * pci_vf_bar_valid - check if VF BARs have resource allocated
+ * @dev: the PCI device
+ * @pos: register offset of SR-IOV capability in PCI config space
+ * Returns true any VF BAR has resource allocated, false
+ * if all VF BARs are empty.
+ */
+static bool pci_vf_bar_valid(struct pci_dev *dev, int pos)
+{
+	int i;
+	u32 bar_value;
+	u32 bar_size_mask =3D ~(PCI_BASE_ADDRESS_SPACE |
+			PCI_BASE_ADDRESS_MEM_TYPE_64 |
+			PCI_BASE_ADDRESS_MEM_PREFETCH);
+
+	for (i =3D 0; i < PCI_SRIOV_NUM_BARS; i++) {
+		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
+		if (bar_value & bar_size_mask)
+			return true;
+	}
+
+	return false;
+}
+
+/*
+ * is_amd_display_adapter - check if it is an AMD/ATI GPU device
+ * @dev: the PCI device
+ *
+ * Returns true if device is an AMD/ATI display adapter,
+ * otherwise return false.
+ */
+
+static bool is_amd_display_adapter(struct pci_dev *dev)
+{
+	return (((dev->class >> 16) =3D=3D PCI_BASE_CLASS_DISPLAY) &&
+		(dev->vendor =3D=3D PCI_VENDOR_ID_ATI ||
+		dev->vendor =3D=3D PCI_VENDOR_ID_AMD));
+}
+
 /**
  * pci_iov_init - initialize the IOV capability
  * @dev: the PCI device
@@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
 		return -ENODEV;
=20
 	pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
-	if (pos)
-		return sriov_init(dev, pos);
-
+	if (pos) {
+	/*
+	 * If the device is an AMD graphics device and it supports
+	 * SR-IOV it will require a large amount of resources.
+	 * Before calling sriov_init() must ensure that the system
+	 * BIOS also supports SR-IOV and that system BIOS has been
+	 * able to allocate enough resources.
+	 * If the VF BARs are zero then the system BIOS does not
+	 * support SR-IOV or it could not allocate the resources
+	 * and this platform will not support AMD graphics SR-IOV.
+	 * Therefore do not call sriov_init().
+	 * If the system BIOS does support SR-IOV then the VF BARs
+	 * will be properly initialized to non-zero values.
+	 */
+		if (is_amd_display_adapter(dev)) {
+			if (pci_vf_bar_valid(dev, pos))
+				return sriov_init(dev, pos);
+		} else {
+			return sriov_init(dev, pos);
+		}
+	}
 	return -ENODEV;
 }
=20
--=20
1.9.1



-Collins Cheng

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alex Williamson]

On Fri, 12 May 2017 02:50:32 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Helgaas,
> 
> Some AMD GPUs have hardware support for graphics SR-IOV.
> If the SR-IOV capable GPU is plugged into the SR-IOV incapable
> platform. It would cause a problem on PCI resource allocation in
> current Linux kernel.
> 
> Therefore in order to allow the PF (Physical Function) device of
> SR-IOV capable GPU to work on the SR-IOV incapable platform,
> it is required to verify conditions for initializing BAR resources
> on AMD SR-IOV capable GPUs.
> 
> If the device is an AMD graphics device and it supports
> SR-IOV it will require a large amount of resources.
> Before calling sriov_init() must ensure that the system
> BIOS also supports SR-IOV and that system BIOS has been
> able to allocate enough resources.
> If the VF BARs are zero then the system BIOS does not
> support SR-IOV or it could not allocate the resources
> and this platform will not support AMD graphics SR-IOV.
> Therefore do not call sriov_init().
> If the system BIOS does support SR-IOV then the VF BARs
> will be properly initialized to non-zero values.
> 
> Below is the patch against to Kernel 4.8 & 4.9. Please review.
> 
> I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in
> quirks.c are for specific devices and one or more device ID are defined
> for the specific devices. However my patch is for all AMD SR-IOV
> capable GPUs, that includes all existing and future AMD server GPUs.
> So it doesn't seem like a good fit to put the fix in quirks.c.


Why is an AMD graphics card unique here?  Doesn't sriov_init() always
need to be able to deal with devices of any type where the BIOS hasn't
initialized the SR-IOV capability?  Some SR-IOV devices can fit their
VFs within a minimum bridge aperture, most cannot.  I don't understand
why the VF resource requirements being exceptionally large dictates
that they receive special handling.  Thanks,

Alex

> Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> ---
>  drivers/pci/iov.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 60 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index e30f05c..e4f1405 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
>  		msleep(100);
>  }
>  
> +/*
> + * pci_vf_bar_valid - check if VF BARs have resource allocated
> + * @dev: the PCI device
> + * @pos: register offset of SR-IOV capability in PCI config space
> + * Returns true any VF BAR has resource allocated, false
> + * if all VF BARs are empty.
> + */
> +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos)
> +{
> +	int i;
> +	u32 bar_value;
> +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> +
> +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> +		if (bar_value & bar_size_mask)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
> +/*
> + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> + * @dev: the PCI device
> + *
> + * Returns true if device is an AMD/ATI display adapter,
> + * otherwise return false.
> + */
> +
> +static bool is_amd_display_adapter(struct pci_dev *dev)
> +{
> +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> +		(dev->vendor == PCI_VENDOR_ID_ATI ||
> +		dev->vendor == PCI_VENDOR_ID_AMD));
> +}
> +
>  /**
>   * pci_iov_init - initialize the IOV capability
>   * @dev: the PCI device
> @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>  		return -ENODEV;
>  
>  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> -	if (pos)
> -		return sriov_init(dev, pos);
> -
> +	if (pos) {
> +	/*
> +	 * If the device is an AMD graphics device and it supports
> +	 * SR-IOV it will require a large amount of resources.
> +	 * Before calling sriov_init() must ensure that the system
> +	 * BIOS also supports SR-IOV and that system BIOS has been
> +	 * able to allocate enough resources.
> +	 * If the VF BARs are zero then the system BIOS does not
> +	 * support SR-IOV or it could not allocate the resources
> +	 * and this platform will not support AMD graphics SR-IOV.
> +	 * Therefore do not call sriov_init().
> +	 * If the system BIOS does support SR-IOV then the VF BARs
> +	 * will be properly initialized to non-zero values.
> +	 */
> +		if (is_amd_display_adapter(dev)) {
> +			if (pci_vf_bar_valid(dev, pos))
> +				return sriov_init(dev, pos);
> +		} else {
> +			return sriov_init(dev, pos);
> +		}
> +	}
>  	return -ENODEV;
>  }
>  

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Williamson,

GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.

If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.

-Collins Cheng

-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com] 
Sent: Friday, May 12, 2017 11:21 AM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Fri, 12 May 2017 02:50:32 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Helgaas,
> 
> Some AMD GPUs have hardware support for graphics SR-IOV.
> If the SR-IOV capable GPU is plugged into the SR-IOV incapable 
> platform. It would cause a problem on PCI resource allocation in 
> current Linux kernel.
> 
> Therefore in order to allow the PF (Physical Function) device of 
> SR-IOV capable GPU to work on the SR-IOV incapable platform, it is 
> required to verify conditions for initializing BAR resources on AMD 
> SR-IOV capable GPUs.
> 
> If the device is an AMD graphics device and it supports SR-IOV it will 
> require a large amount of resources.
> Before calling sriov_init() must ensure that the system BIOS also 
> supports SR-IOV and that system BIOS has been able to allocate enough 
> resources.
> If the VF BARs are zero then the system BIOS does not support SR-IOV 
> or it could not allocate the resources and this platform will not 
> support AMD graphics SR-IOV.
> Therefore do not call sriov_init().
> If the system BIOS does support SR-IOV then the VF BARs will be 
> properly initialized to non-zero values.
> 
> Below is the patch against to Kernel 4.8 & 4.9. Please review.
> 
> I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in 
> quirks.c are for specific devices and one or more device ID are 
> defined for the specific devices. However my patch is for all AMD 
> SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> So it doesn't seem like a good fit to put the fix in quirks.c.


Why is an AMD graphics card unique here?  Doesn't sriov_init() always need to be able to deal with devices of any type where the BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV devices can fit their VFs within a minimum bridge aperture, most cannot.  I don't understand why the VF resource requirements being exceptionally large dictates that they receive special handling.  Thanks,

Alex

> Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> ---
>  drivers/pci/iov.c | 63 
> ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 60 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 
> e30f05c..e4f1405 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
>  		msleep(100);
>  }
>  
> +/*
> + * pci_vf_bar_valid - check if VF BARs have resource allocated
> + * @dev: the PCI device
> + * @pos: register offset of SR-IOV capability in PCI config space
> + * Returns true any VF BAR has resource allocated, false
> + * if all VF BARs are empty.
> + */
> +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> +	int i;
> +	u32 bar_value;
> +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> +
> +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> +		if (bar_value & bar_size_mask)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
> +/*
> + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> + * @dev: the PCI device
> + *
> + * Returns true if device is an AMD/ATI display adapter,
> + * otherwise return false.
> + */
> +
> +static bool is_amd_display_adapter(struct pci_dev *dev) {
> +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> +		(dev->vendor == PCI_VENDOR_ID_ATI ||
> +		dev->vendor == PCI_VENDOR_ID_AMD)); }
> +
>  /**
>   * pci_iov_init - initialize the IOV capability
>   * @dev: the PCI device
> @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>  		return -ENODEV;
>  
>  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> -	if (pos)
> -		return sriov_init(dev, pos);
> -
> +	if (pos) {
> +	/*
> +	 * If the device is an AMD graphics device and it supports
> +	 * SR-IOV it will require a large amount of resources.
> +	 * Before calling sriov_init() must ensure that the system
> +	 * BIOS also supports SR-IOV and that system BIOS has been
> +	 * able to allocate enough resources.
> +	 * If the VF BARs are zero then the system BIOS does not
> +	 * support SR-IOV or it could not allocate the resources
> +	 * and this platform will not support AMD graphics SR-IOV.
> +	 * Therefore do not call sriov_init().
> +	 * If the system BIOS does support SR-IOV then the VF BARs
> +	 * will be properly initialized to non-zero values.
> +	 */
> +		if (is_amd_display_adapter(dev)) {
> +			if (pci_vf_bar_valid(dev, pos))
> +				return sriov_init(dev, pos);
> +		} else {
> +			return sriov_init(dev, pos);
> +		}
> +	}
>  	return -ENODEV;
>  }
>  

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Williamson,

GPU card needs more BAR aperture resource than other PCI devices. For examp=
le, Intel SR-IOV network card only require 512KB memory resource for all VF=
s. AMD SR-IOV GPU card needs 256MB x16 VF =3D 4GB memory resource for frame=
 buffer BAR aperture.

If the system BIOS supports SR-IOV, it will reserve enough resource for all=
 VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the =
enough resource for VF BARs, only PF BAR will be assigned and VF BARs are e=
mpty. Then system boots to Linux kernel and kernel doesn't check if the VF =
BARs are empty or valid. Kernel will re-assign the BAR resources for PF and=
 all VFs. The problem I saw is that kernel will fail to allocate PF BAR res=
ource because some resources are assigned to VF, this is not expected. So k=
ernel might need to do some check before re-assign the PF/VF resource, so t=
hat PF device will be correctly assigned BAR resource and user can use PF d=
evice.

-Collins Cheng

-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com]=20
Sent: Friday, May 12, 2017 11:21 AM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-k=
ernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zyta=
ruk, Kelly <Kelly.Zytaruk@amd.com>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV inc=
apable platform

On Fri, 12 May 2017 02:50:32 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Helgaas,
>=20
> Some AMD GPUs have hardware support for graphics SR-IOV.
> If the SR-IOV capable GPU is plugged into the SR-IOV incapable=20
> platform. It would cause a problem on PCI resource allocation in=20
> current Linux kernel.
>=20
> Therefore in order to allow the PF (Physical Function) device of=20
> SR-IOV capable GPU to work on the SR-IOV incapable platform, it is=20
> required to verify conditions for initializing BAR resources on AMD=20
> SR-IOV capable GPUs.
>=20
> If the device is an AMD graphics device and it supports SR-IOV it will=20
> require a large amount of resources.
> Before calling sriov_init() must ensure that the system BIOS also=20
> supports SR-IOV and that system BIOS has been able to allocate enough=20
> resources.
> If the VF BARs are zero then the system BIOS does not support SR-IOV=20
> or it could not allocate the resources and this platform will not=20
> support AMD graphics SR-IOV.
> Therefore do not call sriov_init().
> If the system BIOS does support SR-IOV then the VF BARs will be=20
> properly initialized to non-zero values.
>=20
> Below is the patch against to Kernel 4.8 & 4.9. Please review.
>=20
> I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in=20
> quirks.c are for specific devices and one or more device ID are=20
> defined for the specific devices. However my patch is for all AMD=20
> SR-IOV capable GPUs, that includes all existing and future AMD server GPU=
s.
> So it doesn't seem like a good fit to put the fix in quirks.c.


Why is an AMD graphics card unique here?  Doesn't sriov_init() always need =
to be able to deal with devices of any type where the BIOS hasn't initializ=
ed the SR-IOV capability?  Some SR-IOV devices can fit their VFs within a m=
inimum bridge aperture, most cannot.  I don't understand why the VF resourc=
e requirements being exceptionally large dictates that they receive special=
 handling.  Thanks,

Alex

> Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> ---
>  drivers/pci/iov.c | 63=20
> ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 60 insertions(+), 3 deletions(-)
>=20
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index=20
> e30f05c..e4f1405 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
>  		msleep(100);
>  }
> =20
> +/*
> + * pci_vf_bar_valid - check if VF BARs have resource allocated
> + * @dev: the PCI device
> + * @pos: register offset of SR-IOV capability in PCI config space
> + * Returns true any VF BAR has resource allocated, false
> + * if all VF BARs are empty.
> + */
> +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> +	int i;
> +	u32 bar_value;
> +	u32 bar_size_mask =3D ~(PCI_BASE_ADDRESS_SPACE |
> +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> +
> +	for (i =3D 0; i < PCI_SRIOV_NUM_BARS; i++) {
> +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> +		if (bar_value & bar_size_mask)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
> +/*
> + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> + * @dev: the PCI device
> + *
> + * Returns true if device is an AMD/ATI display adapter,
> + * otherwise return false.
> + */
> +
> +static bool is_amd_display_adapter(struct pci_dev *dev) {
> +	return (((dev->class >> 16) =3D=3D PCI_BASE_CLASS_DISPLAY) &&
> +		(dev->vendor =3D=3D PCI_VENDOR_ID_ATI ||
> +		dev->vendor =3D=3D PCI_VENDOR_ID_AMD)); }
> +
>  /**
>   * pci_iov_init - initialize the IOV capability
>   * @dev: the PCI device
> @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>  		return -ENODEV;
> =20
>  	pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> -	if (pos)
> -		return sriov_init(dev, pos);
> -
> +	if (pos) {
> +	/*
> +	 * If the device is an AMD graphics device and it supports
> +	 * SR-IOV it will require a large amount of resources.
> +	 * Before calling sriov_init() must ensure that the system
> +	 * BIOS also supports SR-IOV and that system BIOS has been
> +	 * able to allocate enough resources.
> +	 * If the VF BARs are zero then the system BIOS does not
> +	 * support SR-IOV or it could not allocate the resources
> +	 * and this platform will not support AMD graphics SR-IOV.
> +	 * Therefore do not call sriov_init().
> +	 * If the system BIOS does support SR-IOV then the VF BARs
> +	 * will be properly initialized to non-zero values.
> +	 */
> +		if (is_amd_display_adapter(dev)) {
> +			if (pci_vf_bar_valid(dev, pos))
> +				return sriov_init(dev, pos);
> +		} else {
> +			return sriov_init(dev, pos);
> +		}
> +	}
>  	return -ENODEV;
>  }
> =20

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Zytaruk, Kelly]

>-----Original Message-----
>From: Alex Williamson [mailto:alex.williamson@redhat.com]
>Sent: Thursday, May 11, 2017 11:21 PM
>To: Cheng, Collins
>Cc: Bjorn Helgaas; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
>Deucher, Alexander; Zytaruk, Kelly
>Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>incapable platform
>
>On Fri, 12 May 2017 02:50:32 +0000
>"Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>
>> Hi Helgaas,
>>
>> Some AMD GPUs have hardware support for graphics SR-IOV.
>> If the SR-IOV capable GPU is plugged into the SR-IOV incapable
>> platform. It would cause a problem on PCI resource allocation in
>> current Linux kernel.
>>
>> Therefore in order to allow the PF (Physical Function) device of
>> SR-IOV capable GPU to work on the SR-IOV incapable platform, it is
>> required to verify conditions for initializing BAR resources on AMD
>> SR-IOV capable GPUs.
>>
>> If the device is an AMD graphics device and it supports SR-IOV it will
>> require a large amount of resources.
>> Before calling sriov_init() must ensure that the system BIOS also
>> supports SR-IOV and that system BIOS has been able to allocate enough
>> resources.
>> If the VF BARs are zero then the system BIOS does not support SR-IOV
>> or it could not allocate the resources and this platform will not
>> support AMD graphics SR-IOV.
>> Therefore do not call sriov_init().
>> If the system BIOS does support SR-IOV then the VF BARs will be
>> properly initialized to non-zero values.
>>
>> Below is the patch against to Kernel 4.8 & 4.9. Please review.
>>
>> I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in
>> quirks.c are for specific devices and one or more device ID are
>> defined for the specific devices. However my patch is for all AMD
>> SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
>> So it doesn't seem like a good fit to put the fix in quirks.c.
>
>
>Why is an AMD graphics card unique here?  Doesn't sriov_init() always need to be
>able to deal with devices of any type where the BIOS hasn't initialized the SR-IOV
>capability?  Some SR-IOV devices can fit their VFs within a minimum bridge
>aperture, most cannot.  I don't understand why the VF resource requirements
>being exceptionally large dictates that they receive special handling.  Thanks,
>
>Alex
>

Hi Alex,  
Many System Bios's are problematic in that they don't know how to fully support SRIOV.  Up until recently SRIOV devices typically used a small amount of resources, such as a NIC.
The AMD SRIOV GPU uses significant resources and many SBios' cannot handle this properly.  The faulty SBios will attempt to initialize, run out of resources and not indicate any error.
Even though we are not enabling SRIOV on these platforms this prevents us from running our SRIOV GPUs in non-SRIOV mode on these platforms.

Outward looking there is no indication that the SBios had any problems and the capability is set.  We have been able to detect these problematic SBios by noticing that they don't initialize our BARs as we expect them to be initialized.

If you have an alternative solution I would love to hear about it.

Thanks,
Kelly


>> Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>> ---
>>  drivers/pci/iov.c | 63
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>>  1 file changed, 60 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>> e30f05c..e4f1405 100644
>> --- a/drivers/pci/iov.c
>> +++ b/drivers/pci/iov.c
>> @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
>>  		msleep(100);
>>  }
>>
>> +/*
>> + * pci_vf_bar_valid - check if VF BARs have resource allocated
>> + * @dev: the PCI device
>> + * @pos: register offset of SR-IOV capability in PCI config space
>> + * Returns true any VF BAR has resource allocated, false
>> + * if all VF BARs are empty.
>> + */
>> +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>> +	int i;
>> +	u32 bar_value;
>> +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
>> +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
>> +			PCI_BASE_ADDRESS_MEM_PREFETCH);
>> +
>> +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>> +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4,
>&bar_value);
>> +		if (bar_value & bar_size_mask)
>> +			return true;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>> +/*
>> + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
>> + * @dev: the PCI device
>> + *
>> + * Returns true if device is an AMD/ATI display adapter,
>> + * otherwise return false.
>> + */
>> +
>> +static bool is_amd_display_adapter(struct pci_dev *dev) {
>> +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
>> +		(dev->vendor == PCI_VENDOR_ID_ATI ||
>> +		dev->vendor == PCI_VENDOR_ID_AMD)); }
>> +
>>  /**
>>   * pci_iov_init - initialize the IOV capability
>>   * @dev: the PCI device
>> @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>>  		return -ENODEV;
>>
>>  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
>> -	if (pos)
>> -		return sriov_init(dev, pos);
>> -
>> +	if (pos) {
>> +	/*
>> +	 * If the device is an AMD graphics device and it supports
>> +	 * SR-IOV it will require a large amount of resources.
>> +	 * Before calling sriov_init() must ensure that the system
>> +	 * BIOS also supports SR-IOV and that system BIOS has been
>> +	 * able to allocate enough resources.
>> +	 * If the VF BARs are zero then the system BIOS does not
>> +	 * support SR-IOV or it could not allocate the resources
>> +	 * and this platform will not support AMD graphics SR-IOV.
>> +	 * Therefore do not call sriov_init().
>> +	 * If the system BIOS does support SR-IOV then the VF BARs
>> +	 * will be properly initialized to non-zero values.
>> +	 */
>> +		if (is_amd_display_adapter(dev)) {
>> +			if (pci_vf_bar_valid(dev, pos))
>> +				return sriov_init(dev, pos);
>> +		} else {
>> +			return sriov_init(dev, pos);
>> +		}
>> +	}
>>  	return -ENODEV;
>>  }
>>

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Zytaruk, Kelly]

>-----Original Message-----
>From: Alex Williamson [mailto:alex.williamson@redhat.com]
>Sent: Thursday, May 11, 2017 11:21 PM
>To: Cheng, Collins
>Cc: Bjorn Helgaas; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org=
;
>Deucher, Alexander; Zytaruk, Kelly
>Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>incapable platform
>
>On Fri, 12 May 2017 02:50:32 +0000
>"Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>
>> Hi Helgaas,
>>
>> Some AMD GPUs have hardware support for graphics SR-IOV.
>> If the SR-IOV capable GPU is plugged into the SR-IOV incapable
>> platform. It would cause a problem on PCI resource allocation in
>> current Linux kernel.
>>
>> Therefore in order to allow the PF (Physical Function) device of
>> SR-IOV capable GPU to work on the SR-IOV incapable platform, it is
>> required to verify conditions for initializing BAR resources on AMD
>> SR-IOV capable GPUs.
>>
>> If the device is an AMD graphics device and it supports SR-IOV it will
>> require a large amount of resources.
>> Before calling sriov_init() must ensure that the system BIOS also
>> supports SR-IOV and that system BIOS has been able to allocate enough
>> resources.
>> If the VF BARs are zero then the system BIOS does not support SR-IOV
>> or it could not allocate the resources and this platform will not
>> support AMD graphics SR-IOV.
>> Therefore do not call sriov_init().
>> If the system BIOS does support SR-IOV then the VF BARs will be
>> properly initialized to non-zero values.
>>
>> Below is the patch against to Kernel 4.8 & 4.9. Please review.
>>
>> I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in
>> quirks.c are for specific devices and one or more device ID are
>> defined for the specific devices. However my patch is for all AMD
>> SR-IOV capable GPUs, that includes all existing and future AMD server GP=
Us.
>> So it doesn't seem like a good fit to put the fix in quirks.c.
>
>
>Why is an AMD graphics card unique here?  Doesn't sriov_init() always need=
 to be
>able to deal with devices of any type where the BIOS hasn't initialized th=
e SR-IOV
>capability?  Some SR-IOV devices can fit their VFs within a minimum bridge
>aperture, most cannot.  I don't understand why the VF resource requirement=
s
>being exceptionally large dictates that they receive special handling.  Th=
anks,
>
>Alex
>

Hi Alex, =20
Many System Bios's are problematic in that they don't know how to fully sup=
port SRIOV.  Up until recently SRIOV devices typically used a small amount =
of resources, such as a NIC.
The AMD SRIOV GPU uses significant resources and many SBios' cannot handle =
this properly.  The faulty SBios will attempt to initialize, run out of res=
ources and not indicate any error.
Even though we are not enabling SRIOV on these platforms this prevents us f=
rom running our SRIOV GPUs in non-SRIOV mode on these platforms.

Outward looking there is no indication that the SBios had any problems and =
the capability is set.  We have been able to detect these problematic SBios=
 by noticing that they don't initialize our BARs as we expect them to be in=
itialized.

If you have an alternative solution I would love to hear about it.

Thanks,
Kelly


>> Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>> ---
>>  drivers/pci/iov.c | 63
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>>  1 file changed, 60 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>> e30f05c..e4f1405 100644
>> --- a/drivers/pci/iov.c
>> +++ b/drivers/pci/iov.c
>> @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev=
)
>>  		msleep(100);
>>  }
>>
>> +/*
>> + * pci_vf_bar_valid - check if VF BARs have resource allocated
>> + * @dev: the PCI device
>> + * @pos: register offset of SR-IOV capability in PCI config space
>> + * Returns true any VF BAR has resource allocated, false
>> + * if all VF BARs are empty.
>> + */
>> +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>> +	int i;
>> +	u32 bar_value;
>> +	u32 bar_size_mask =3D ~(PCI_BASE_ADDRESS_SPACE |
>> +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
>> +			PCI_BASE_ADDRESS_MEM_PREFETCH);
>> +
>> +	for (i =3D 0; i < PCI_SRIOV_NUM_BARS; i++) {
>> +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4,
>&bar_value);
>> +		if (bar_value & bar_size_mask)
>> +			return true;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>> +/*
>> + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
>> + * @dev: the PCI device
>> + *
>> + * Returns true if device is an AMD/ATI display adapter,
>> + * otherwise return false.
>> + */
>> +
>> +static bool is_amd_display_adapter(struct pci_dev *dev) {
>> +	return (((dev->class >> 16) =3D=3D PCI_BASE_CLASS_DISPLAY) &&
>> +		(dev->vendor =3D=3D PCI_VENDOR_ID_ATI ||
>> +		dev->vendor =3D=3D PCI_VENDOR_ID_AMD)); }
>> +
>>  /**
>>   * pci_iov_init - initialize the IOV capability
>>   * @dev: the PCI device
>> @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>>  		return -ENODEV;
>>
>>  	pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
>> -	if (pos)
>> -		return sriov_init(dev, pos);
>> -
>> +	if (pos) {
>> +	/*
>> +	 * If the device is an AMD graphics device and it supports
>> +	 * SR-IOV it will require a large amount of resources.
>> +	 * Before calling sriov_init() must ensure that the system
>> +	 * BIOS also supports SR-IOV and that system BIOS has been
>> +	 * able to allocate enough resources.
>> +	 * If the VF BARs are zero then the system BIOS does not
>> +	 * support SR-IOV or it could not allocate the resources
>> +	 * and this platform will not support AMD graphics SR-IOV.
>> +	 * Therefore do not call sriov_init().
>> +	 * If the system BIOS does support SR-IOV then the VF BARs
>> +	 * will be properly initialized to non-zero values.
>> +	 */
>> +		if (is_amd_display_adapter(dev)) {
>> +			if (pci_vf_bar_valid(dev, pos))
>> +				return sriov_init(dev, pos);
>> +		} else {
>> +			return sriov_init(dev, pos);
>> +		}
>> +	}
>>  	return -ENODEV;
>>  }
>>

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alex Williamson]

On Fri, 12 May 2017 03:42:46 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Williamson,
> 
> GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
> 
> If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.

So the problem is that something bad happens when the kernel is trying
to reallocate resources in order to fulfill the requirements of the
VFs, leaving the PF resources incorrectly programmed?  Why not just fix
that bug rather than creating special handling for this vendor/class of
device which disables any attempt to fixup resources for SR-IOV?  IOW,
this patch just avoids the problem for your devices rather than fixing
the bug.  I'd suggest fixing the bug such that the PF is left in a
functional state if the kernel is unable to allocate sufficient
resources for the VFs.  Thanks,

Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com] 
> Sent: Friday, May 12, 2017 11:21 AM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform
> 
> On Fri, 12 May 2017 02:50:32 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> 
> > Hi Helgaas,
> > 
> > Some AMD GPUs have hardware support for graphics SR-IOV.
> > If the SR-IOV capable GPU is plugged into the SR-IOV incapable 
> > platform. It would cause a problem on PCI resource allocation in 
> > current Linux kernel.
> > 
> > Therefore in order to allow the PF (Physical Function) device of 
> > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is 
> > required to verify conditions for initializing BAR resources on AMD 
> > SR-IOV capable GPUs.
> > 
> > If the device is an AMD graphics device and it supports SR-IOV it will 
> > require a large amount of resources.
> > Before calling sriov_init() must ensure that the system BIOS also 
> > supports SR-IOV and that system BIOS has been able to allocate enough 
> > resources.
> > If the VF BARs are zero then the system BIOS does not support SR-IOV 
> > or it could not allocate the resources and this platform will not 
> > support AMD graphics SR-IOV.
> > Therefore do not call sriov_init().
> > If the system BIOS does support SR-IOV then the VF BARs will be 
> > properly initialized to non-zero values.
> > 
> > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> > 
> > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in 
> > quirks.c are for specific devices and one or more device ID are 
> > defined for the specific devices. However my patch is for all AMD 
> > SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> > So it doesn't seem like a good fit to put the fix in quirks.c.  
> 
> 
> Why is an AMD graphics card unique here?  Doesn't sriov_init() always need to be able to deal with devices of any type where the BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV devices can fit their VFs within a minimum bridge aperture, most cannot.  I don't understand why the VF resource requirements being exceptionally large dictates that they receive special handling.  Thanks,
> 
> Alex
> 
> > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> > ---
> >  drivers/pci/iov.c | 63 
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> >  1 file changed, 60 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 
> > e30f05c..e4f1405 100644
> > --- a/drivers/pci/iov.c
> > +++ b/drivers/pci/iov.c
> > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
> >  		msleep(100);
> >  }
> >  
> > +/*
> > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > + * @dev: the PCI device
> > + * @pos: register offset of SR-IOV capability in PCI config space
> > + * Returns true any VF BAR has resource allocated, false
> > + * if all VF BARs are empty.
> > + */
> > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > +	int i;
> > +	u32 bar_value;
> > +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> > +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> > +
> > +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> > +		if (bar_value & bar_size_mask)
> > +			return true;
> > +	}
> > +
> > +	return false;
> > +}
> > +
> > +/*
> > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> > + * @dev: the PCI device
> > + *
> > + * Returns true if device is an AMD/ATI display adapter,
> > + * otherwise return false.
> > + */
> > +
> > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> > +		(dev->vendor == PCI_VENDOR_ID_ATI ||
> > +		dev->vendor == PCI_VENDOR_ID_AMD)); }
> > +
> >  /**
> >   * pci_iov_init - initialize the IOV capability
> >   * @dev: the PCI device
> > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> >  		return -ENODEV;
> >  
> >  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > -	if (pos)
> > -		return sriov_init(dev, pos);
> > -
> > +	if (pos) {
> > +	/*
> > +	 * If the device is an AMD graphics device and it supports
> > +	 * SR-IOV it will require a large amount of resources.
> > +	 * Before calling sriov_init() must ensure that the system
> > +	 * BIOS also supports SR-IOV and that system BIOS has been
> > +	 * able to allocate enough resources.
> > +	 * If the VF BARs are zero then the system BIOS does not
> > +	 * support SR-IOV or it could not allocate the resources
> > +	 * and this platform will not support AMD graphics SR-IOV.
> > +	 * Therefore do not call sriov_init().
> > +	 * If the system BIOS does support SR-IOV then the VF BARs
> > +	 * will be properly initialized to non-zero values.
> > +	 */
> > +		if (is_amd_display_adapter(dev)) {
> > +			if (pci_vf_bar_valid(dev, pos))
> > +				return sriov_init(dev, pos);
> > +		} else {
> > +			return sriov_init(dev, pos);
> > +		}
> > +	}
> >  	return -ENODEV;
> >  }
> >    
> 

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Williamson,

I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.

I did not try to fix the issue from the kernel resource allocation perspective, it is because:
	1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
	2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.

-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com] 
Sent: Friday, May 12, 2017 12:01 PM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Fri, 12 May 2017 03:42:46 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Williamson,
> 
> GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
> 
> If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.

So the problem is that something bad happens when the kernel is trying to reallocate resources in order to fulfill the requirements of the VFs, leaving the PF resources incorrectly programmed?  Why not just fix that bug rather than creating special handling for this vendor/class of device which disables any attempt to fixup resources for SR-IOV?  IOW, this patch just avoids the problem for your devices rather than fixing the bug.  I'd suggest fixing the bug such that the PF is left in a functional state if the kernel is unable to allocate sufficient resources for the VFs.  Thanks,

Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Friday, May 12, 2017 11:21 AM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> linux-kernel@vger.kernel.org; Deucher, Alexander 
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> SR-IOV incapable platform
> 
> On Fri, 12 May 2017 02:50:32 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> 
> > Hi Helgaas,
> > 
> > Some AMD GPUs have hardware support for graphics SR-IOV.
> > If the SR-IOV capable GPU is plugged into the SR-IOV incapable 
> > platform. It would cause a problem on PCI resource allocation in 
> > current Linux kernel.
> > 
> > Therefore in order to allow the PF (Physical Function) device of 
> > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is 
> > required to verify conditions for initializing BAR resources on AMD 
> > SR-IOV capable GPUs.
> > 
> > If the device is an AMD graphics device and it supports SR-IOV it 
> > will require a large amount of resources.
> > Before calling sriov_init() must ensure that the system BIOS also 
> > supports SR-IOV and that system BIOS has been able to allocate 
> > enough resources.
> > If the VF BARs are zero then the system BIOS does not support SR-IOV 
> > or it could not allocate the resources and this platform will not 
> > support AMD graphics SR-IOV.
> > Therefore do not call sriov_init().
> > If the system BIOS does support SR-IOV then the VF BARs will be 
> > properly initialized to non-zero values.
> > 
> > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> > 
> > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes 
> > in quirks.c are for specific devices and one or more device ID are 
> > defined for the specific devices. However my patch is for all AMD 
> > SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> > So it doesn't seem like a good fit to put the fix in quirks.c.  
> 
> 
> Why is an AMD graphics card unique here?  Doesn't sriov_init() always 
> need to be able to deal with devices of any type where the BIOS hasn't 
> initialized the SR-IOV capability?  Some SR-IOV devices can fit their 
> VFs within a minimum bridge aperture, most cannot.  I don't understand 
> why the VF resource requirements being exceptionally large dictates 
> that they receive special handling.  Thanks,
> 
> Alex
> 
> > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> > ---
> >  drivers/pci/iov.c | 63
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> >  1 file changed, 60 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> > e30f05c..e4f1405 100644
> > --- a/drivers/pci/iov.c
> > +++ b/drivers/pci/iov.c
> > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
> >  		msleep(100);
> >  }
> >  
> > +/*
> > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > + * @dev: the PCI device
> > + * @pos: register offset of SR-IOV capability in PCI config space
> > + * Returns true any VF BAR has resource allocated, false
> > + * if all VF BARs are empty.
> > + */
> > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > +	int i;
> > +	u32 bar_value;
> > +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> > +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> > +
> > +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> > +		if (bar_value & bar_size_mask)
> > +			return true;
> > +	}
> > +
> > +	return false;
> > +}
> > +
> > +/*
> > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> > + * @dev: the PCI device
> > + *
> > + * Returns true if device is an AMD/ATI display adapter,
> > + * otherwise return false.
> > + */
> > +
> > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> > +		(dev->vendor == PCI_VENDOR_ID_ATI ||
> > +		dev->vendor == PCI_VENDOR_ID_AMD)); }
> > +
> >  /**
> >   * pci_iov_init - initialize the IOV capability
> >   * @dev: the PCI device
> > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> >  		return -ENODEV;
> >  
> >  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > -	if (pos)
> > -		return sriov_init(dev, pos);
> > -
> > +	if (pos) {
> > +	/*
> > +	 * If the device is an AMD graphics device and it supports
> > +	 * SR-IOV it will require a large amount of resources.
> > +	 * Before calling sriov_init() must ensure that the system
> > +	 * BIOS also supports SR-IOV and that system BIOS has been
> > +	 * able to allocate enough resources.
> > +	 * If the VF BARs are zero then the system BIOS does not
> > +	 * support SR-IOV or it could not allocate the resources
> > +	 * and this platform will not support AMD graphics SR-IOV.
> > +	 * Therefore do not call sriov_init().
> > +	 * If the system BIOS does support SR-IOV then the VF BARs
> > +	 * will be properly initialized to non-zero values.
> > +	 */
> > +		if (is_amd_display_adapter(dev)) {
> > +			if (pci_vf_bar_valid(dev, pos))
> > +				return sriov_init(dev, pos);
> > +		} else {
> > +			return sriov_init(dev, pos);
> > +		}
> > +	}
> >  	return -ENODEV;
> >  }
> >    
> 

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Williamson,

I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NI=
C. I don't think it is redundant to check the VF BAR valid before call srio=
v_init(), it is safe and saving boot time, also there is no a better method=
 to know if system BIOS has correctly initialized the SR-IOV capability or =
not.

I did not try to fix the issue from the kernel resource allocation perspect=
ive, it is because:
	1. I am not very familiar with the PCI resource allocation scheme in kerne=
l. For example, in sriov_init(), kernel will re-assign the PCI resource for=
 both VF and PF. I don't understand why kernel allocates resource for VF fi=
rstly, then PF. If it is PF firstly, then this issue could be avoided.
	2. I am not sure if kernel has error handler if PCI resource allocation fa=
iled. In this case, kernel cannot allocate enough resource to PF. It should=
 trigger some error handler to either just keep original BAR values set by =
system BIOS, or disable this device and log errors.

-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com]=20
Sent: Friday, May 12, 2017 12:01 PM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-k=
ernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zyta=
ruk, Kelly <Kelly.Zytaruk@amd.com>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV inc=
apable platform

On Fri, 12 May 2017 03:42:46 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Williamson,
>=20
> GPU card needs more BAR aperture resource than other PCI devices. For exa=
mple, Intel SR-IOV network card only require 512KB memory resource for all =
VFs. AMD SR-IOV GPU card needs 256MB x16 VF =3D 4GB memory resource for fra=
me buffer BAR aperture.
>=20
> If the system BIOS supports SR-IOV, it will reserve enough resource for a=
ll VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate th=
e enough resource for VF BARs, only PF BAR will be assigned and VF BARs are=
 empty. Then system boots to Linux kernel and kernel doesn't check if the V=
F BARs are empty or valid. Kernel will re-assign the BAR resources for PF a=
nd all VFs. The problem I saw is that kernel will fail to allocate PF BAR r=
esource because some resources are assigned to VF, this is not expected. So=
 kernel might need to do some check before re-assign the PF/VF resource, so=
 that PF device will be correctly assigned BAR resource and user can use PF=
 device.

So the problem is that something bad happens when the kernel is trying to r=
eallocate resources in order to fulfill the requirements of the VFs, leavin=
g the PF resources incorrectly programmed?  Why not just fix that bug rathe=
r than creating special handling for this vendor/class of device which disa=
bles any attempt to fixup resources for SR-IOV?  IOW, this patch just avoid=
s the problem for your devices rather than fixing the bug.  I'd suggest fix=
ing the bug such that the PF is left in a functional state if the kernel is=
 unable to allocate sufficient resources for the VFs.  Thanks,

Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Friday, May 12, 2017 11:21 AM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;=20
> linux-kernel@vger.kernel.org; Deucher, Alexander=20
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the=20
> SR-IOV incapable platform
>=20
> On Fri, 12 May 2017 02:50:32 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>=20
> > Hi Helgaas,
> >=20
> > Some AMD GPUs have hardware support for graphics SR-IOV.
> > If the SR-IOV capable GPU is plugged into the SR-IOV incapable=20
> > platform. It would cause a problem on PCI resource allocation in=20
> > current Linux kernel.
> >=20
> > Therefore in order to allow the PF (Physical Function) device of=20
> > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is=20
> > required to verify conditions for initializing BAR resources on AMD=20
> > SR-IOV capable GPUs.
> >=20
> > If the device is an AMD graphics device and it supports SR-IOV it=20
> > will require a large amount of resources.
> > Before calling sriov_init() must ensure that the system BIOS also=20
> > supports SR-IOV and that system BIOS has been able to allocate=20
> > enough resources.
> > If the VF BARs are zero then the system BIOS does not support SR-IOV=20
> > or it could not allocate the resources and this platform will not=20
> > support AMD graphics SR-IOV.
> > Therefore do not call sriov_init().
> > If the system BIOS does support SR-IOV then the VF BARs will be=20
> > properly initialized to non-zero values.
> >=20
> > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> >=20
> > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes=20
> > in quirks.c are for specific devices and one or more device ID are=20
> > defined for the specific devices. However my patch is for all AMD=20
> > SR-IOV capable GPUs, that includes all existing and future AMD server G=
PUs.
> > So it doesn't seem like a good fit to put the fix in quirks.c. =20
>=20
>=20
> Why is an AMD graphics card unique here?  Doesn't sriov_init() always=20
> need to be able to deal with devices of any type where the BIOS hasn't=20
> initialized the SR-IOV capability?  Some SR-IOV devices can fit their=20
> VFs within a minimum bridge aperture, most cannot.  I don't understand=20
> why the VF resource requirements being exceptionally large dictates=20
> that they receive special handling.  Thanks,
>=20
> Alex
>=20
> > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> > ---
> >  drivers/pci/iov.c | 63
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> >  1 file changed, 60 insertions(+), 3 deletions(-)
> >=20
> > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> > e30f05c..e4f1405 100644
> > --- a/drivers/pci/iov.c
> > +++ b/drivers/pci/iov.c
> > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *de=
v)
> >  		msleep(100);
> >  }
> > =20
> > +/*
> > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > + * @dev: the PCI device
> > + * @pos: register offset of SR-IOV capability in PCI config space
> > + * Returns true any VF BAR has resource allocated, false
> > + * if all VF BARs are empty.
> > + */
> > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > +	int i;
> > +	u32 bar_value;
> > +	u32 bar_size_mask =3D ~(PCI_BASE_ADDRESS_SPACE |
> > +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> > +
> > +	for (i =3D 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> > +		if (bar_value & bar_size_mask)
> > +			return true;
> > +	}
> > +
> > +	return false;
> > +}
> > +
> > +/*
> > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> > + * @dev: the PCI device
> > + *
> > + * Returns true if device is an AMD/ATI display adapter,
> > + * otherwise return false.
> > + */
> > +
> > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > +	return (((dev->class >> 16) =3D=3D PCI_BASE_CLASS_DISPLAY) &&
> > +		(dev->vendor =3D=3D PCI_VENDOR_ID_ATI ||
> > +		dev->vendor =3D=3D PCI_VENDOR_ID_AMD)); }
> > +
> >  /**
> >   * pci_iov_init - initialize the IOV capability
> >   * @dev: the PCI device
> > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> >  		return -ENODEV;
> > =20
> >  	pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > -	if (pos)
> > -		return sriov_init(dev, pos);
> > -
> > +	if (pos) {
> > +	/*
> > +	 * If the device is an AMD graphics device and it supports
> > +	 * SR-IOV it will require a large amount of resources.
> > +	 * Before calling sriov_init() must ensure that the system
> > +	 * BIOS also supports SR-IOV and that system BIOS has been
> > +	 * able to allocate enough resources.
> > +	 * If the VF BARs are zero then the system BIOS does not
> > +	 * support SR-IOV or it could not allocate the resources
> > +	 * and this platform will not support AMD graphics SR-IOV.
> > +	 * Therefore do not call sriov_init().
> > +	 * If the system BIOS does support SR-IOV then the VF BARs
> > +	 * will be properly initialized to non-zero values.
> > +	 */
> > +		if (is_amd_display_adapter(dev)) {
> > +			if (pci_vf_bar_valid(dev, pos))
> > +				return sriov_init(dev, pos);
> > +		} else {
> > +			return sriov_init(dev, pos);
> > +		}
> > +	}
> >  	return -ENODEV;
> >  }
> >   =20
>=20

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alex Williamson]

On Fri, 12 May 2017 04:51:43 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Williamson,
> 
> I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.

It also masks an underlying bug and creates a maintenance issue that we
won't know when it's safe to remove this workaround.  I don't think
faster boot is valid rationale, in one case SR-IOV is completely
disabled, the other we attempt to allocate the resources the BIOS
failed to provide.  I expect this is also a corner case, the BIOS
should typically support SR-IOV, therefore this situation should be an
exception.
 
> I did not try to fix the issue from the kernel resource allocation perspective, it is because:
> 	1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
> 	2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.

I think these are the issues we should be trying to solve and I'm sure
folks on the linux-pci list can help us identify the bug.  Minimally,
failure to allocate VF resources should leave the device in no worse
condition than before it tried.  Perhaps you could post more details
about the issue, boot with pci=earlydump, post dmesg of a boot where
the PF resources are incorrectly re-allocated, and include lspci -vvv
for the SR-IOV device.  Also, please test with the latest upstream
kernel, upstream only patches old kernels through stable backports of
commits to the latest kernel.  Adding Yinghai as a resource allocation
expert. Thanks,

Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com] 
> Sent: Friday, May 12, 2017 12:01 PM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform
> 
> On Fri, 12 May 2017 03:42:46 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> 
> > Hi Williamson,
> > 
> > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
> > 
> > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.  
> 
> So the problem is that something bad happens when the kernel is trying to reallocate resources in order to fulfill the requirements of the VFs, leaving the PF resources incorrectly programmed?  Why not just fix that bug rather than creating special handling for this vendor/class of device which disables any attempt to fixup resources for SR-IOV?  IOW, this patch just avoids the problem for your devices rather than fixing the bug.  I'd suggest fixing the bug such that the PF is left in a functional state if the kernel is unable to allocate sufficient resources for the VFs.  Thanks,
> 
> Alex
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Friday, May 12, 2017 11:21 AM
> > To: Cheng, Collins <Collins.Cheng@amd.com>
> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> > linux-kernel@vger.kernel.org; Deucher, Alexander 
> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> > SR-IOV incapable platform
> > 
> > On Fri, 12 May 2017 02:50:32 +0000
> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >   
> > > Hi Helgaas,
> > > 
> > > Some AMD GPUs have hardware support for graphics SR-IOV.
> > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable 
> > > platform. It would cause a problem on PCI resource allocation in 
> > > current Linux kernel.
> > > 
> > > Therefore in order to allow the PF (Physical Function) device of 
> > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is 
> > > required to verify conditions for initializing BAR resources on AMD 
> > > SR-IOV capable GPUs.
> > > 
> > > If the device is an AMD graphics device and it supports SR-IOV it 
> > > will require a large amount of resources.
> > > Before calling sriov_init() must ensure that the system BIOS also 
> > > supports SR-IOV and that system BIOS has been able to allocate 
> > > enough resources.
> > > If the VF BARs are zero then the system BIOS does not support SR-IOV 
> > > or it could not allocate the resources and this platform will not 
> > > support AMD graphics SR-IOV.
> > > Therefore do not call sriov_init().
> > > If the system BIOS does support SR-IOV then the VF BARs will be 
> > > properly initialized to non-zero values.
> > > 
> > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> > > 
> > > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes 
> > > in quirks.c are for specific devices and one or more device ID are 
> > > defined for the specific devices. However my patch is for all AMD 
> > > SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> > > So it doesn't seem like a good fit to put the fix in quirks.c.    
> > 
> > 
> > Why is an AMD graphics card unique here?  Doesn't sriov_init() always 
> > need to be able to deal with devices of any type where the BIOS hasn't 
> > initialized the SR-IOV capability?  Some SR-IOV devices can fit their 
> > VFs within a minimum bridge aperture, most cannot.  I don't understand 
> > why the VF resource requirements being exceptionally large dictates 
> > that they receive special handling.  Thanks,
> > 
> > Alex
> >   
> > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> > > ---
> > >  drivers/pci/iov.c | 63
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> > >  1 file changed, 60 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> > > e30f05c..e4f1405 100644
> > > --- a/drivers/pci/iov.c
> > > +++ b/drivers/pci/iov.c
> > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
> > >  		msleep(100);
> > >  }
> > >  
> > > +/*
> > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > > + * @dev: the PCI device
> > > + * @pos: register offset of SR-IOV capability in PCI config space
> > > + * Returns true any VF BAR has resource allocated, false
> > > + * if all VF BARs are empty.
> > > + */
> > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > > +	int i;
> > > +	u32 bar_value;
> > > +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> > > +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > > +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> > > +
> > > +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > > +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> > > +		if (bar_value & bar_size_mask)
> > > +			return true;
> > > +	}
> > > +
> > > +	return false;
> > > +}
> > > +
> > > +/*
> > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> > > + * @dev: the PCI device
> > > + *
> > > + * Returns true if device is an AMD/ATI display adapter,
> > > + * otherwise return false.
> > > + */
> > > +
> > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > > +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> > > +		(dev->vendor == PCI_VENDOR_ID_ATI ||
> > > +		dev->vendor == PCI_VENDOR_ID_AMD)); }
> > > +
> > >  /**
> > >   * pci_iov_init - initialize the IOV capability
> > >   * @dev: the PCI device
> > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> > >  		return -ENODEV;
> > >  
> > >  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > > -	if (pos)
> > > -		return sriov_init(dev, pos);
> > > -
> > > +	if (pos) {
> > > +	/*
> > > +	 * If the device is an AMD graphics device and it supports
> > > +	 * SR-IOV it will require a large amount of resources.
> > > +	 * Before calling sriov_init() must ensure that the system
> > > +	 * BIOS also supports SR-IOV and that system BIOS has been
> > > +	 * able to allocate enough resources.
> > > +	 * If the VF BARs are zero then the system BIOS does not
> > > +	 * support SR-IOV or it could not allocate the resources
> > > +	 * and this platform will not support AMD graphics SR-IOV.
> > > +	 * Therefore do not call sriov_init().
> > > +	 * If the system BIOS does support SR-IOV then the VF BARs
> > > +	 * will be properly initialized to non-zero values.
> > > +	 */
> > > +		if (is_amd_display_adapter(dev)) {
> > > +			if (pci_vf_bar_valid(dev, pos))
> > > +				return sriov_init(dev, pos);
> > > +		} else {
> > > +			return sriov_init(dev, pos);
> > > +		}
> > > +	}
> > >  	return -ENODEV;
> > >  }
> > >      
> >   
> 

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Williamson,

We cannot assume BIOS supports SR-IOV, actually only newer server motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server motherboard BIOS doesn't support SR-IOV. This issue would happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.

I agree that failure to allocate VF resources should leave the device in no worse condition than before it tried. I hope kernel could allocate PF device resource before allocating VF device resource, and keep PF device resource valid and functional if failed to allocate VF device resource.

I will send out dmesg log lspci info tomorrow. Thanks.


-Collins Cheng

-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com] 
Sent: Friday, May 12, 2017 10:43 PM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Fri, 12 May 2017 04:51:43 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Williamson,
> 
> I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.

It also masks an underlying bug and creates a maintenance issue that we won't know when it's safe to remove this workaround.  I don't think faster boot is valid rationale, in one case SR-IOV is completely disabled, the other we attempt to allocate the resources the BIOS failed to provide.  I expect this is also a corner case, the BIOS should typically support SR-IOV, therefore this situation should be an exception.
 
> I did not try to fix the issue from the kernel resource allocation perspective, it is because:
> 	1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
> 	2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.

I think these are the issues we should be trying to solve and I'm sure folks on the linux-pci list can help us identify the bug.  Minimally, failure to allocate VF resources should leave the device in no worse condition than before it tried.  Perhaps you could post more details about the issue, boot with pci=earlydump, post dmesg of a boot where the PF resources are incorrectly re-allocated, and include lspci -vvv for the SR-IOV device.  Also, please test with the latest upstream kernel, upstream only patches old kernels through stable backports of commits to the latest kernel.  Adding Yinghai as a resource allocation expert. Thanks,

Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Friday, May 12, 2017 12:01 PM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> linux-kernel@vger.kernel.org; Deucher, Alexander 
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> SR-IOV incapable platform
> 
> On Fri, 12 May 2017 03:42:46 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> 
> > Hi Williamson,
> > 
> > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
> > 
> > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.  
> 
> So the problem is that something bad happens when the kernel is trying 
> to reallocate resources in order to fulfill the requirements of the 
> VFs, leaving the PF resources incorrectly programmed?  Why not just 
> fix that bug rather than creating special handling for this 
> vendor/class of device which disables any attempt to fixup resources 
> for SR-IOV?  IOW, this patch just avoids the problem for your devices 
> rather than fixing the bug.  I'd suggest fixing the bug such that the 
> PF is left in a functional state if the kernel is unable to allocate 
> sufficient resources for the VFs.  Thanks,
> 
> Alex
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Friday, May 12, 2017 11:21 AM
> > To: Cheng, Collins <Collins.Cheng@amd.com>
> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> > linux-kernel@vger.kernel.org; Deucher, Alexander 
> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> > SR-IOV incapable platform
> > 
> > On Fri, 12 May 2017 02:50:32 +0000
> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >   
> > > Hi Helgaas,
> > > 
> > > Some AMD GPUs have hardware support for graphics SR-IOV.
> > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable 
> > > platform. It would cause a problem on PCI resource allocation in 
> > > current Linux kernel.
> > > 
> > > Therefore in order to allow the PF (Physical Function) device of 
> > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is 
> > > required to verify conditions for initializing BAR resources on 
> > > AMD SR-IOV capable GPUs.
> > > 
> > > If the device is an AMD graphics device and it supports SR-IOV it 
> > > will require a large amount of resources.
> > > Before calling sriov_init() must ensure that the system BIOS also 
> > > supports SR-IOV and that system BIOS has been able to allocate 
> > > enough resources.
> > > If the VF BARs are zero then the system BIOS does not support 
> > > SR-IOV or it could not allocate the resources and this platform 
> > > will not support AMD graphics SR-IOV.
> > > Therefore do not call sriov_init().
> > > If the system BIOS does support SR-IOV then the VF BARs will be 
> > > properly initialized to non-zero values.
> > > 
> > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> > > 
> > > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes 
> > > in quirks.c are for specific devices and one or more device ID are 
> > > defined for the specific devices. However my patch is for all AMD 
> > > SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> > > So it doesn't seem like a good fit to put the fix in quirks.c.    
> > 
> > 
> > Why is an AMD graphics card unique here?  Doesn't sriov_init() 
> > always need to be able to deal with devices of any type where the 
> > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV devices 
> > can fit their VFs within a minimum bridge aperture, most cannot.  I 
> > don't understand why the VF resource requirements being 
> > exceptionally large dictates that they receive special handling.  
> > Thanks,
> > 
> > Alex
> >   
> > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> > > ---
> > >  drivers/pci/iov.c | 63
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> > >  1 file changed, 60 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> > > e30f05c..e4f1405 100644
> > > --- a/drivers/pci/iov.c
> > > +++ b/drivers/pci/iov.c
> > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
> > >  		msleep(100);
> > >  }
> > >  
> > > +/*
> > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > > + * @dev: the PCI device
> > > + * @pos: register offset of SR-IOV capability in PCI config space
> > > + * Returns true any VF BAR has resource allocated, false
> > > + * if all VF BARs are empty.
> > > + */
> > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > > +	int i;
> > > +	u32 bar_value;
> > > +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> > > +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > > +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> > > +
> > > +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > > +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> > > +		if (bar_value & bar_size_mask)
> > > +			return true;
> > > +	}
> > > +
> > > +	return false;
> > > +}
> > > +
> > > +/*
> > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> > > + * @dev: the PCI device
> > > + *
> > > + * Returns true if device is an AMD/ATI display adapter,
> > > + * otherwise return false.
> > > + */
> > > +
> > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > > +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> > > +		(dev->vendor == PCI_VENDOR_ID_ATI ||
> > > +		dev->vendor == PCI_VENDOR_ID_AMD)); }
> > > +
> > >  /**
> > >   * pci_iov_init - initialize the IOV capability
> > >   * @dev: the PCI device
> > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> > >  		return -ENODEV;
> > >  
> > >  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > > -	if (pos)
> > > -		return sriov_init(dev, pos);
> > > -
> > > +	if (pos) {
> > > +	/*
> > > +	 * If the device is an AMD graphics device and it supports
> > > +	 * SR-IOV it will require a large amount of resources.
> > > +	 * Before calling sriov_init() must ensure that the system
> > > +	 * BIOS also supports SR-IOV and that system BIOS has been
> > > +	 * able to allocate enough resources.
> > > +	 * If the VF BARs are zero then the system BIOS does not
> > > +	 * support SR-IOV or it could not allocate the resources
> > > +	 * and this platform will not support AMD graphics SR-IOV.
> > > +	 * Therefore do not call sriov_init().
> > > +	 * If the system BIOS does support SR-IOV then the VF BARs
> > > +	 * will be properly initialized to non-zero values.
> > > +	 */
> > > +		if (is_amd_display_adapter(dev)) {
> > > +			if (pci_vf_bar_valid(dev, pos))
> > > +				return sriov_init(dev, pos);
> > > +		} else {
> > > +			return sriov_init(dev, pos);
> > > +		}
> > > +	}
> > >  	return -ENODEV;
> > >  }
> > >      
> >   
> 

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Williamson,

We cannot assume BIOS supports SR-IOV, actually only newer server motherboa=
rd BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server mo=
therboard BIOS doesn't support SR-IOV. This issue would happen if an user p=
lugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.

I agree that failure to allocate VF resources should leave the device in no=
 worse condition than before it tried. I hope kernel could allocate PF devi=
ce resource before allocating VF device resource, and keep PF device resour=
ce valid and functional if failed to allocate VF device resource.

I will send out dmesg log lspci info tomorrow. Thanks.


-Collins Cheng

-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com]=20
Sent: Friday, May 12, 2017 10:43 PM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-k=
ernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zyta=
ruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV inc=
apable platform

On Fri, 12 May 2017 04:51:43 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Williamson,
>=20
> I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV =
NIC. I don't think it is redundant to check the VF BAR valid before call sr=
iov_init(), it is safe and saving boot time, also there is no a better meth=
od to know if system BIOS has correctly initialized the SR-IOV capability o=
r not.

It also masks an underlying bug and creates a maintenance issue that we won=
't know when it's safe to remove this workaround.  I don't think faster boo=
t is valid rationale, in one case SR-IOV is completely disabled, the other =
we attempt to allocate the resources the BIOS failed to provide.  I expect =
this is also a corner case, the BIOS should typically support SR-IOV, there=
fore this situation should be an exception.
=20
> I did not try to fix the issue from the kernel resource allocation perspe=
ctive, it is because:
> 	1. I am not very familiar with the PCI resource allocation scheme in ker=
nel. For example, in sriov_init(), kernel will re-assign the PCI resource f=
or both VF and PF. I don't understand why kernel allocates resource for VF =
firstly, then PF. If it is PF firstly, then this issue could be avoided.
> 	2. I am not sure if kernel has error handler if PCI resource allocation =
failed. In this case, kernel cannot allocate enough resource to PF. It shou=
ld trigger some error handler to either just keep original BAR values set b=
y system BIOS, or disable this device and log errors.

I think these are the issues we should be trying to solve and I'm sure folk=
s on the linux-pci list can help us identify the bug.  Minimally, failure t=
o allocate VF resources should leave the device in no worse condition than =
before it tried.  Perhaps you could post more details about the issue, boot=
 with pci=3Dearlydump, post dmesg of a boot where the PF resources are inco=
rrectly re-allocated, and include lspci -vvv for the SR-IOV device.  Also, =
please test with the latest upstream kernel, upstream only patches old kern=
els through stable backports of commits to the latest kernel.  Adding Yingh=
ai as a resource allocation expert. Thanks,

Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Friday, May 12, 2017 12:01 PM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;=20
> linux-kernel@vger.kernel.org; Deucher, Alexander=20
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the=20
> SR-IOV incapable platform
>=20
> On Fri, 12 May 2017 03:42:46 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>=20
> > Hi Williamson,
> >=20
> > GPU card needs more BAR aperture resource than other PCI devices. For e=
xample, Intel SR-IOV network card only require 512KB memory resource for al=
l VFs. AMD SR-IOV GPU card needs 256MB x16 VF =3D 4GB memory resource for f=
rame buffer BAR aperture.
> >=20
> > If the system BIOS supports SR-IOV, it will reserve enough resource for=
 all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate =
the enough resource for VF BARs, only PF BAR will be assigned and VF BARs a=
re empty. Then system boots to Linux kernel and kernel doesn't check if the=
 VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF=
 and all VFs. The problem I saw is that kernel will fail to allocate PF BAR=
 resource because some resources are assigned to VF, this is not expected. =
So kernel might need to do some check before re-assign the PF/VF resource, =
so that PF device will be correctly assigned BAR resource and user can use =
PF device. =20
>=20
> So the problem is that something bad happens when the kernel is trying=20
> to reallocate resources in order to fulfill the requirements of the=20
> VFs, leaving the PF resources incorrectly programmed?  Why not just=20
> fix that bug rather than creating special handling for this=20
> vendor/class of device which disables any attempt to fixup resources=20
> for SR-IOV?  IOW, this patch just avoids the problem for your devices=20
> rather than fixing the bug.  I'd suggest fixing the bug such that the=20
> PF is left in a functional state if the kernel is unable to allocate=20
> sufficient resources for the VFs.  Thanks,
>=20
> Alex
>=20
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Friday, May 12, 2017 11:21 AM
> > To: Cheng, Collins <Collins.Cheng@amd.com>
> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;=20
> > linux-kernel@vger.kernel.org; Deucher, Alexander=20
> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the=20
> > SR-IOV incapable platform
> >=20
> > On Fri, 12 May 2017 02:50:32 +0000
> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >  =20
> > > Hi Helgaas,
> > >=20
> > > Some AMD GPUs have hardware support for graphics SR-IOV.
> > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable=20
> > > platform. It would cause a problem on PCI resource allocation in=20
> > > current Linux kernel.
> > >=20
> > > Therefore in order to allow the PF (Physical Function) device of=20
> > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is=20
> > > required to verify conditions for initializing BAR resources on=20
> > > AMD SR-IOV capable GPUs.
> > >=20
> > > If the device is an AMD graphics device and it supports SR-IOV it=20
> > > will require a large amount of resources.
> > > Before calling sriov_init() must ensure that the system BIOS also=20
> > > supports SR-IOV and that system BIOS has been able to allocate=20
> > > enough resources.
> > > If the VF BARs are zero then the system BIOS does not support=20
> > > SR-IOV or it could not allocate the resources and this platform=20
> > > will not support AMD graphics SR-IOV.
> > > Therefore do not call sriov_init().
> > > If the system BIOS does support SR-IOV then the VF BARs will be=20
> > > properly initialized to non-zero values.
> > >=20
> > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> > >=20
> > > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes=20
> > > in quirks.c are for specific devices and one or more device ID are=20
> > > defined for the specific devices. However my patch is for all AMD=20
> > > SR-IOV capable GPUs, that includes all existing and future AMD server=
 GPUs.
> > > So it doesn't seem like a good fit to put the fix in quirks.c.   =20
> >=20
> >=20
> > Why is an AMD graphics card unique here?  Doesn't sriov_init()=20
> > always need to be able to deal with devices of any type where the=20
> > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV devices=20
> > can fit their VFs within a minimum bridge aperture, most cannot.  I=20
> > don't understand why the VF resource requirements being=20
> > exceptionally large dictates that they receive special handling. =20
> > Thanks,
> >=20
> > Alex
> >  =20
> > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> > > ---
> > >  drivers/pci/iov.c | 63
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> > >  1 file changed, 60 insertions(+), 3 deletions(-)
> > >=20
> > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> > > e30f05c..e4f1405 100644
> > > --- a/drivers/pci/iov.c
> > > +++ b/drivers/pci/iov.c
> > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *=
dev)
> > >  		msleep(100);
> > >  }
> > > =20
> > > +/*
> > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > > + * @dev: the PCI device
> > > + * @pos: register offset of SR-IOV capability in PCI config space
> > > + * Returns true any VF BAR has resource allocated, false
> > > + * if all VF BARs are empty.
> > > + */
> > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > > +	int i;
> > > +	u32 bar_value;
> > > +	u32 bar_size_mask =3D ~(PCI_BASE_ADDRESS_SPACE |
> > > +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > > +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> > > +
> > > +	for (i =3D 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > > +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value=
);
> > > +		if (bar_value & bar_size_mask)
> > > +			return true;
> > > +	}
> > > +
> > > +	return false;
> > > +}
> > > +
> > > +/*
> > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> > > + * @dev: the PCI device
> > > + *
> > > + * Returns true if device is an AMD/ATI display adapter,
> > > + * otherwise return false.
> > > + */
> > > +
> > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > > +	return (((dev->class >> 16) =3D=3D PCI_BASE_CLASS_DISPLAY) &&
> > > +		(dev->vendor =3D=3D PCI_VENDOR_ID_ATI ||
> > > +		dev->vendor =3D=3D PCI_VENDOR_ID_AMD)); }
> > > +
> > >  /**
> > >   * pci_iov_init - initialize the IOV capability
> > >   * @dev: the PCI device
> > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> > >  		return -ENODEV;
> > > =20
> > >  	pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > > -	if (pos)
> > > -		return sriov_init(dev, pos);
> > > -
> > > +	if (pos) {
> > > +	/*
> > > +	 * If the device is an AMD graphics device and it supports
> > > +	 * SR-IOV it will require a large amount of resources.
> > > +	 * Before calling sriov_init() must ensure that the system
> > > +	 * BIOS also supports SR-IOV and that system BIOS has been
> > > +	 * able to allocate enough resources.
> > > +	 * If the VF BARs are zero then the system BIOS does not
> > > +	 * support SR-IOV or it could not allocate the resources
> > > +	 * and this platform will not support AMD graphics SR-IOV.
> > > +	 * Therefore do not call sriov_init().
> > > +	 * If the system BIOS does support SR-IOV then the VF BARs
> > > +	 * will be properly initialized to non-zero values.
> > > +	 */
> > > +		if (is_amd_display_adapter(dev)) {
> > > +			if (pci_vf_bar_valid(dev, pos))
> > > +				return sriov_init(dev, pos);
> > > +		} else {
> > > +			return sriov_init(dev, pos);
> > > +		}
> > > +	}
> > >  	return -ENODEV;
> > >  }
> > >     =20
> >  =20
>=20

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alex Williamson]

On Mon, 15 May 2017 08:19:28 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Williamson,
> 
> We cannot assume BIOS supports SR-IOV, actually only newer server motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server motherboard BIOS doesn't support SR-IOV. This issue would happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.

Servers should be supporting SR-IOV for a long time now.  What really
is there to a BIOS supporting SR-IOV anyway, it's simply reserving
sufficient bus number and MMIO resources such that we can enable the
VFs.  This process isn't exclusively reserved for the BIOS.  Some
platforms may choose to only initialize boot devices, leaving the rest
for the OS to program.  The initial proposal here to disable SR-IOV if
not programmed at OS hand-off disables even the possibility of the OS
reallocating resources for this device.

> I agree that failure to allocate VF resources should leave the device in no worse condition than before it tried. I hope kernel could allocate PF device resource before allocating VF device resource, and keep PF device resource valid and functional if failed to allocate VF device resource.
> 
> I will send out dmesg log lspci info tomorrow. Thanks.

Thanks,
Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com] 
> Sent: Friday, May 12, 2017 10:43 PM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform
> 
> On Fri, 12 May 2017 04:51:43 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> 
> > Hi Williamson,
> > 
> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.  
> 
> It also masks an underlying bug and creates a maintenance issue that we won't know when it's safe to remove this workaround.  I don't think faster boot is valid rationale, in one case SR-IOV is completely disabled, the other we attempt to allocate the resources the BIOS failed to provide.  I expect this is also a corner case, the BIOS should typically support SR-IOV, therefore this situation should be an exception.
>  
> > I did not try to fix the issue from the kernel resource allocation perspective, it is because:
> > 	1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
> > 	2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.  
> 
> I think these are the issues we should be trying to solve and I'm sure folks on the linux-pci list can help us identify the bug.  Minimally, failure to allocate VF resources should leave the device in no worse condition than before it tried.  Perhaps you could post more details about the issue, boot with pci=earlydump, post dmesg of a boot where the PF resources are incorrectly re-allocated, and include lspci -vvv for the SR-IOV device.  Also, please test with the latest upstream kernel, upstream only patches old kernels through stable backports of commits to the latest kernel.  Adding Yinghai as a resource allocation expert. Thanks,
> 
> Alex
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Friday, May 12, 2017 12:01 PM
> > To: Cheng, Collins <Collins.Cheng@amd.com>
> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> > linux-kernel@vger.kernel.org; Deucher, Alexander 
> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> > SR-IOV incapable platform
> > 
> > On Fri, 12 May 2017 03:42:46 +0000
> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >   
> > > Hi Williamson,
> > > 
> > > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
> > > 
> > > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.    
> > 
> > So the problem is that something bad happens when the kernel is trying 
> > to reallocate resources in order to fulfill the requirements of the 
> > VFs, leaving the PF resources incorrectly programmed?  Why not just 
> > fix that bug rather than creating special handling for this 
> > vendor/class of device which disables any attempt to fixup resources 
> > for SR-IOV?  IOW, this patch just avoids the problem for your devices 
> > rather than fixing the bug.  I'd suggest fixing the bug such that the 
> > PF is left in a functional state if the kernel is unable to allocate 
> > sufficient resources for the VFs.  Thanks,
> > 
> > Alex
> >   
> > > -----Original Message-----
> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > > Sent: Friday, May 12, 2017 11:21 AM
> > > To: Cheng, Collins <Collins.Cheng@amd.com>
> > > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> > > linux-kernel@vger.kernel.org; Deucher, Alexander 
> > > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> > > SR-IOV incapable platform
> > > 
> > > On Fri, 12 May 2017 02:50:32 +0000
> > > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> > >     
> > > > Hi Helgaas,
> > > > 
> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable 
> > > > platform. It would cause a problem on PCI resource allocation in 
> > > > current Linux kernel.
> > > > 
> > > > Therefore in order to allow the PF (Physical Function) device of 
> > > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is 
> > > > required to verify conditions for initializing BAR resources on 
> > > > AMD SR-IOV capable GPUs.
> > > > 
> > > > If the device is an AMD graphics device and it supports SR-IOV it 
> > > > will require a large amount of resources.
> > > > Before calling sriov_init() must ensure that the system BIOS also 
> > > > supports SR-IOV and that system BIOS has been able to allocate 
> > > > enough resources.
> > > > If the VF BARs are zero then the system BIOS does not support 
> > > > SR-IOV or it could not allocate the resources and this platform 
> > > > will not support AMD graphics SR-IOV.
> > > > Therefore do not call sriov_init().
> > > > If the system BIOS does support SR-IOV then the VF BARs will be 
> > > > properly initialized to non-zero values.
> > > > 
> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> > > > 
> > > > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes 
> > > > in quirks.c are for specific devices and one or more device ID are 
> > > > defined for the specific devices. However my patch is for all AMD 
> > > > SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> > > > So it doesn't seem like a good fit to put the fix in quirks.c.      
> > > 
> > > 
> > > Why is an AMD graphics card unique here?  Doesn't sriov_init() 
> > > always need to be able to deal with devices of any type where the 
> > > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV devices 
> > > can fit their VFs within a minimum bridge aperture, most cannot.  I 
> > > don't understand why the VF resource requirements being 
> > > exceptionally large dictates that they receive special handling.  
> > > Thanks,
> > > 
> > > Alex
> > >     
> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> > > > ---
> > > >  drivers/pci/iov.c | 63
> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> > > > e30f05c..e4f1405 100644
> > > > --- a/drivers/pci/iov.c
> > > > +++ b/drivers/pci/iov.c
> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
> > > >  		msleep(100);
> > > >  }
> > > >  
> > > > +/*
> > > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > > > + * @dev: the PCI device
> > > > + * @pos: register offset of SR-IOV capability in PCI config space
> > > > + * Returns true any VF BAR has resource allocated, false
> > > > + * if all VF BARs are empty.
> > > > + */
> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > > > +	int i;
> > > > +	u32 bar_value;
> > > > +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> > > > +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > > > +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> > > > +
> > > > +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > > > +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> > > > +		if (bar_value & bar_size_mask)
> > > > +			return true;
> > > > +	}
> > > > +
> > > > +	return false;
> > > > +}
> > > > +
> > > > +/*
> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> > > > + * @dev: the PCI device
> > > > + *
> > > > + * Returns true if device is an AMD/ATI display adapter,
> > > > + * otherwise return false.
> > > > + */
> > > > +
> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > > > +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> > > > +		(dev->vendor == PCI_VENDOR_ID_ATI ||
> > > > +		dev->vendor == PCI_VENDOR_ID_AMD)); }
> > > > +
> > > >  /**
> > > >   * pci_iov_init - initialize the IOV capability
> > > >   * @dev: the PCI device
> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> > > >  		return -ENODEV;
> > > >  
> > > >  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > > > -	if (pos)
> > > > -		return sriov_init(dev, pos);
> > > > -
> > > > +	if (pos) {
> > > > +	/*
> > > > +	 * If the device is an AMD graphics device and it supports
> > > > +	 * SR-IOV it will require a large amount of resources.
> > > > +	 * Before calling sriov_init() must ensure that the system
> > > > +	 * BIOS also supports SR-IOV and that system BIOS has been
> > > > +	 * able to allocate enough resources.
> > > > +	 * If the VF BARs are zero then the system BIOS does not
> > > > +	 * support SR-IOV or it could not allocate the resources
> > > > +	 * and this platform will not support AMD graphics SR-IOV.
> > > > +	 * Therefore do not call sriov_init().
> > > > +	 * If the system BIOS does support SR-IOV then the VF BARs
> > > > +	 * will be properly initialized to non-zero values.
> > > > +	 */
> > > > +		if (is_amd_display_adapter(dev)) {
> > > > +			if (pci_vf_bar_valid(dev, pos))
> > > > +				return sriov_init(dev, pos);
> > > > +		} else {
> > > > +			return sriov_init(dev, pos);
> > > > +		}
> > > > +	}
> > > >  	return -ENODEV;
> > > >  }
> > > >        
> > >     
> >   
> 

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Williamson,

Sorry I am busy on other task, I will look if I can get the dmesg log tomorrow.

My submitted patch is try to disable the possibility of the OS reallocating resources for VF device in sriov_init(), if VF BAR is empty at BIOS/OS hand-off.

-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com] 
Sent: Tuesday, May 16, 2017 1:54 AM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Mon, 15 May 2017 08:19:28 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Williamson,
> 
> We cannot assume BIOS supports SR-IOV, actually only newer server motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server motherboard BIOS doesn't support SR-IOV. This issue would happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.

Servers should be supporting SR-IOV for a long time now.  What really is there to a BIOS supporting SR-IOV anyway, it's simply reserving sufficient bus number and MMIO resources such that we can enable the VFs.  This process isn't exclusively reserved for the BIOS.  Some platforms may choose to only initialize boot devices, leaving the rest for the OS to program.  The initial proposal here to disable SR-IOV if not programmed at OS hand-off disables even the possibility of the OS reallocating resources for this device.

> I agree that failure to allocate VF resources should leave the device in no worse condition than before it tried. I hope kernel could allocate PF device resource before allocating VF device resource, and keep PF device resource valid and functional if failed to allocate VF device resource.
> 
> I will send out dmesg log lspci info tomorrow. Thanks.

Thanks,
Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Friday, May 12, 2017 10:43 PM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> linux-kernel@vger.kernel.org; Deucher, Alexander 
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; 
> Yinghai Lu <yinghai@kernel.org>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> SR-IOV incapable platform
> 
> On Fri, 12 May 2017 04:51:43 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> 
> > Hi Williamson,
> > 
> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.  
> 
> It also masks an underlying bug and creates a maintenance issue that we won't know when it's safe to remove this workaround.  I don't think faster boot is valid rationale, in one case SR-IOV is completely disabled, the other we attempt to allocate the resources the BIOS failed to provide.  I expect this is also a corner case, the BIOS should typically support SR-IOV, therefore this situation should be an exception.
>  
> > I did not try to fix the issue from the kernel resource allocation perspective, it is because:
> > 	1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
> > 	2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.  
> 
> I think these are the issues we should be trying to solve and I'm sure 
> folks on the linux-pci list can help us identify the bug.  Minimally, 
> failure to allocate VF resources should leave the device in no worse 
> condition than before it tried.  Perhaps you could post more details 
> about the issue, boot with pci=earlydump, post dmesg of a boot where 
> the PF resources are incorrectly re-allocated, and include lspci -vvv 
> for the SR-IOV device.  Also, please test with the latest upstream 
> kernel, upstream only patches old kernels through stable backports of 
> commits to the latest kernel.  Adding Yinghai as a resource allocation 
> expert. Thanks,
> 
> Alex
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Friday, May 12, 2017 12:01 PM
> > To: Cheng, Collins <Collins.Cheng@amd.com>
> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> > linux-kernel@vger.kernel.org; Deucher, Alexander 
> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> > SR-IOV incapable platform
> > 
> > On Fri, 12 May 2017 03:42:46 +0000
> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >   
> > > Hi Williamson,
> > > 
> > > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
> > > 
> > > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.    
> > 
> > So the problem is that something bad happens when the kernel is 
> > trying to reallocate resources in order to fulfill the requirements 
> > of the VFs, leaving the PF resources incorrectly programmed?  Why 
> > not just fix that bug rather than creating special handling for this 
> > vendor/class of device which disables any attempt to fixup resources 
> > for SR-IOV?  IOW, this patch just avoids the problem for your 
> > devices rather than fixing the bug.  I'd suggest fixing the bug such 
> > that the PF is left in a functional state if the kernel is unable to 
> > allocate sufficient resources for the VFs.  Thanks,
> > 
> > Alex
> >   
> > > -----Original Message-----
> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > > Sent: Friday, May 12, 2017 11:21 AM
> > > To: Cheng, Collins <Collins.Cheng@amd.com>
> > > Cc: Bjorn Helgaas <bhelgaas@google.com>; 
> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, 
> > > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly 
> > > <Kelly.Zytaruk@amd.com>
> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> > > SR-IOV incapable platform
> > > 
> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins" 
> > > <Collins.Cheng@amd.com> wrote:
> > >     
> > > > Hi Helgaas,
> > > > 
> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable 
> > > > platform. It would cause a problem on PCI resource allocation in 
> > > > current Linux kernel.
> > > > 
> > > > Therefore in order to allow the PF (Physical Function) device of 
> > > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it 
> > > > is required to verify conditions for initializing BAR resources 
> > > > on AMD SR-IOV capable GPUs.
> > > > 
> > > > If the device is an AMD graphics device and it supports SR-IOV 
> > > > it will require a large amount of resources.
> > > > Before calling sriov_init() must ensure that the system BIOS 
> > > > also supports SR-IOV and that system BIOS has been able to 
> > > > allocate enough resources.
> > > > If the VF BARs are zero then the system BIOS does not support 
> > > > SR-IOV or it could not allocate the resources and this platform 
> > > > will not support AMD graphics SR-IOV.
> > > > Therefore do not call sriov_init().
> > > > If the system BIOS does support SR-IOV then the VF BARs will be 
> > > > properly initialized to non-zero values.
> > > > 
> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> > > > 
> > > > I checked the drivers/pci/quirks.c, it looks the 
> > > > workarounds/fixes in quirks.c are for specific devices and one 
> > > > or more device ID are defined for the specific devices. However 
> > > > my patch is for all AMD SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> > > > So it doesn't seem like a good fit to put the fix in quirks.c.      
> > > 
> > > 
> > > Why is an AMD graphics card unique here?  Doesn't sriov_init() 
> > > always need to be able to deal with devices of any type where the 
> > > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV 
> > > devices can fit their VFs within a minimum bridge aperture, most 
> > > cannot.  I don't understand why the VF resource requirements being 
> > > exceptionally large dictates that they receive special handling.
> > > Thanks,
> > > 
> > > Alex
> > >     
> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> > > > ---
> > > >  drivers/pci/iov.c | 63
> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> > > > e30f05c..e4f1405 100644
> > > > --- a/drivers/pci/iov.c
> > > > +++ b/drivers/pci/iov.c
> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
> > > >  		msleep(100);
> > > >  }
> > > >  
> > > > +/*
> > > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > > > + * @dev: the PCI device
> > > > + * @pos: register offset of SR-IOV capability in PCI config 
> > > > +space
> > > > + * Returns true any VF BAR has resource allocated, false
> > > > + * if all VF BARs are empty.
> > > > + */
> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > > > +	int i;
> > > > +	u32 bar_value;
> > > > +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> > > > +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > > > +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> > > > +
> > > > +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > > > +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> > > > +		if (bar_value & bar_size_mask)
> > > > +			return true;
> > > > +	}
> > > > +
> > > > +	return false;
> > > > +}
> > > > +
> > > > +/*
> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU 
> > > > +device
> > > > + * @dev: the PCI device
> > > > + *
> > > > + * Returns true if device is an AMD/ATI display adapter,
> > > > + * otherwise return false.
> > > > + */
> > > > +
> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > > > +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> > > > +		(dev->vendor == PCI_VENDOR_ID_ATI ||
> > > > +		dev->vendor == PCI_VENDOR_ID_AMD)); }
> > > > +
> > > >  /**
> > > >   * pci_iov_init - initialize the IOV capability
> > > >   * @dev: the PCI device
> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> > > >  		return -ENODEV;
> > > >  
> > > >  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > > > -	if (pos)
> > > > -		return sriov_init(dev, pos);
> > > > -
> > > > +	if (pos) {
> > > > +	/*
> > > > +	 * If the device is an AMD graphics device and it supports
> > > > +	 * SR-IOV it will require a large amount of resources.
> > > > +	 * Before calling sriov_init() must ensure that the system
> > > > +	 * BIOS also supports SR-IOV and that system BIOS has been
> > > > +	 * able to allocate enough resources.
> > > > +	 * If the VF BARs are zero then the system BIOS does not
> > > > +	 * support SR-IOV or it could not allocate the resources
> > > > +	 * and this platform will not support AMD graphics SR-IOV.
> > > > +	 * Therefore do not call sriov_init().
> > > > +	 * If the system BIOS does support SR-IOV then the VF BARs
> > > > +	 * will be properly initialized to non-zero values.
> > > > +	 */
> > > > +		if (is_amd_display_adapter(dev)) {
> > > > +			if (pci_vf_bar_valid(dev, pos))
> > > > +				return sriov_init(dev, pos);
> > > > +		} else {
> > > > +			return sriov_init(dev, pos);
> > > > +		}
> > > > +	}
> > > >  	return -ENODEV;
> > > >  }
> > > >        
> > >     
> >   
> 

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Williamson,

Sorry I am busy on other task, I will look if I can get the dmesg log tomor=
row.

My submitted patch is try to disable the possibility of the OS reallocating=
 resources for VF device in sriov_init(), if VF BAR is empty at BIOS/OS han=
d-off.

-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com]=20
Sent: Tuesday, May 16, 2017 1:54 AM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-k=
ernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zyta=
ruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV inc=
apable platform

On Mon, 15 May 2017 08:19:28 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Williamson,
>=20
> We cannot assume BIOS supports SR-IOV, actually only newer server motherb=
oard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server =
motherboard BIOS doesn't support SR-IOV. This issue would happen if an user=
 plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.

Servers should be supporting SR-IOV for a long time now.  What really is th=
ere to a BIOS supporting SR-IOV anyway, it's simply reserving sufficient bu=
s number and MMIO resources such that we can enable the VFs.  This process =
isn't exclusively reserved for the BIOS.  Some platforms may choose to only=
 initialize boot devices, leaving the rest for the OS to program.  The init=
ial proposal here to disable SR-IOV if not programmed at OS hand-off disabl=
es even the possibility of the OS reallocating resources for this device.

> I agree that failure to allocate VF resources should leave the device in =
no worse condition than before it tried. I hope kernel could allocate PF de=
vice resource before allocating VF device resource, and keep PF device reso=
urce valid and functional if failed to allocate VF device resource.
>=20
> I will send out dmesg log lspci info tomorrow. Thanks.

Thanks,
Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Friday, May 12, 2017 10:43 PM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;=20
> linux-kernel@vger.kernel.org; Deucher, Alexander=20
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>;=20
> Yinghai Lu <yinghai@kernel.org>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the=20
> SR-IOV incapable platform
>=20
> On Fri, 12 May 2017 04:51:43 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>=20
> > Hi Williamson,
> >=20
> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IO=
V NIC. I don't think it is redundant to check the VF BAR valid before call =
sriov_init(), it is safe and saving boot time, also there is no a better me=
thod to know if system BIOS has correctly initialized the SR-IOV capability=
 or not. =20
>=20
> It also masks an underlying bug and creates a maintenance issue that we w=
on't know when it's safe to remove this workaround.  I don't think faster b=
oot is valid rationale, in one case SR-IOV is completely disabled, the othe=
r we attempt to allocate the resources the BIOS failed to provide.  I expec=
t this is also a corner case, the BIOS should typically support SR-IOV, the=
refore this situation should be an exception.
> =20
> > I did not try to fix the issue from the kernel resource allocation pers=
pective, it is because:
> > 	1. I am not very familiar with the PCI resource allocation scheme in k=
ernel. For example, in sriov_init(), kernel will re-assign the PCI resource=
 for both VF and PF. I don't understand why kernel allocates resource for V=
F firstly, then PF. If it is PF firstly, then this issue could be avoided.
> > 	2. I am not sure if kernel has error handler if PCI resource allocatio=
n failed. In this case, kernel cannot allocate enough resource to PF. It sh=
ould trigger some error handler to either just keep original BAR values set=
 by system BIOS, or disable this device and log errors. =20
>=20
> I think these are the issues we should be trying to solve and I'm sure=20
> folks on the linux-pci list can help us identify the bug.  Minimally,=20
> failure to allocate VF resources should leave the device in no worse=20
> condition than before it tried.  Perhaps you could post more details=20
> about the issue, boot with pci=3Dearlydump, post dmesg of a boot where=20
> the PF resources are incorrectly re-allocated, and include lspci -vvv=20
> for the SR-IOV device.  Also, please test with the latest upstream=20
> kernel, upstream only patches old kernels through stable backports of=20
> commits to the latest kernel.  Adding Yinghai as a resource allocation=20
> expert. Thanks,
>=20
> Alex
>=20
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Friday, May 12, 2017 12:01 PM
> > To: Cheng, Collins <Collins.Cheng@amd.com>
> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;=20
> > linux-kernel@vger.kernel.org; Deucher, Alexander=20
> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the=20
> > SR-IOV incapable platform
> >=20
> > On Fri, 12 May 2017 03:42:46 +0000
> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >  =20
> > > Hi Williamson,
> > >=20
> > > GPU card needs more BAR aperture resource than other PCI devices. For=
 example, Intel SR-IOV network card only require 512KB memory resource for =
all VFs. AMD SR-IOV GPU card needs 256MB x16 VF =3D 4GB memory resource for=
 frame buffer BAR aperture.
> > >=20
> > > If the system BIOS supports SR-IOV, it will reserve enough resource f=
or all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocat=
e the enough resource for VF BARs, only PF BAR will be assigned and VF BARs=
 are empty. Then system boots to Linux kernel and kernel doesn't check if t=
he VF BARs are empty or valid. Kernel will re-assign the BAR resources for =
PF and all VFs. The problem I saw is that kernel will fail to allocate PF B=
AR resource because some resources are assigned to VF, this is not expected=
. So kernel might need to do some check before re-assign the PF/VF resource=
, so that PF device will be correctly assigned BAR resource and user can us=
e PF device.   =20
> >=20
> > So the problem is that something bad happens when the kernel is=20
> > trying to reallocate resources in order to fulfill the requirements=20
> > of the VFs, leaving the PF resources incorrectly programmed?  Why=20
> > not just fix that bug rather than creating special handling for this=20
> > vendor/class of device which disables any attempt to fixup resources=20
> > for SR-IOV?  IOW, this patch just avoids the problem for your=20
> > devices rather than fixing the bug.  I'd suggest fixing the bug such=20
> > that the PF is left in a functional state if the kernel is unable to=20
> > allocate sufficient resources for the VFs.  Thanks,
> >=20
> > Alex
> >  =20
> > > -----Original Message-----
> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > > Sent: Friday, May 12, 2017 11:21 AM
> > > To: Cheng, Collins <Collins.Cheng@amd.com>
> > > Cc: Bjorn Helgaas <bhelgaas@google.com>;=20
> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher,=20
> > > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly=20
> > > <Kelly.Zytaruk@amd.com>
> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the=20
> > > SR-IOV incapable platform
> > >=20
> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins"=20
> > > <Collins.Cheng@amd.com> wrote:
> > >    =20
> > > > Hi Helgaas,
> > > >=20
> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable=20
> > > > platform. It would cause a problem on PCI resource allocation in=20
> > > > current Linux kernel.
> > > >=20
> > > > Therefore in order to allow the PF (Physical Function) device of=20
> > > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it=20
> > > > is required to verify conditions for initializing BAR resources=20
> > > > on AMD SR-IOV capable GPUs.
> > > >=20
> > > > If the device is an AMD graphics device and it supports SR-IOV=20
> > > > it will require a large amount of resources.
> > > > Before calling sriov_init() must ensure that the system BIOS=20
> > > > also supports SR-IOV and that system BIOS has been able to=20
> > > > allocate enough resources.
> > > > If the VF BARs are zero then the system BIOS does not support=20
> > > > SR-IOV or it could not allocate the resources and this platform=20
> > > > will not support AMD graphics SR-IOV.
> > > > Therefore do not call sriov_init().
> > > > If the system BIOS does support SR-IOV then the VF BARs will be=20
> > > > properly initialized to non-zero values.
> > > >=20
> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> > > >=20
> > > > I checked the drivers/pci/quirks.c, it looks the=20
> > > > workarounds/fixes in quirks.c are for specific devices and one=20
> > > > or more device ID are defined for the specific devices. However=20
> > > > my patch is for all AMD SR-IOV capable GPUs, that includes all exis=
ting and future AMD server GPUs.
> > > > So it doesn't seem like a good fit to put the fix in quirks.c.     =
=20
> > >=20
> > >=20
> > > Why is an AMD graphics card unique here?  Doesn't sriov_init()=20
> > > always need to be able to deal with devices of any type where the=20
> > > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV=20
> > > devices can fit their VFs within a minimum bridge aperture, most=20
> > > cannot.  I don't understand why the VF resource requirements being=20
> > > exceptionally large dictates that they receive special handling.
> > > Thanks,
> > >=20
> > > Alex
> > >    =20
> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> > > > ---
> > > >  drivers/pci/iov.c | 63
> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
> > > >=20
> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> > > > e30f05c..e4f1405 100644
> > > > --- a/drivers/pci/iov.c
> > > > +++ b/drivers/pci/iov.c
> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev=
 *dev)
> > > >  		msleep(100);
> > > >  }
> > > > =20
> > > > +/*
> > > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > > > + * @dev: the PCI device
> > > > + * @pos: register offset of SR-IOV capability in PCI config=20
> > > > +space
> > > > + * Returns true any VF BAR has resource allocated, false
> > > > + * if all VF BARs are empty.
> > > > + */
> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > > > +	int i;
> > > > +	u32 bar_value;
> > > > +	u32 bar_size_mask =3D ~(PCI_BASE_ADDRESS_SPACE |
> > > > +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > > > +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> > > > +
> > > > +	for (i =3D 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > > > +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_val=
ue);
> > > > +		if (bar_value & bar_size_mask)
> > > > +			return true;
> > > > +	}
> > > > +
> > > > +	return false;
> > > > +}
> > > > +
> > > > +/*
> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU=20
> > > > +device
> > > > + * @dev: the PCI device
> > > > + *
> > > > + * Returns true if device is an AMD/ATI display adapter,
> > > > + * otherwise return false.
> > > > + */
> > > > +
> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > > > +	return (((dev->class >> 16) =3D=3D PCI_BASE_CLASS_DISPLAY) &&
> > > > +		(dev->vendor =3D=3D PCI_VENDOR_ID_ATI ||
> > > > +		dev->vendor =3D=3D PCI_VENDOR_ID_AMD)); }
> > > > +
> > > >  /**
> > > >   * pci_iov_init - initialize the IOV capability
> > > >   * @dev: the PCI device
> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> > > >  		return -ENODEV;
> > > > =20
> > > >  	pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > > > -	if (pos)
> > > > -		return sriov_init(dev, pos);
> > > > -
> > > > +	if (pos) {
> > > > +	/*
> > > > +	 * If the device is an AMD graphics device and it supports
> > > > +	 * SR-IOV it will require a large amount of resources.
> > > > +	 * Before calling sriov_init() must ensure that the system
> > > > +	 * BIOS also supports SR-IOV and that system BIOS has been
> > > > +	 * able to allocate enough resources.
> > > > +	 * If the VF BARs are zero then the system BIOS does not
> > > > +	 * support SR-IOV or it could not allocate the resources
> > > > +	 * and this platform will not support AMD graphics SR-IOV.
> > > > +	 * Therefore do not call sriov_init().
> > > > +	 * If the system BIOS does support SR-IOV then the VF BARs
> > > > +	 * will be properly initialized to non-zero values.
> > > > +	 */
> > > > +		if (is_amd_display_adapter(dev)) {
> > > > +			if (pci_vf_bar_valid(dev, pos))
> > > > +				return sriov_init(dev, pos);
> > > > +		} else {
> > > > +			return sriov_init(dev, pos);
> > > > +		}
> > > > +	}
> > > >  	return -ENODEV;
> > > >  }
> > > >       =20
> > >    =20
> >  =20
>=20

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alexander Duyck]

On Mon, May 15, 2017 at 10:53 AM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Mon, 15 May 2017 08:19:28 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>
>> Hi Williamson,
>>
>> We cannot assume BIOS supports SR-IOV, actually only newer server motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server motherboard BIOS doesn't support SR-IOV. This issue would happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.
>
> Servers should be supporting SR-IOV for a long time now.  What really
> is there to a BIOS supporting SR-IOV anyway, it's simply reserving
> sufficient bus number and MMIO resources such that we can enable the
> VFs.  This process isn't exclusively reserved for the BIOS.  Some
> platforms may choose to only initialize boot devices, leaving the rest
> for the OS to program.  The initial proposal here to disable SR-IOV if
> not programmed at OS hand-off disables even the possibility of the OS
> reallocating resources for this device.

There are differences between supporting SR-IOV and supporting SR-IOV
on devices with massive resources. I know I have seen NICs that will
keep a system from completing POST if SR-IOV is enabled, and MMIO
beyond 4G is not. My guess would be that the issues being seen are
probably that they disable SR-IOV in the BIOS in such a setup and end
up running into issues when they try to boot into the Linux kernel as
it goes through and tries to allocate resources for SR-IOV even though
it was disabled in the BIOS.

It might make sense to add a kernel parameter something like a
"pci=nosriov" that would allow for disabling SR-IOV and related
resource allocation if that is what we are talking about. That way you
could plug in these types of devices into a system with a legacy bios
or that doesn't wan to allocate addresses above 32b for MMIO, and this
parameter would be all that is needed to disable SR-IOV so you could
plug in a NIC that has SR-IOV associated with it.

>> I agree that failure to allocate VF resources should leave the device in no worse condition than before it tried. I hope kernel could allocate PF device resource before allocating VF device resource, and keep PF device resource valid and functional if failed to allocate VF device resource.
>>
>> I will send out dmesg log lspci info tomorrow. Thanks.
>
> Thanks,
> Alex
>
>> -----Original Message-----
>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> Sent: Friday, May 12, 2017 10:43 PM
>> To: Cheng, Collins <Collins.Cheng@amd.com>
>> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
>> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform
>>
>> On Fri, 12 May 2017 04:51:43 +0000
>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>
>> > Hi Williamson,
>> >
>> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.
>>
>> It also masks an underlying bug and creates a maintenance issue that we won't know when it's safe to remove this workaround.  I don't think faster boot is valid rationale, in one case SR-IOV is completely disabled, the other we attempt to allocate the resources the BIOS failed to provide.  I expect this is also a corner case, the BIOS should typically support SR-IOV, therefore this situation should be an exception.
>>
>> > I did not try to fix the issue from the kernel resource allocation perspective, it is because:
>> >     1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
>> >     2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.
>>
>> I think these are the issues we should be trying to solve and I'm sure folks on the linux-pci list can help us identify the bug.  Minimally, failure to allocate VF resources should leave the device in no worse condition than before it tried.  Perhaps you could post more details about the issue, boot with pci=earlydump, post dmesg of a boot where the PF resources are incorrectly re-allocated, and include lspci -vvv for the SR-IOV device.  Also, please test with the latest upstream kernel, upstream only patches old kernels through stable backports of commits to the latest kernel.  Adding Yinghai as a resource allocation expert. Thanks,
>>
>> Alex
>>
>> > -----Original Message-----
>> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> > Sent: Friday, May 12, 2017 12:01 PM
>> > To: Cheng, Collins <Collins.Cheng@amd.com>
>> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>> > linux-kernel@vger.kernel.org; Deucher, Alexander
>> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
>> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>> > SR-IOV incapable platform
>> >
>> > On Fri, 12 May 2017 03:42:46 +0000
>> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>> >
>> > > Hi Williamson,
>> > >
>> > > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
>> > >
>> > > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.
>> >
>> > So the problem is that something bad happens when the kernel is trying
>> > to reallocate resources in order to fulfill the requirements of the
>> > VFs, leaving the PF resources incorrectly programmed?  Why not just
>> > fix that bug rather than creating special handling for this
>> > vendor/class of device which disables any attempt to fixup resources
>> > for SR-IOV?  IOW, this patch just avoids the problem for your devices
>> > rather than fixing the bug.  I'd suggest fixing the bug such that the
>> > PF is left in a functional state if the kernel is unable to allocate
>> > sufficient resources for the VFs.  Thanks,
>> >
>> > Alex
>> >
>> > > -----Original Message-----
>> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> > > Sent: Friday, May 12, 2017 11:21 AM
>> > > To: Cheng, Collins <Collins.Cheng@amd.com>
>> > > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>> > > linux-kernel@vger.kernel.org; Deucher, Alexander
>> > > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
>> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>> > > SR-IOV incapable platform
>> > >
>> > > On Fri, 12 May 2017 02:50:32 +0000
>> > > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>> > >
>> > > > Hi Helgaas,
>> > > >
>> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
>> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable
>> > > > platform. It would cause a problem on PCI resource allocation in
>> > > > current Linux kernel.
>> > > >
>> > > > Therefore in order to allow the PF (Physical Function) device of
>> > > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is
>> > > > required to verify conditions for initializing BAR resources on
>> > > > AMD SR-IOV capable GPUs.
>> > > >
>> > > > If the device is an AMD graphics device and it supports SR-IOV it
>> > > > will require a large amount of resources.
>> > > > Before calling sriov_init() must ensure that the system BIOS also
>> > > > supports SR-IOV and that system BIOS has been able to allocate
>> > > > enough resources.
>> > > > If the VF BARs are zero then the system BIOS does not support
>> > > > SR-IOV or it could not allocate the resources and this platform
>> > > > will not support AMD graphics SR-IOV.
>> > > > Therefore do not call sriov_init().
>> > > > If the system BIOS does support SR-IOV then the VF BARs will be
>> > > > properly initialized to non-zero values.
>> > > >
>> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
>> > > >
>> > > > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes
>> > > > in quirks.c are for specific devices and one or more device ID are
>> > > > defined for the specific devices. However my patch is for all AMD
>> > > > SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
>> > > > So it doesn't seem like a good fit to put the fix in quirks.c.
>> > >
>> > >
>> > > Why is an AMD graphics card unique here?  Doesn't sriov_init()
>> > > always need to be able to deal with devices of any type where the
>> > > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV devices
>> > > can fit their VFs within a minimum bridge aperture, most cannot.  I
>> > > don't understand why the VF resource requirements being
>> > > exceptionally large dictates that they receive special handling.
>> > > Thanks,
>> > >
>> > > Alex
>> > >
>> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>> > > > ---
>> > > >  drivers/pci/iov.c | 63
>> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
>> > > >
>> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>> > > > e30f05c..e4f1405 100644
>> > > > --- a/drivers/pci/iov.c
>> > > > +++ b/drivers/pci/iov.c
>> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
>> > > >                 msleep(100);
>> > > >  }
>> > > >
>> > > > +/*
>> > > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
>> > > > + * @dev: the PCI device
>> > > > + * @pos: register offset of SR-IOV capability in PCI config space
>> > > > + * Returns true any VF BAR has resource allocated, false
>> > > > + * if all VF BARs are empty.
>> > > > + */
>> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>> > > > +       int i;
>> > > > +       u32 bar_value;
>> > > > +       u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
>> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
>> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
>> > > > +
>> > > > +       for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
>> > > > +               if (bar_value & bar_size_mask)
>> > > > +                       return true;
>> > > > +       }
>> > > > +
>> > > > +       return false;
>> > > > +}
>> > > > +
>> > > > +/*
>> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
>> > > > + * @dev: the PCI device
>> > > > + *
>> > > > + * Returns true if device is an AMD/ATI display adapter,
>> > > > + * otherwise return false.
>> > > > + */
>> > > > +
>> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
>> > > > +       return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
>> > > > +               (dev->vendor == PCI_VENDOR_ID_ATI ||
>> > > > +               dev->vendor == PCI_VENDOR_ID_AMD)); }
>> > > > +
>> > > >  /**
>> > > >   * pci_iov_init - initialize the IOV capability
>> > > >   * @dev: the PCI device
>> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>> > > >                 return -ENODEV;
>> > > >
>> > > >         pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
>> > > > -       if (pos)
>> > > > -               return sriov_init(dev, pos);
>> > > > -
>> > > > +       if (pos) {
>> > > > +       /*
>> > > > +        * If the device is an AMD graphics device and it supports
>> > > > +        * SR-IOV it will require a large amount of resources.
>> > > > +        * Before calling sriov_init() must ensure that the system
>> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
>> > > > +        * able to allocate enough resources.
>> > > > +        * If the VF BARs are zero then the system BIOS does not
>> > > > +        * support SR-IOV or it could not allocate the resources
>> > > > +        * and this platform will not support AMD graphics SR-IOV.
>> > > > +        * Therefore do not call sriov_init().
>> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
>> > > > +        * will be properly initialized to non-zero values.
>> > > > +        */
>> > > > +               if (is_amd_display_adapter(dev)) {
>> > > > +                       if (pci_vf_bar_valid(dev, pos))
>> > > > +                               return sriov_init(dev, pos);
>> > > > +               } else {
>> > > > +                       return sriov_init(dev, pos);
>> > > > +               }
>> > > > +       }
>> > > >         return -ENODEV;
>> > > >  }
>> > > >
>> > >
>> >
>>
>

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alexander Duyck]

On Mon, May 15, 2017 at 10:53 AM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Mon, 15 May 2017 08:19:28 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>
>> Hi Williamson,
>>
>> We cannot assume BIOS supports SR-IOV, actually only newer server mother=
board BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server=
 motherboard BIOS doesn't support SR-IOV. This issue would happen if an use=
r plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.
>
> Servers should be supporting SR-IOV for a long time now.  What really
> is there to a BIOS supporting SR-IOV anyway, it's simply reserving
> sufficient bus number and MMIO resources such that we can enable the
> VFs.  This process isn't exclusively reserved for the BIOS.  Some
> platforms may choose to only initialize boot devices, leaving the rest
> for the OS to program.  The initial proposal here to disable SR-IOV if
> not programmed at OS hand-off disables even the possibility of the OS
> reallocating resources for this device.

There are differences between supporting SR-IOV and supporting SR-IOV
on devices with massive resources. I know I have seen NICs that will
keep a system from completing POST if SR-IOV is enabled, and MMIO
beyond 4G is not. My guess would be that the issues being seen are
probably that they disable SR-IOV in the BIOS in such a setup and end
up running into issues when they try to boot into the Linux kernel as
it goes through and tries to allocate resources for SR-IOV even though
it was disabled in the BIOS.

It might make sense to add a kernel parameter something like a
"pci=3Dnosriov" that would allow for disabling SR-IOV and related
resource allocation if that is what we are talking about. That way you
could plug in these types of devices into a system with a legacy bios
or that doesn't wan to allocate addresses above 32b for MMIO, and this
parameter would be all that is needed to disable SR-IOV so you could
plug in a NIC that has SR-IOV associated with it.

>> I agree that failure to allocate VF resources should leave the device in=
 no worse condition than before it tried. I hope kernel could allocate PF d=
evice resource before allocating VF device resource, and keep PF device res=
ource valid and functional if failed to allocate VF device resource.
>>
>> I will send out dmesg log lspci info tomorrow. Thanks.
>
> Thanks,
> Alex
>
>> -----Original Message-----
>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> Sent: Friday, May 12, 2017 10:43 PM
>> To: Cheng, Collins <Collins.Cheng@amd.com>
>> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linu=
x-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Z=
ytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
>> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV =
incapable platform
>>
>> On Fri, 12 May 2017 04:51:43 +0000
>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>
>> > Hi Williamson,
>> >
>> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-I=
OV NIC. I don't think it is redundant to check the VF BAR valid before call=
 sriov_init(), it is safe and saving boot time, also there is no a better m=
ethod to know if system BIOS has correctly initialized the SR-IOV capabilit=
y or not.
>>
>> It also masks an underlying bug and creates a maintenance issue that we =
won't know when it's safe to remove this workaround.  I don't think faster =
boot is valid rationale, in one case SR-IOV is completely disabled, the oth=
er we attempt to allocate the resources the BIOS failed to provide.  I expe=
ct this is also a corner case, the BIOS should typically support SR-IOV, th=
erefore this situation should be an exception.
>>
>> > I did not try to fix the issue from the kernel resource allocation per=
spective, it is because:
>> >     1. I am not very familiar with the PCI resource allocation scheme =
in kernel. For example, in sriov_init(), kernel will re-assign the PCI reso=
urce for both VF and PF. I don't understand why kernel allocates resource f=
or VF firstly, then PF. If it is PF firstly, then this issue could be avoid=
ed.
>> >     2. I am not sure if kernel has error handler if PCI resource alloc=
ation failed. In this case, kernel cannot allocate enough resource to PF. I=
t should trigger some error handler to either just keep original BAR values=
 set by system BIOS, or disable this device and log errors.
>>
>> I think these are the issues we should be trying to solve and I'm sure f=
olks on the linux-pci list can help us identify the bug.  Minimally, failur=
e to allocate VF resources should leave the device in no worse condition th=
an before it tried.  Perhaps you could post more details about the issue, b=
oot with pci=3Dearlydump, post dmesg of a boot where the PF resources are i=
ncorrectly re-allocated, and include lspci -vvv for the SR-IOV device.  Als=
o, please test with the latest upstream kernel, upstream only patches old k=
ernels through stable backports of commits to the latest kernel.  Adding Yi=
nghai as a resource allocation expert. Thanks,
>>
>> Alex
>>
>> > -----Original Message-----
>> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> > Sent: Friday, May 12, 2017 12:01 PM
>> > To: Cheng, Collins <Collins.Cheng@amd.com>
>> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>> > linux-kernel@vger.kernel.org; Deucher, Alexander
>> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
>> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>> > SR-IOV incapable platform
>> >
>> > On Fri, 12 May 2017 03:42:46 +0000
>> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>> >
>> > > Hi Williamson,
>> > >
>> > > GPU card needs more BAR aperture resource than other PCI devices. Fo=
r example, Intel SR-IOV network card only require 512KB memory resource for=
 all VFs. AMD SR-IOV GPU card needs 256MB x16 VF =3D 4GB memory resource fo=
r frame buffer BAR aperture.
>> > >
>> > > If the system BIOS supports SR-IOV, it will reserve enough resource =
for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot alloca=
te the enough resource for VF BARs, only PF BAR will be assigned and VF BAR=
s are empty. Then system boots to Linux kernel and kernel doesn't check if =
the VF BARs are empty or valid. Kernel will re-assign the BAR resources for=
 PF and all VFs. The problem I saw is that kernel will fail to allocate PF =
BAR resource because some resources are assigned to VF, this is not expecte=
d. So kernel might need to do some check before re-assign the PF/VF resourc=
e, so that PF device will be correctly assigned BAR resource and user can u=
se PF device.
>> >
>> > So the problem is that something bad happens when the kernel is trying
>> > to reallocate resources in order to fulfill the requirements of the
>> > VFs, leaving the PF resources incorrectly programmed?  Why not just
>> > fix that bug rather than creating special handling for this
>> > vendor/class of device which disables any attempt to fixup resources
>> > for SR-IOV?  IOW, this patch just avoids the problem for your devices
>> > rather than fixing the bug.  I'd suggest fixing the bug such that the
>> > PF is left in a functional state if the kernel is unable to allocate
>> > sufficient resources for the VFs.  Thanks,
>> >
>> > Alex
>> >
>> > > -----Original Message-----
>> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> > > Sent: Friday, May 12, 2017 11:21 AM
>> > > To: Cheng, Collins <Collins.Cheng@amd.com>
>> > > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>> > > linux-kernel@vger.kernel.org; Deucher, Alexander
>> > > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
>> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>> > > SR-IOV incapable platform
>> > >
>> > > On Fri, 12 May 2017 02:50:32 +0000
>> > > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>> > >
>> > > > Hi Helgaas,
>> > > >
>> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
>> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable
>> > > > platform. It would cause a problem on PCI resource allocation in
>> > > > current Linux kernel.
>> > > >
>> > > > Therefore in order to allow the PF (Physical Function) device of
>> > > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is
>> > > > required to verify conditions for initializing BAR resources on
>> > > > AMD SR-IOV capable GPUs.
>> > > >
>> > > > If the device is an AMD graphics device and it supports SR-IOV it
>> > > > will require a large amount of resources.
>> > > > Before calling sriov_init() must ensure that the system BIOS also
>> > > > supports SR-IOV and that system BIOS has been able to allocate
>> > > > enough resources.
>> > > > If the VF BARs are zero then the system BIOS does not support
>> > > > SR-IOV or it could not allocate the resources and this platform
>> > > > will not support AMD graphics SR-IOV.
>> > > > Therefore do not call sriov_init().
>> > > > If the system BIOS does support SR-IOV then the VF BARs will be
>> > > > properly initialized to non-zero values.
>> > > >
>> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
>> > > >
>> > > > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes
>> > > > in quirks.c are for specific devices and one or more device ID are
>> > > > defined for the specific devices. However my patch is for all AMD
>> > > > SR-IOV capable GPUs, that includes all existing and future AMD ser=
ver GPUs.
>> > > > So it doesn't seem like a good fit to put the fix in quirks.c.
>> > >
>> > >
>> > > Why is an AMD graphics card unique here?  Doesn't sriov_init()
>> > > always need to be able to deal with devices of any type where the
>> > > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV devices
>> > > can fit their VFs within a minimum bridge aperture, most cannot.  I
>> > > don't understand why the VF resource requirements being
>> > > exceptionally large dictates that they receive special handling.
>> > > Thanks,
>> > >
>> > > Alex
>> > >
>> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>> > > > ---
>> > > >  drivers/pci/iov.c | 63
>> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
>> > > >
>> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>> > > > e30f05c..e4f1405 100644
>> > > > --- a/drivers/pci/iov.c
>> > > > +++ b/drivers/pci/iov.c
>> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_de=
v *dev)
>> > > >                 msleep(100);
>> > > >  }
>> > > >
>> > > > +/*
>> > > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
>> > > > + * @dev: the PCI device
>> > > > + * @pos: register offset of SR-IOV capability in PCI config space
>> > > > + * Returns true any VF BAR has resource allocated, false
>> > > > + * if all VF BARs are empty.
>> > > > + */
>> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>> > > > +       int i;
>> > > > +       u32 bar_value;
>> > > > +       u32 bar_size_mask =3D ~(PCI_BASE_ADDRESS_SPACE |
>> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
>> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
>> > > > +
>> > > > +       for (i =3D 0; i < PCI_SRIOV_NUM_BARS; i++) {
>> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i=
 * 4, &bar_value);
>> > > > +               if (bar_value & bar_size_mask)
>> > > > +                       return true;
>> > > > +       }
>> > > > +
>> > > > +       return false;
>> > > > +}
>> > > > +
>> > > > +/*
>> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
>> > > > + * @dev: the PCI device
>> > > > + *
>> > > > + * Returns true if device is an AMD/ATI display adapter,
>> > > > + * otherwise return false.
>> > > > + */
>> > > > +
>> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
>> > > > +       return (((dev->class >> 16) =3D=3D PCI_BASE_CLASS_DISPLAY)=
 &&
>> > > > +               (dev->vendor =3D=3D PCI_VENDOR_ID_ATI ||
>> > > > +               dev->vendor =3D=3D PCI_VENDOR_ID_AMD)); }
>> > > > +
>> > > >  /**
>> > > >   * pci_iov_init - initialize the IOV capability
>> > > >   * @dev: the PCI device
>> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>> > > >                 return -ENODEV;
>> > > >
>> > > >         pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV)=
;
>> > > > -       if (pos)
>> > > > -               return sriov_init(dev, pos);
>> > > > -
>> > > > +       if (pos) {
>> > > > +       /*
>> > > > +        * If the device is an AMD graphics device and it supports
>> > > > +        * SR-IOV it will require a large amount of resources.
>> > > > +        * Before calling sriov_init() must ensure that the system
>> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
>> > > > +        * able to allocate enough resources.
>> > > > +        * If the VF BARs are zero then the system BIOS does not
>> > > > +        * support SR-IOV or it could not allocate the resources
>> > > > +        * and this platform will not support AMD graphics SR-IOV.
>> > > > +        * Therefore do not call sriov_init().
>> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
>> > > > +        * will be properly initialized to non-zero values.
>> > > > +        */
>> > > > +               if (is_amd_display_adapter(dev)) {
>> > > > +                       if (pci_vf_bar_valid(dev, pos))
>> > > > +                               return sriov_init(dev, pos);
>> > > > +               } else {
>> > > > +                       return sriov_init(dev, pos);
>> > > > +               }
>> > > > +       }
>> > > >         return -ENODEV;
>> > > >  }
>> > > >
>> > >
>> >
>>
>

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Alex,

Yes, I hope kernel can disable SR-IOV and related VF resource allocation if the system BIOS is not SR-IOV capable.

Adding the parameter "pci=nosriov" sounds a doable solution, but it would need user to add this parameter manually, right? I think an automatic detection would be better. My patch is trying to auto detect and bypass VF resource allocation.


-Collins Cheng


-----Original Message-----
From: Alexander Duyck [mailto:alexander.duyck@gmail.com] 
Sent: Friday, May 19, 2017 11:44 PM
To: Alex Williamson <alex.williamson@redhat.com>
Cc: Cheng, Collins <Collins.Cheng@amd.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Mon, May 15, 2017 at 10:53 AM, Alex Williamson <alex.williamson@redhat.com> wrote:
> On Mon, 15 May 2017 08:19:28 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>
>> Hi Williamson,
>>
>> We cannot assume BIOS supports SR-IOV, actually only newer server motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server motherboard BIOS doesn't support SR-IOV. This issue would happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.
>
> Servers should be supporting SR-IOV for a long time now.  What really 
> is there to a BIOS supporting SR-IOV anyway, it's simply reserving 
> sufficient bus number and MMIO resources such that we can enable the 
> VFs.  This process isn't exclusively reserved for the BIOS.  Some 
> platforms may choose to only initialize boot devices, leaving the rest 
> for the OS to program.  The initial proposal here to disable SR-IOV if 
> not programmed at OS hand-off disables even the possibility of the OS 
> reallocating resources for this device.

There are differences between supporting SR-IOV and supporting SR-IOV on devices with massive resources. I know I have seen NICs that will keep a system from completing POST if SR-IOV is enabled, and MMIO beyond 4G is not. My guess would be that the issues being seen are probably that they disable SR-IOV in the BIOS in such a setup and end up running into issues when they try to boot into the Linux kernel as it goes through and tries to allocate resources for SR-IOV even though it was disabled in the BIOS.

It might make sense to add a kernel parameter something like a "pci=nosriov" that would allow for disabling SR-IOV and related resource allocation if that is what we are talking about. That way you could plug in these types of devices into a system with a legacy bios or that doesn't wan to allocate addresses above 32b for MMIO, and this parameter would be all that is needed to disable SR-IOV so you could plug in a NIC that has SR-IOV associated with it.

>> I agree that failure to allocate VF resources should leave the device in no worse condition than before it tried. I hope kernel could allocate PF device resource before allocating VF device resource, and keep PF device resource valid and functional if failed to allocate VF device resource.
>>
>> I will send out dmesg log lspci info tomorrow. Thanks.
>
> Thanks,
> Alex
>
>> -----Original Message-----
>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> Sent: Friday, May 12, 2017 10:43 PM
>> To: Cheng, Collins <Collins.Cheng@amd.com>
>> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
>> linux-kernel@vger.kernel.org; Deucher, Alexander 
>> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; 
>> Yinghai Lu <yinghai@kernel.org>
>> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
>> SR-IOV incapable platform
>>
>> On Fri, 12 May 2017 04:51:43 +0000
>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>
>> > Hi Williamson,
>> >
>> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.
>>
>> It also masks an underlying bug and creates a maintenance issue that we won't know when it's safe to remove this workaround.  I don't think faster boot is valid rationale, in one case SR-IOV is completely disabled, the other we attempt to allocate the resources the BIOS failed to provide.  I expect this is also a corner case, the BIOS should typically support SR-IOV, therefore this situation should be an exception.
>>
>> > I did not try to fix the issue from the kernel resource allocation perspective, it is because:
>> >     1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
>> >     2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.
>>
>> I think these are the issues we should be trying to solve and I'm 
>> sure folks on the linux-pci list can help us identify the bug.  
>> Minimally, failure to allocate VF resources should leave the device 
>> in no worse condition than before it tried.  Perhaps you could post 
>> more details about the issue, boot with pci=earlydump, post dmesg of 
>> a boot where the PF resources are incorrectly re-allocated, and 
>> include lspci -vvv for the SR-IOV device.  Also, please test with the 
>> latest upstream kernel, upstream only patches old kernels through 
>> stable backports of commits to the latest kernel.  Adding Yinghai as 
>> a resource allocation expert. Thanks,
>>
>> Alex
>>
>> > -----Original Message-----
>> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> > Sent: Friday, May 12, 2017 12:01 PM
>> > To: Cheng, Collins <Collins.Cheng@amd.com>
>> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
>> > linux-kernel@vger.kernel.org; Deucher, Alexander 
>> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
>> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
>> > SR-IOV incapable platform
>> >
>> > On Fri, 12 May 2017 03:42:46 +0000
>> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>> >
>> > > Hi Williamson,
>> > >
>> > > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
>> > >
>> > > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.
>> >
>> > So the problem is that something bad happens when the kernel is 
>> > trying to reallocate resources in order to fulfill the requirements 
>> > of the VFs, leaving the PF resources incorrectly programmed?  Why 
>> > not just fix that bug rather than creating special handling for 
>> > this vendor/class of device which disables any attempt to fixup 
>> > resources for SR-IOV?  IOW, this patch just avoids the problem for 
>> > your devices rather than fixing the bug.  I'd suggest fixing the 
>> > bug such that the PF is left in a functional state if the kernel is 
>> > unable to allocate sufficient resources for the VFs.  Thanks,
>> >
>> > Alex
>> >
>> > > -----Original Message-----
>> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> > > Sent: Friday, May 12, 2017 11:21 AM
>> > > To: Cheng, Collins <Collins.Cheng@amd.com>
>> > > Cc: Bjorn Helgaas <bhelgaas@google.com>; 
>> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, 
>> > > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly 
>> > > <Kelly.Zytaruk@amd.com>
>> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
>> > > SR-IOV incapable platform
>> > >
>> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins" 
>> > > <Collins.Cheng@amd.com> wrote:
>> > >
>> > > > Hi Helgaas,
>> > > >
>> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
>> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable 
>> > > > platform. It would cause a problem on PCI resource allocation 
>> > > > in current Linux kernel.
>> > > >
>> > > > Therefore in order to allow the PF (Physical Function) device 
>> > > > of SR-IOV capable GPU to work on the SR-IOV incapable platform, 
>> > > > it is required to verify conditions for initializing BAR 
>> > > > resources on AMD SR-IOV capable GPUs.
>> > > >
>> > > > If the device is an AMD graphics device and it supports SR-IOV 
>> > > > it will require a large amount of resources.
>> > > > Before calling sriov_init() must ensure that the system BIOS 
>> > > > also supports SR-IOV and that system BIOS has been able to 
>> > > > allocate enough resources.
>> > > > If the VF BARs are zero then the system BIOS does not support 
>> > > > SR-IOV or it could not allocate the resources and this platform 
>> > > > will not support AMD graphics SR-IOV.
>> > > > Therefore do not call sriov_init().
>> > > > If the system BIOS does support SR-IOV then the VF BARs will be 
>> > > > properly initialized to non-zero values.
>> > > >
>> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
>> > > >
>> > > > I checked the drivers/pci/quirks.c, it looks the 
>> > > > workarounds/fixes in quirks.c are for specific devices and one 
>> > > > or more device ID are defined for the specific devices. However 
>> > > > my patch is for all AMD SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
>> > > > So it doesn't seem like a good fit to put the fix in quirks.c.
>> > >
>> > >
>> > > Why is an AMD graphics card unique here?  Doesn't sriov_init() 
>> > > always need to be able to deal with devices of any type where the 
>> > > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV 
>> > > devices can fit their VFs within a minimum bridge aperture, most 
>> > > cannot.  I don't understand why the VF resource requirements 
>> > > being exceptionally large dictates that they receive special handling.
>> > > Thanks,
>> > >
>> > > Alex
>> > >
>> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>> > > > ---
>> > > >  drivers/pci/iov.c | 63
>> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
>> > > >
>> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>> > > > e30f05c..e4f1405 100644
>> > > > --- a/drivers/pci/iov.c
>> > > > +++ b/drivers/pci/iov.c
>> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
>> > > >                 msleep(100);
>> > > >  }
>> > > >
>> > > > +/*
>> > > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
>> > > > + * @dev: the PCI device
>> > > > + * @pos: register offset of SR-IOV capability in PCI config 
>> > > > +space
>> > > > + * Returns true any VF BAR has resource allocated, false
>> > > > + * if all VF BARs are empty.
>> > > > + */
>> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>> > > > +       int i;
>> > > > +       u32 bar_value;
>> > > > +       u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
>> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
>> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
>> > > > +
>> > > > +       for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
>> > > > +               if (bar_value & bar_size_mask)
>> > > > +                       return true;
>> > > > +       }
>> > > > +
>> > > > +       return false;
>> > > > +}
>> > > > +
>> > > > +/*
>> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU 
>> > > > +device
>> > > > + * @dev: the PCI device
>> > > > + *
>> > > > + * Returns true if device is an AMD/ATI display adapter,
>> > > > + * otherwise return false.
>> > > > + */
>> > > > +
>> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
>> > > > +       return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
>> > > > +               (dev->vendor == PCI_VENDOR_ID_ATI ||
>> > > > +               dev->vendor == PCI_VENDOR_ID_AMD)); }
>> > > > +
>> > > >  /**
>> > > >   * pci_iov_init - initialize the IOV capability
>> > > >   * @dev: the PCI device
>> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>> > > >                 return -ENODEV;
>> > > >
>> > > >         pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
>> > > > -       if (pos)
>> > > > -               return sriov_init(dev, pos);
>> > > > -
>> > > > +       if (pos) {
>> > > > +       /*
>> > > > +        * If the device is an AMD graphics device and it supports
>> > > > +        * SR-IOV it will require a large amount of resources.
>> > > > +        * Before calling sriov_init() must ensure that the system
>> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
>> > > > +        * able to allocate enough resources.
>> > > > +        * If the VF BARs are zero then the system BIOS does not
>> > > > +        * support SR-IOV or it could not allocate the resources
>> > > > +        * and this platform will not support AMD graphics SR-IOV.
>> > > > +        * Therefore do not call sriov_init().
>> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
>> > > > +        * will be properly initialized to non-zero values.
>> > > > +        */
>> > > > +               if (is_amd_display_adapter(dev)) {
>> > > > +                       if (pci_vf_bar_valid(dev, pos))
>> > > > +                               return sriov_init(dev, pos);
>> > > > +               } else {
>> > > > +                       return sriov_init(dev, pos);
>> > > > +               }
>> > > > +       }
>> > > >         return -ENODEV;
>> > > >  }
>> > > >
>> > >
>> >
>>
>

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

SGkgQWxleCwNCg0KWWVzLCBJIGhvcGUga2VybmVsIGNhbiBkaXNhYmxlIFNSLUlPViBhbmQgcmVs
YXRlZCBWRiByZXNvdXJjZSBhbGxvY2F0aW9uIGlmIHRoZSBzeXN0ZW0gQklPUyBpcyBub3QgU1It
SU9WIGNhcGFibGUuDQoNCkFkZGluZyB0aGUgcGFyYW1ldGVyICJwY2k9bm9zcmlvdiIgc291bmRz
IGEgZG9hYmxlIHNvbHV0aW9uLCBidXQgaXQgd291bGQgbmVlZCB1c2VyIHRvIGFkZCB0aGlzIHBh
cmFtZXRlciBtYW51YWxseSwgcmlnaHQ/IEkgdGhpbmsgYW4gYXV0b21hdGljIGRldGVjdGlvbiB3
b3VsZCBiZSBiZXR0ZXIuIE15IHBhdGNoIGlzIHRyeWluZyB0byBhdXRvIGRldGVjdCBhbmQgYnlw
YXNzIFZGIHJlc291cmNlIGFsbG9jYXRpb24uDQoNCg0KLUNvbGxpbnMgQ2hlbmcNCg0KDQotLS0t
LU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogQWxleGFuZGVyIER1eWNrIFttYWlsdG86YWxl
eGFuZGVyLmR1eWNrQGdtYWlsLmNvbV0gDQpTZW50OiBGcmlkYXksIE1heSAxOSwgMjAxNyAxMTo0
NCBQTQ0KVG86IEFsZXggV2lsbGlhbXNvbiA8YWxleC53aWxsaWFtc29uQHJlZGhhdC5jb20+DQpD
YzogQ2hlbmcsIENvbGxpbnMgPENvbGxpbnMuQ2hlbmdAYW1kLmNvbT47IEJqb3JuIEhlbGdhYXMg
PGJoZWxnYWFzQGdvb2dsZS5jb20+OyBsaW51eC1wY2lAdmdlci5rZXJuZWwub3JnOyBsaW51eC1r
ZXJuZWxAdmdlci5rZXJuZWwub3JnOyBEZXVjaGVyLCBBbGV4YW5kZXIgPEFsZXhhbmRlci5EZXVj
aGVyQGFtZC5jb20+OyBaeXRhcnVrLCBLZWxseSA8S2VsbHkuWnl0YXJ1a0BhbWQuY29tPjsgWWlu
Z2hhaSBMdSA8eWluZ2hhaUBrZXJuZWwub3JnPg0KU3ViamVjdDogUmU6IFtQQVRDSF0gUENJOiBN
YWtlIFNSLUlPViBjYXBhYmxlIEdQVSB3b3JraW5nIG9uIHRoZSBTUi1JT1YgaW5jYXBhYmxlIHBs
YXRmb3JtDQoNCk9uIE1vbiwgTWF5IDE1LCAyMDE3IGF0IDEwOjUzIEFNLCBBbGV4IFdpbGxpYW1z
b24gPGFsZXgud2lsbGlhbXNvbkByZWRoYXQuY29tPiB3cm90ZToNCj4gT24gTW9uLCAxNSBNYXkg
MjAxNyAwODoxOToyOCArMDAwMA0KPiAiQ2hlbmcsIENvbGxpbnMiIDxDb2xsaW5zLkNoZW5nQGFt
ZC5jb20+IHdyb3RlOg0KPg0KPj4gSGkgV2lsbGlhbXNvbiwNCj4+DQo+PiBXZSBjYW5ub3QgYXNz
dW1lIEJJT1Mgc3VwcG9ydHMgU1ItSU9WLCBhY3R1YWxseSBvbmx5IG5ld2VyIHNlcnZlciBtb3Ro
ZXJib2FyZCBCSU9TIHN1cHBvcnRzIFNSLUlPVi4gTm9ybWFsIGRlc2t0b3AgbW90aGVyYm9hcmQg
QklPUyBvciBvbGRlciBzZXJ2ZXIgbW90aGVyYm9hcmQgQklPUyBkb2Vzbid0IHN1cHBvcnQgU1It
SU9WLiBUaGlzIGlzc3VlIHdvdWxkIGhhcHBlbiBpZiBhbiB1c2VyIHBsdWdzIG91ciBBTUQgU1It
SU9WIGNhcGFibGUgR1BVIGNhcmQgdG8gYSBub3JtYWwgZGVza3RvcCBtb3RoZXJib2FyZC4NCj4N
Cj4gU2VydmVycyBzaG91bGQgYmUgc3VwcG9ydGluZyBTUi1JT1YgZm9yIGEgbG9uZyB0aW1lIG5v
dy4gIFdoYXQgcmVhbGx5IA0KPiBpcyB0aGVyZSB0byBhIEJJT1Mgc3VwcG9ydGluZyBTUi1JT1Yg
YW55d2F5LCBpdCdzIHNpbXBseSByZXNlcnZpbmcgDQo+IHN1ZmZpY2llbnQgYnVzIG51bWJlciBh
bmQgTU1JTyByZXNvdXJjZXMgc3VjaCB0aGF0IHdlIGNhbiBlbmFibGUgdGhlIA0KPiBWRnMuICBU
aGlzIHByb2Nlc3MgaXNuJ3QgZXhjbHVzaXZlbHkgcmVzZXJ2ZWQgZm9yIHRoZSBCSU9TLiAgU29t
ZSANCj4gcGxhdGZvcm1zIG1heSBjaG9vc2UgdG8gb25seSBpbml0aWFsaXplIGJvb3QgZGV2aWNl
cywgbGVhdmluZyB0aGUgcmVzdCANCj4gZm9yIHRoZSBPUyB0byBwcm9ncmFtLiAgVGhlIGluaXRp
YWwgcHJvcG9zYWwgaGVyZSB0byBkaXNhYmxlIFNSLUlPViBpZiANCj4gbm90IHByb2dyYW1tZWQg
YXQgT1MgaGFuZC1vZmYgZGlzYWJsZXMgZXZlbiB0aGUgcG9zc2liaWxpdHkgb2YgdGhlIE9TIA0K
PiByZWFsbG9jYXRpbmcgcmVzb3VyY2VzIGZvciB0aGlzIGRldmljZS4NCg0KVGhlcmUgYXJlIGRp
ZmZlcmVuY2VzIGJldHdlZW4gc3VwcG9ydGluZyBTUi1JT1YgYW5kIHN1cHBvcnRpbmcgU1ItSU9W
IG9uIGRldmljZXMgd2l0aCBtYXNzaXZlIHJlc291cmNlcy4gSSBrbm93IEkgaGF2ZSBzZWVuIE5J
Q3MgdGhhdCB3aWxsIGtlZXAgYSBzeXN0ZW0gZnJvbSBjb21wbGV0aW5nIFBPU1QgaWYgU1ItSU9W
IGlzIGVuYWJsZWQsIGFuZCBNTUlPIGJleW9uZCA0RyBpcyBub3QuIE15IGd1ZXNzIHdvdWxkIGJl
IHRoYXQgdGhlIGlzc3VlcyBiZWluZyBzZWVuIGFyZSBwcm9iYWJseSB0aGF0IHRoZXkgZGlzYWJs
ZSBTUi1JT1YgaW4gdGhlIEJJT1MgaW4gc3VjaCBhIHNldHVwIGFuZCBlbmQgdXAgcnVubmluZyBp
bnRvIGlzc3VlcyB3aGVuIHRoZXkgdHJ5IHRvIGJvb3QgaW50byB0aGUgTGludXgga2VybmVsIGFz
IGl0IGdvZXMgdGhyb3VnaCBhbmQgdHJpZXMgdG8gYWxsb2NhdGUgcmVzb3VyY2VzIGZvciBTUi1J
T1YgZXZlbiB0aG91Z2ggaXQgd2FzIGRpc2FibGVkIGluIHRoZSBCSU9TLg0KDQpJdCBtaWdodCBt
YWtlIHNlbnNlIHRvIGFkZCBhIGtlcm5lbCBwYXJhbWV0ZXIgc29tZXRoaW5nIGxpa2UgYSAicGNp
PW5vc3Jpb3YiIHRoYXQgd291bGQgYWxsb3cgZm9yIGRpc2FibGluZyBTUi1JT1YgYW5kIHJlbGF0
ZWQgcmVzb3VyY2UgYWxsb2NhdGlvbiBpZiB0aGF0IGlzIHdoYXQgd2UgYXJlIHRhbGtpbmcgYWJv
dXQuIFRoYXQgd2F5IHlvdSBjb3VsZCBwbHVnIGluIHRoZXNlIHR5cGVzIG9mIGRldmljZXMgaW50
byBhIHN5c3RlbSB3aXRoIGEgbGVnYWN5IGJpb3Mgb3IgdGhhdCBkb2Vzbid0IHdhbiB0byBhbGxv
Y2F0ZSBhZGRyZXNzZXMgYWJvdmUgMzJiIGZvciBNTUlPLCBhbmQgdGhpcyBwYXJhbWV0ZXIgd291
bGQgYmUgYWxsIHRoYXQgaXMgbmVlZGVkIHRvIGRpc2FibGUgU1ItSU9WIHNvIHlvdSBjb3VsZCBw
bHVnIGluIGEgTklDIHRoYXQgaGFzIFNSLUlPViBhc3NvY2lhdGVkIHdpdGggaXQuDQoNCj4+IEkg
YWdyZWUgdGhhdCBmYWlsdXJlIHRvIGFsbG9jYXRlIFZGIHJlc291cmNlcyBzaG91bGQgbGVhdmUg
dGhlIGRldmljZSBpbiBubyB3b3JzZSBjb25kaXRpb24gdGhhbiBiZWZvcmUgaXQgdHJpZWQuIEkg
aG9wZSBrZXJuZWwgY291bGQgYWxsb2NhdGUgUEYgZGV2aWNlIHJlc291cmNlIGJlZm9yZSBhbGxv
Y2F0aW5nIFZGIGRldmljZSByZXNvdXJjZSwgYW5kIGtlZXAgUEYgZGV2aWNlIHJlc291cmNlIHZh
bGlkIGFuZCBmdW5jdGlvbmFsIGlmIGZhaWxlZCB0byBhbGxvY2F0ZSBWRiBkZXZpY2UgcmVzb3Vy
Y2UuDQo+Pg0KPj4gSSB3aWxsIHNlbmQgb3V0IGRtZXNnIGxvZyBsc3BjaSBpbmZvIHRvbW9ycm93
LiBUaGFua3MuDQo+DQo+IFRoYW5rcywNCj4gQWxleA0KPg0KPj4gLS0tLS1PcmlnaW5hbCBNZXNz
YWdlLS0tLS0NCj4+IEZyb206IEFsZXggV2lsbGlhbXNvbiBbbWFpbHRvOmFsZXgud2lsbGlhbXNv
bkByZWRoYXQuY29tXQ0KPj4gU2VudDogRnJpZGF5LCBNYXkgMTIsIDIwMTcgMTA6NDMgUE0NCj4+
IFRvOiBDaGVuZywgQ29sbGlucyA8Q29sbGlucy5DaGVuZ0BhbWQuY29tPg0KPj4gQ2M6IEJqb3Ju
IEhlbGdhYXMgPGJoZWxnYWFzQGdvb2dsZS5jb20+OyBsaW51eC1wY2lAdmdlci5rZXJuZWwub3Jn
OyANCj4+IGxpbnV4LWtlcm5lbEB2Z2VyLmtlcm5lbC5vcmc7IERldWNoZXIsIEFsZXhhbmRlciAN
Cj4+IDxBbGV4YW5kZXIuRGV1Y2hlckBhbWQuY29tPjsgWnl0YXJ1aywgS2VsbHkgPEtlbGx5Llp5
dGFydWtAYW1kLmNvbT47IA0KPj4gWWluZ2hhaSBMdSA8eWluZ2hhaUBrZXJuZWwub3JnPg0KPj4g
U3ViamVjdDogUmU6IFtQQVRDSF0gUENJOiBNYWtlIFNSLUlPViBjYXBhYmxlIEdQVSB3b3JraW5n
IG9uIHRoZSANCj4+IFNSLUlPViBpbmNhcGFibGUgcGxhdGZvcm0NCj4+DQo+PiBPbiBGcmksIDEy
IE1heSAyMDE3IDA0OjUxOjQzICswMDAwDQo+PiAiQ2hlbmcsIENvbGxpbnMiIDxDb2xsaW5zLkNo
ZW5nQGFtZC5jb20+IHdyb3RlOg0KPj4NCj4+ID4gSGkgV2lsbGlhbXNvbiwNCj4+ID4NCj4+ID4g
SSB2ZXJpZmllZCB0aGUgcGF0Y2ggaXMgd29ya2luZyBmb3IgYm90aCBBTUQgU1ItSU9WIEdQVSBh
bmQgSW50ZWwgU1ItSU9WIE5JQy4gSSBkb24ndCB0aGluayBpdCBpcyByZWR1bmRhbnQgdG8gY2hl
Y2sgdGhlIFZGIEJBUiB2YWxpZCBiZWZvcmUgY2FsbCBzcmlvdl9pbml0KCksIGl0IGlzIHNhZmUg
YW5kIHNhdmluZyBib290IHRpbWUsIGFsc28gdGhlcmUgaXMgbm8gYSBiZXR0ZXIgbWV0aG9kIHRv
IGtub3cgaWYgc3lzdGVtIEJJT1MgaGFzIGNvcnJlY3RseSBpbml0aWFsaXplZCB0aGUgU1ItSU9W
IGNhcGFiaWxpdHkgb3Igbm90Lg0KPj4NCj4+IEl0IGFsc28gbWFza3MgYW4gdW5kZXJseWluZyBi
dWcgYW5kIGNyZWF0ZXMgYSBtYWludGVuYW5jZSBpc3N1ZSB0aGF0IHdlIHdvbid0IGtub3cgd2hl
biBpdCdzIHNhZmUgdG8gcmVtb3ZlIHRoaXMgd29ya2Fyb3VuZC4gIEkgZG9uJ3QgdGhpbmsgZmFz
dGVyIGJvb3QgaXMgdmFsaWQgcmF0aW9uYWxlLCBpbiBvbmUgY2FzZSBTUi1JT1YgaXMgY29tcGxl
dGVseSBkaXNhYmxlZCwgdGhlIG90aGVyIHdlIGF0dGVtcHQgdG8gYWxsb2NhdGUgdGhlIHJlc291
cmNlcyB0aGUgQklPUyBmYWlsZWQgdG8gcHJvdmlkZS4gIEkgZXhwZWN0IHRoaXMgaXMgYWxzbyBh
IGNvcm5lciBjYXNlLCB0aGUgQklPUyBzaG91bGQgdHlwaWNhbGx5IHN1cHBvcnQgU1ItSU9WLCB0
aGVyZWZvcmUgdGhpcyBzaXR1YXRpb24gc2hvdWxkIGJlIGFuIGV4Y2VwdGlvbi4NCj4+DQo+PiA+
IEkgZGlkIG5vdCB0cnkgdG8gZml4IHRoZSBpc3N1ZSBmcm9tIHRoZSBrZXJuZWwgcmVzb3VyY2Ug
YWxsb2NhdGlvbiBwZXJzcGVjdGl2ZSwgaXQgaXMgYmVjYXVzZToNCj4+ID4gICAgIDEuIEkgYW0g
bm90IHZlcnkgZmFtaWxpYXIgd2l0aCB0aGUgUENJIHJlc291cmNlIGFsbG9jYXRpb24gc2NoZW1l
IGluIGtlcm5lbC4gRm9yIGV4YW1wbGUsIGluIHNyaW92X2luaXQoKSwga2VybmVsIHdpbGwgcmUt
YXNzaWduIHRoZSBQQ0kgcmVzb3VyY2UgZm9yIGJvdGggVkYgYW5kIFBGLiBJIGRvbid0IHVuZGVy
c3RhbmQgd2h5IGtlcm5lbCBhbGxvY2F0ZXMgcmVzb3VyY2UgZm9yIFZGIGZpcnN0bHksIHRoZW4g
UEYuIElmIGl0IGlzIFBGIGZpcnN0bHksIHRoZW4gdGhpcyBpc3N1ZSBjb3VsZCBiZSBhdm9pZGVk
Lg0KPj4gPiAgICAgMi4gSSBhbSBub3Qgc3VyZSBpZiBrZXJuZWwgaGFzIGVycm9yIGhhbmRsZXIg
aWYgUENJIHJlc291cmNlIGFsbG9jYXRpb24gZmFpbGVkLiBJbiB0aGlzIGNhc2UsIGtlcm5lbCBj
YW5ub3QgYWxsb2NhdGUgZW5vdWdoIHJlc291cmNlIHRvIFBGLiBJdCBzaG91bGQgdHJpZ2dlciBz
b21lIGVycm9yIGhhbmRsZXIgdG8gZWl0aGVyIGp1c3Qga2VlcCBvcmlnaW5hbCBCQVIgdmFsdWVz
IHNldCBieSBzeXN0ZW0gQklPUywgb3IgZGlzYWJsZSB0aGlzIGRldmljZSBhbmQgbG9nIGVycm9y
cy4NCj4+DQo+PiBJIHRoaW5rIHRoZXNlIGFyZSB0aGUgaXNzdWVzIHdlIHNob3VsZCBiZSB0cnlp
bmcgdG8gc29sdmUgYW5kIEknbSANCj4+IHN1cmUgZm9sa3Mgb24gdGhlIGxpbnV4LXBjaSBsaXN0
IGNhbiBoZWxwIHVzIGlkZW50aWZ5IHRoZSBidWcuICANCj4+IE1pbmltYWxseSwgZmFpbHVyZSB0
byBhbGxvY2F0ZSBWRiByZXNvdXJjZXMgc2hvdWxkIGxlYXZlIHRoZSBkZXZpY2UgDQo+PiBpbiBu
byB3b3JzZSBjb25kaXRpb24gdGhhbiBiZWZvcmUgaXQgdHJpZWQuICBQZXJoYXBzIHlvdSBjb3Vs
ZCBwb3N0IA0KPj4gbW9yZSBkZXRhaWxzIGFib3V0IHRoZSBpc3N1ZSwgYm9vdCB3aXRoIHBjaT1l
YXJseWR1bXAsIHBvc3QgZG1lc2cgb2YgDQo+PiBhIGJvb3Qgd2hlcmUgdGhlIFBGIHJlc291cmNl
cyBhcmUgaW5jb3JyZWN0bHkgcmUtYWxsb2NhdGVkLCBhbmQgDQo+PiBpbmNsdWRlIGxzcGNpIC12
dnYgZm9yIHRoZSBTUi1JT1YgZGV2aWNlLiAgQWxzbywgcGxlYXNlIHRlc3Qgd2l0aCB0aGUgDQo+
PiBsYXRlc3QgdXBzdHJlYW0ga2VybmVsLCB1cHN0cmVhbSBvbmx5IHBhdGNoZXMgb2xkIGtlcm5l
bHMgdGhyb3VnaCANCj4+IHN0YWJsZSBiYWNrcG9ydHMgb2YgY29tbWl0cyB0byB0aGUgbGF0ZXN0
IGtlcm5lbC4gIEFkZGluZyBZaW5naGFpIGFzIA0KPj4gYSByZXNvdXJjZSBhbGxvY2F0aW9uIGV4
cGVydC4gVGhhbmtzLA0KPj4NCj4+IEFsZXgNCj4+DQo+PiA+IC0tLS0tT3JpZ2luYWwgTWVzc2Fn
ZS0tLS0tDQo+PiA+IEZyb206IEFsZXggV2lsbGlhbXNvbiBbbWFpbHRvOmFsZXgud2lsbGlhbXNv
bkByZWRoYXQuY29tXQ0KPj4gPiBTZW50OiBGcmlkYXksIE1heSAxMiwgMjAxNyAxMjowMSBQTQ0K
Pj4gPiBUbzogQ2hlbmcsIENvbGxpbnMgPENvbGxpbnMuQ2hlbmdAYW1kLmNvbT4NCj4+ID4gQ2M6
IEJqb3JuIEhlbGdhYXMgPGJoZWxnYWFzQGdvb2dsZS5jb20+OyBsaW51eC1wY2lAdmdlci5rZXJu
ZWwub3JnOyANCj4+ID4gbGludXgta2VybmVsQHZnZXIua2VybmVsLm9yZzsgRGV1Y2hlciwgQWxl
eGFuZGVyIA0KPj4gPiA8QWxleGFuZGVyLkRldWNoZXJAYW1kLmNvbT47IFp5dGFydWssIEtlbGx5
IDxLZWxseS5aeXRhcnVrQGFtZC5jb20+DQo+PiA+IFN1YmplY3Q6IFJlOiBbUEFUQ0hdIFBDSTog
TWFrZSBTUi1JT1YgY2FwYWJsZSBHUFUgd29ya2luZyBvbiB0aGUgDQo+PiA+IFNSLUlPViBpbmNh
cGFibGUgcGxhdGZvcm0NCj4+ID4NCj4+ID4gT24gRnJpLCAxMiBNYXkgMjAxNyAwMzo0Mjo0NiAr
MDAwMA0KPj4gPiAiQ2hlbmcsIENvbGxpbnMiIDxDb2xsaW5zLkNoZW5nQGFtZC5jb20+IHdyb3Rl
Og0KPj4gPg0KPj4gPiA+IEhpIFdpbGxpYW1zb24sDQo+PiA+ID4NCj4+ID4gPiBHUFUgY2FyZCBu
ZWVkcyBtb3JlIEJBUiBhcGVydHVyZSByZXNvdXJjZSB0aGFuIG90aGVyIFBDSSBkZXZpY2VzLiBG
b3IgZXhhbXBsZSwgSW50ZWwgU1ItSU9WIG5ldHdvcmsgY2FyZCBvbmx5IHJlcXVpcmUgNTEyS0Ig
bWVtb3J5IHJlc291cmNlIGZvciBhbGwgVkZzLiBBTUQgU1ItSU9WIEdQVSBjYXJkIG5lZWRzIDI1
Nk1CIHgxNiBWRiA9IDRHQiBtZW1vcnkgcmVzb3VyY2UgZm9yIGZyYW1lIGJ1ZmZlciBCQVIgYXBl
cnR1cmUuDQo+PiA+ID4NCj4+ID4gPiBJZiB0aGUgc3lzdGVtIEJJT1Mgc3VwcG9ydHMgU1ItSU9W
LCBpdCB3aWxsIHJlc2VydmUgZW5vdWdoIHJlc291cmNlIGZvciBhbGwgVkYgQkFScy4gSWYgdGhl
IHN5c3RlbSBCSU9TIGRvZXNuJ3Qgc3VwcG9ydCBTUi1JT1Ygb3IgY2Fubm90IGFsbG9jYXRlIHRo
ZSBlbm91Z2ggcmVzb3VyY2UgZm9yIFZGIEJBUnMsIG9ubHkgUEYgQkFSIHdpbGwgYmUgYXNzaWdu
ZWQgYW5kIFZGIEJBUnMgYXJlIGVtcHR5LiBUaGVuIHN5c3RlbSBib290cyB0byBMaW51eCBrZXJu
ZWwgYW5kIGtlcm5lbCBkb2Vzbid0IGNoZWNrIGlmIHRoZSBWRiBCQVJzIGFyZSBlbXB0eSBvciB2
YWxpZC4gS2VybmVsIHdpbGwgcmUtYXNzaWduIHRoZSBCQVIgcmVzb3VyY2VzIGZvciBQRiBhbmQg
YWxsIFZGcy4gVGhlIHByb2JsZW0gSSBzYXcgaXMgdGhhdCBrZXJuZWwgd2lsbCBmYWlsIHRvIGFs
bG9jYXRlIFBGIEJBUiByZXNvdXJjZSBiZWNhdXNlIHNvbWUgcmVzb3VyY2VzIGFyZSBhc3NpZ25l
ZCB0byBWRiwgdGhpcyBpcyBub3QgZXhwZWN0ZWQuIFNvIGtlcm5lbCBtaWdodCBuZWVkIHRvIGRv
IHNvbWUgY2hlY2sgYmVmb3JlIHJlLWFzc2lnbiB0aGUgUEYvVkYgcmVzb3VyY2UsIHNvIHRoYXQg
UEYgZGV2aWNlIHdpbGwgYmUgY29ycmVjdGx5IGFzc2lnbmVkIEJBUiByZXNvdXJjZSBhbmQgdXNl
ciBjYW4gdXNlIFBGIGRldmljZS4NCj4+ID4NCj4+ID4gU28gdGhlIHByb2JsZW0gaXMgdGhhdCBz
b21ldGhpbmcgYmFkIGhhcHBlbnMgd2hlbiB0aGUga2VybmVsIGlzIA0KPj4gPiB0cnlpbmcgdG8g
cmVhbGxvY2F0ZSByZXNvdXJjZXMgaW4gb3JkZXIgdG8gZnVsZmlsbCB0aGUgcmVxdWlyZW1lbnRz
IA0KPj4gPiBvZiB0aGUgVkZzLCBsZWF2aW5nIHRoZSBQRiByZXNvdXJjZXMgaW5jb3JyZWN0bHkg
cHJvZ3JhbW1lZD8gIFdoeSANCj4+ID4gbm90IGp1c3QgZml4IHRoYXQgYnVnIHJhdGhlciB0aGFu
IGNyZWF0aW5nIHNwZWNpYWwgaGFuZGxpbmcgZm9yIA0KPj4gPiB0aGlzIHZlbmRvci9jbGFzcyBv
ZiBkZXZpY2Ugd2hpY2ggZGlzYWJsZXMgYW55IGF0dGVtcHQgdG8gZml4dXAgDQo+PiA+IHJlc291
cmNlcyBmb3IgU1ItSU9WPyAgSU9XLCB0aGlzIHBhdGNoIGp1c3QgYXZvaWRzIHRoZSBwcm9ibGVt
IGZvciANCj4+ID4geW91ciBkZXZpY2VzIHJhdGhlciB0aGFuIGZpeGluZyB0aGUgYnVnLiAgSSdk
IHN1Z2dlc3QgZml4aW5nIHRoZSANCj4+ID4gYnVnIHN1Y2ggdGhhdCB0aGUgUEYgaXMgbGVmdCBp
biBhIGZ1bmN0aW9uYWwgc3RhdGUgaWYgdGhlIGtlcm5lbCBpcyANCj4+ID4gdW5hYmxlIHRvIGFs
bG9jYXRlIHN1ZmZpY2llbnQgcmVzb3VyY2VzIGZvciB0aGUgVkZzLiAgVGhhbmtzLA0KPj4gPg0K
Pj4gPiBBbGV4DQo+PiA+DQo+PiA+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4+ID4g
PiBGcm9tOiBBbGV4IFdpbGxpYW1zb24gW21haWx0bzphbGV4LndpbGxpYW1zb25AcmVkaGF0LmNv
bV0NCj4+ID4gPiBTZW50OiBGcmlkYXksIE1heSAxMiwgMjAxNyAxMToyMSBBTQ0KPj4gPiA+IFRv
OiBDaGVuZywgQ29sbGlucyA8Q29sbGlucy5DaGVuZ0BhbWQuY29tPg0KPj4gPiA+IENjOiBCam9y
biBIZWxnYWFzIDxiaGVsZ2Fhc0Bnb29nbGUuY29tPjsgDQo+PiA+ID4gbGludXgtcGNpQHZnZXIu
a2VybmVsLm9yZzsgbGludXgta2VybmVsQHZnZXIua2VybmVsLm9yZzsgRGV1Y2hlciwgDQo+PiA+
ID4gQWxleGFuZGVyIDxBbGV4YW5kZXIuRGV1Y2hlckBhbWQuY29tPjsgWnl0YXJ1aywgS2VsbHkg
DQo+PiA+ID4gPEtlbGx5Llp5dGFydWtAYW1kLmNvbT4NCj4+ID4gPiBTdWJqZWN0OiBSZTogW1BB
VENIXSBQQ0k6IE1ha2UgU1ItSU9WIGNhcGFibGUgR1BVIHdvcmtpbmcgb24gdGhlIA0KPj4gPiA+
IFNSLUlPViBpbmNhcGFibGUgcGxhdGZvcm0NCj4+ID4gPg0KPj4gPiA+IE9uIEZyaSwgMTIgTWF5
IDIwMTcgMDI6NTA6MzIgKzAwMDAgIkNoZW5nLCBDb2xsaW5zIiANCj4+ID4gPiA8Q29sbGlucy5D
aGVuZ0BhbWQuY29tPiB3cm90ZToNCj4+ID4gPg0KPj4gPiA+ID4gSGkgSGVsZ2FhcywNCj4+ID4g
PiA+DQo+PiA+ID4gPiBTb21lIEFNRCBHUFVzIGhhdmUgaGFyZHdhcmUgc3VwcG9ydCBmb3IgZ3Jh
cGhpY3MgU1ItSU9WLg0KPj4gPiA+ID4gSWYgdGhlIFNSLUlPViBjYXBhYmxlIEdQVSBpcyBwbHVn
Z2VkIGludG8gdGhlIFNSLUlPViBpbmNhcGFibGUgDQo+PiA+ID4gPiBwbGF0Zm9ybS4gSXQgd291
bGQgY2F1c2UgYSBwcm9ibGVtIG9uIFBDSSByZXNvdXJjZSBhbGxvY2F0aW9uIA0KPj4gPiA+ID4g
aW4gY3VycmVudCBMaW51eCBrZXJuZWwuDQo+PiA+ID4gPg0KPj4gPiA+ID4gVGhlcmVmb3JlIGlu
IG9yZGVyIHRvIGFsbG93IHRoZSBQRiAoUGh5c2ljYWwgRnVuY3Rpb24pIGRldmljZSANCj4+ID4g
PiA+IG9mIFNSLUlPViBjYXBhYmxlIEdQVSB0byB3b3JrIG9uIHRoZSBTUi1JT1YgaW5jYXBhYmxl
IHBsYXRmb3JtLCANCj4+ID4gPiA+IGl0IGlzIHJlcXVpcmVkIHRvIHZlcmlmeSBjb25kaXRpb25z
IGZvciBpbml0aWFsaXppbmcgQkFSIA0KPj4gPiA+ID4gcmVzb3VyY2VzIG9uIEFNRCBTUi1JT1Yg
Y2FwYWJsZSBHUFVzLg0KPj4gPiA+ID4NCj4+ID4gPiA+IElmIHRoZSBkZXZpY2UgaXMgYW4gQU1E
IGdyYXBoaWNzIGRldmljZSBhbmQgaXQgc3VwcG9ydHMgU1ItSU9WIA0KPj4gPiA+ID4gaXQgd2ls
bCByZXF1aXJlIGEgbGFyZ2UgYW1vdW50IG9mIHJlc291cmNlcy4NCj4+ID4gPiA+IEJlZm9yZSBj
YWxsaW5nIHNyaW92X2luaXQoKSBtdXN0IGVuc3VyZSB0aGF0IHRoZSBzeXN0ZW0gQklPUyANCj4+
ID4gPiA+IGFsc28gc3VwcG9ydHMgU1ItSU9WIGFuZCB0aGF0IHN5c3RlbSBCSU9TIGhhcyBiZWVu
IGFibGUgdG8gDQo+PiA+ID4gPiBhbGxvY2F0ZSBlbm91Z2ggcmVzb3VyY2VzLg0KPj4gPiA+ID4g
SWYgdGhlIFZGIEJBUnMgYXJlIHplcm8gdGhlbiB0aGUgc3lzdGVtIEJJT1MgZG9lcyBub3Qgc3Vw
cG9ydCANCj4+ID4gPiA+IFNSLUlPViBvciBpdCBjb3VsZCBub3QgYWxsb2NhdGUgdGhlIHJlc291
cmNlcyBhbmQgdGhpcyBwbGF0Zm9ybSANCj4+ID4gPiA+IHdpbGwgbm90IHN1cHBvcnQgQU1EIGdy
YXBoaWNzIFNSLUlPVi4NCj4+ID4gPiA+IFRoZXJlZm9yZSBkbyBub3QgY2FsbCBzcmlvdl9pbml0
KCkuDQo+PiA+ID4gPiBJZiB0aGUgc3lzdGVtIEJJT1MgZG9lcyBzdXBwb3J0IFNSLUlPViB0aGVu
IHRoZSBWRiBCQVJzIHdpbGwgYmUgDQo+PiA+ID4gPiBwcm9wZXJseSBpbml0aWFsaXplZCB0byBu
b24temVybyB2YWx1ZXMuDQo+PiA+ID4gPg0KPj4gPiA+ID4gQmVsb3cgaXMgdGhlIHBhdGNoIGFn
YWluc3QgdG8gS2VybmVsIDQuOCAmIDQuOS4gUGxlYXNlIHJldmlldy4NCj4+ID4gPiA+DQo+PiA+
ID4gPiBJIGNoZWNrZWQgdGhlIGRyaXZlcnMvcGNpL3F1aXJrcy5jLCBpdCBsb29rcyB0aGUgDQo+
PiA+ID4gPiB3b3JrYXJvdW5kcy9maXhlcyBpbiBxdWlya3MuYyBhcmUgZm9yIHNwZWNpZmljIGRl
dmljZXMgYW5kIG9uZSANCj4+ID4gPiA+IG9yIG1vcmUgZGV2aWNlIElEIGFyZSBkZWZpbmVkIGZv
ciB0aGUgc3BlY2lmaWMgZGV2aWNlcy4gSG93ZXZlciANCj4+ID4gPiA+IG15IHBhdGNoIGlzIGZv
ciBhbGwgQU1EIFNSLUlPViBjYXBhYmxlIEdQVXMsIHRoYXQgaW5jbHVkZXMgYWxsIGV4aXN0aW5n
IGFuZCBmdXR1cmUgQU1EIHNlcnZlciBHUFVzLg0KPj4gPiA+ID4gU28gaXQgZG9lc24ndCBzZWVt
IGxpa2UgYSBnb29kIGZpdCB0byBwdXQgdGhlIGZpeCBpbiBxdWlya3MuYy4NCj4+ID4gPg0KPj4g
PiA+DQo+PiA+ID4gV2h5IGlzIGFuIEFNRCBncmFwaGljcyBjYXJkIHVuaXF1ZSBoZXJlPyAgRG9l
c24ndCBzcmlvdl9pbml0KCkgDQo+PiA+ID4gYWx3YXlzIG5lZWQgdG8gYmUgYWJsZSB0byBkZWFs
IHdpdGggZGV2aWNlcyBvZiBhbnkgdHlwZSB3aGVyZSB0aGUgDQo+PiA+ID4gQklPUyBoYXNuJ3Qg
aW5pdGlhbGl6ZWQgdGhlIFNSLUlPViBjYXBhYmlsaXR5PyAgU29tZSBTUi1JT1YgDQo+PiA+ID4g
ZGV2aWNlcyBjYW4gZml0IHRoZWlyIFZGcyB3aXRoaW4gYSBtaW5pbXVtIGJyaWRnZSBhcGVydHVy
ZSwgbW9zdCANCj4+ID4gPiBjYW5ub3QuICBJIGRvbid0IHVuZGVyc3RhbmQgd2h5IHRoZSBWRiBy
ZXNvdXJjZSByZXF1aXJlbWVudHMgDQo+PiA+ID4gYmVpbmcgZXhjZXB0aW9uYWxseSBsYXJnZSBk
aWN0YXRlcyB0aGF0IHRoZXkgcmVjZWl2ZSBzcGVjaWFsIGhhbmRsaW5nLg0KPj4gPiA+IFRoYW5r
cywNCj4+ID4gPg0KPj4gPiA+IEFsZXgNCj4+ID4gPg0KPj4gPiA+ID4gU2lnbmVkLW9mZi1ieTog
Q29sbGlucyBDaGVuZyA8Y29sbGlucy5jaGVuZ0BhbWQuY29tPg0KPj4gPiA+ID4gLS0tDQo+PiA+
ID4gPiAgZHJpdmVycy9wY2kvaW92LmMgfCA2Mw0KPj4gPiA+ID4gKysrKysrKysrKysrKysrKysr
KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKy0tLQ0KPj4gPiA+ID4gIDEgZmlsZSBj
aGFuZ2VkLCA2MCBpbnNlcnRpb25zKCspLCAzIGRlbGV0aW9ucygtKQ0KPj4gPiA+ID4NCj4+ID4g
PiA+IGRpZmYgLS1naXQgYS9kcml2ZXJzL3BjaS9pb3YuYyBiL2RyaXZlcnMvcGNpL2lvdi5jIGlu
ZGV4DQo+PiA+ID4gPiBlMzBmMDVjLi5lNGYxNDA1IDEwMDY0NA0KPj4gPiA+ID4gLS0tIGEvZHJp
dmVycy9wY2kvaW92LmMNCj4+ID4gPiA+ICsrKyBiL2RyaXZlcnMvcGNpL2lvdi5jDQo+PiA+ID4g
PiBAQCAtNTIzLDYgKzUyMyw0NSBAQCBzdGF0aWMgdm9pZCBzcmlvdl9yZXN0b3JlX3N0YXRlKHN0
cnVjdCBwY2lfZGV2ICpkZXYpDQo+PiA+ID4gPiAgICAgICAgICAgICAgICAgbXNsZWVwKDEwMCk7
DQo+PiA+ID4gPiAgfQ0KPj4gPiA+ID4NCj4+ID4gPiA+ICsvKg0KPj4gPiA+ID4gKyAqIHBjaV92
Zl9iYXJfdmFsaWQgLSBjaGVjayBpZiBWRiBCQVJzIGhhdmUgcmVzb3VyY2UgYWxsb2NhdGVkDQo+
PiA+ID4gPiArICogQGRldjogdGhlIFBDSSBkZXZpY2UNCj4+ID4gPiA+ICsgKiBAcG9zOiByZWdp
c3RlciBvZmZzZXQgb2YgU1ItSU9WIGNhcGFiaWxpdHkgaW4gUENJIGNvbmZpZyANCj4+ID4gPiA+
ICtzcGFjZQ0KPj4gPiA+ID4gKyAqIFJldHVybnMgdHJ1ZSBhbnkgVkYgQkFSIGhhcyByZXNvdXJj
ZSBhbGxvY2F0ZWQsIGZhbHNlDQo+PiA+ID4gPiArICogaWYgYWxsIFZGIEJBUnMgYXJlIGVtcHR5
Lg0KPj4gPiA+ID4gKyAqLw0KPj4gPiA+ID4gK3N0YXRpYyBib29sIHBjaV92Zl9iYXJfdmFsaWQo
c3RydWN0IHBjaV9kZXYgKmRldiwgaW50IHBvcykgew0KPj4gPiA+ID4gKyAgICAgICBpbnQgaTsN
Cj4+ID4gPiA+ICsgICAgICAgdTMyIGJhcl92YWx1ZTsNCj4+ID4gPiA+ICsgICAgICAgdTMyIGJh
cl9zaXplX21hc2sgPSB+KFBDSV9CQVNFX0FERFJFU1NfU1BBQ0UgfA0KPj4gPiA+ID4gKyAgICAg
ICAgICAgICAgICAgICAgICAgUENJX0JBU0VfQUREUkVTU19NRU1fVFlQRV82NCB8DQo+PiA+ID4g
PiArICAgICAgICAgICAgICAgICAgICAgICBQQ0lfQkFTRV9BRERSRVNTX01FTV9QUkVGRVRDSCk7
DQo+PiA+ID4gPiArDQo+PiA+ID4gPiArICAgICAgIGZvciAoaSA9IDA7IGkgPCBQQ0lfU1JJT1Zf
TlVNX0JBUlM7IGkrKykgew0KPj4gPiA+ID4gKyAgICAgICAgICAgICAgIHBjaV9yZWFkX2NvbmZp
Z19kd29yZChkZXYsIHBvcyArIFBDSV9TUklPVl9CQVIgKyBpICogNCwgJmJhcl92YWx1ZSk7DQo+
PiA+ID4gPiArICAgICAgICAgICAgICAgaWYgKGJhcl92YWx1ZSAmIGJhcl9zaXplX21hc2spDQo+
PiA+ID4gPiArICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4gdHJ1ZTsNCj4+ID4gPiA+ICsg
ICAgICAgfQ0KPj4gPiA+ID4gKw0KPj4gPiA+ID4gKyAgICAgICByZXR1cm4gZmFsc2U7DQo+PiA+
ID4gPiArfQ0KPj4gPiA+ID4gKw0KPj4gPiA+ID4gKy8qDQo+PiA+ID4gPiArICogaXNfYW1kX2Rp
c3BsYXlfYWRhcHRlciAtIGNoZWNrIGlmIGl0IGlzIGFuIEFNRC9BVEkgR1BVIA0KPj4gPiA+ID4g
K2RldmljZQ0KPj4gPiA+ID4gKyAqIEBkZXY6IHRoZSBQQ0kgZGV2aWNlDQo+PiA+ID4gPiArICoN
Cj4+ID4gPiA+ICsgKiBSZXR1cm5zIHRydWUgaWYgZGV2aWNlIGlzIGFuIEFNRC9BVEkgZGlzcGxh
eSBhZGFwdGVyLA0KPj4gPiA+ID4gKyAqIG90aGVyd2lzZSByZXR1cm4gZmFsc2UuDQo+PiA+ID4g
PiArICovDQo+PiA+ID4gPiArDQo+PiA+ID4gPiArc3RhdGljIGJvb2wgaXNfYW1kX2Rpc3BsYXlf
YWRhcHRlcihzdHJ1Y3QgcGNpX2RldiAqZGV2KSB7DQo+PiA+ID4gPiArICAgICAgIHJldHVybiAo
KChkZXYtPmNsYXNzID4+IDE2KSA9PSBQQ0lfQkFTRV9DTEFTU19ESVNQTEFZKSAmJg0KPj4gPiA+
ID4gKyAgICAgICAgICAgICAgIChkZXYtPnZlbmRvciA9PSBQQ0lfVkVORE9SX0lEX0FUSSB8fA0K
Pj4gPiA+ID4gKyAgICAgICAgICAgICAgIGRldi0+dmVuZG9yID09IFBDSV9WRU5ET1JfSURfQU1E
KSk7IH0NCj4+ID4gPiA+ICsNCj4+ID4gPiA+ICAvKioNCj4+ID4gPiA+ICAgKiBwY2lfaW92X2lu
aXQgLSBpbml0aWFsaXplIHRoZSBJT1YgY2FwYWJpbGl0eQ0KPj4gPiA+ID4gICAqIEBkZXY6IHRo
ZSBQQ0kgZGV2aWNlDQo+PiA+ID4gPiBAQCAtNTM3LDkgKzU3NiwyNyBAQCBpbnQgcGNpX2lvdl9p
bml0KHN0cnVjdCBwY2lfZGV2ICpkZXYpDQo+PiA+ID4gPiAgICAgICAgICAgICAgICAgcmV0dXJu
IC1FTk9ERVY7DQo+PiA+ID4gPg0KPj4gPiA+ID4gICAgICAgICBwb3MgPSBwY2lfZmluZF9leHRf
Y2FwYWJpbGl0eShkZXYsIFBDSV9FWFRfQ0FQX0lEX1NSSU9WKTsNCj4+ID4gPiA+IC0gICAgICAg
aWYgKHBvcykNCj4+ID4gPiA+IC0gICAgICAgICAgICAgICByZXR1cm4gc3Jpb3ZfaW5pdChkZXYs
IHBvcyk7DQo+PiA+ID4gPiAtDQo+PiA+ID4gPiArICAgICAgIGlmIChwb3MpIHsNCj4+ID4gPiA+
ICsgICAgICAgLyoNCj4+ID4gPiA+ICsgICAgICAgICogSWYgdGhlIGRldmljZSBpcyBhbiBBTUQg
Z3JhcGhpY3MgZGV2aWNlIGFuZCBpdCBzdXBwb3J0cw0KPj4gPiA+ID4gKyAgICAgICAgKiBTUi1J
T1YgaXQgd2lsbCByZXF1aXJlIGEgbGFyZ2UgYW1vdW50IG9mIHJlc291cmNlcy4NCj4+ID4gPiA+
ICsgICAgICAgICogQmVmb3JlIGNhbGxpbmcgc3Jpb3ZfaW5pdCgpIG11c3QgZW5zdXJlIHRoYXQg
dGhlIHN5c3RlbQ0KPj4gPiA+ID4gKyAgICAgICAgKiBCSU9TIGFsc28gc3VwcG9ydHMgU1ItSU9W
IGFuZCB0aGF0IHN5c3RlbSBCSU9TIGhhcyBiZWVuDQo+PiA+ID4gPiArICAgICAgICAqIGFibGUg
dG8gYWxsb2NhdGUgZW5vdWdoIHJlc291cmNlcy4NCj4+ID4gPiA+ICsgICAgICAgICogSWYgdGhl
IFZGIEJBUnMgYXJlIHplcm8gdGhlbiB0aGUgc3lzdGVtIEJJT1MgZG9lcyBub3QNCj4+ID4gPiA+
ICsgICAgICAgICogc3VwcG9ydCBTUi1JT1Ygb3IgaXQgY291bGQgbm90IGFsbG9jYXRlIHRoZSBy
ZXNvdXJjZXMNCj4+ID4gPiA+ICsgICAgICAgICogYW5kIHRoaXMgcGxhdGZvcm0gd2lsbCBub3Qg
c3VwcG9ydCBBTUQgZ3JhcGhpY3MgU1ItSU9WLg0KPj4gPiA+ID4gKyAgICAgICAgKiBUaGVyZWZv
cmUgZG8gbm90IGNhbGwgc3Jpb3ZfaW5pdCgpLg0KPj4gPiA+ID4gKyAgICAgICAgKiBJZiB0aGUg
c3lzdGVtIEJJT1MgZG9lcyBzdXBwb3J0IFNSLUlPViB0aGVuIHRoZSBWRiBCQVJzDQo+PiA+ID4g
PiArICAgICAgICAqIHdpbGwgYmUgcHJvcGVybHkgaW5pdGlhbGl6ZWQgdG8gbm9uLXplcm8gdmFs
dWVzLg0KPj4gPiA+ID4gKyAgICAgICAgKi8NCj4+ID4gPiA+ICsgICAgICAgICAgICAgICBpZiAo
aXNfYW1kX2Rpc3BsYXlfYWRhcHRlcihkZXYpKSB7DQo+PiA+ID4gPiArICAgICAgICAgICAgICAg
ICAgICAgICBpZiAocGNpX3ZmX2Jhcl92YWxpZChkZXYsIHBvcykpDQo+PiA+ID4gPiArICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVybiBzcmlvdl9pbml0KGRldiwgcG9zKTsNCj4+
ID4gPiA+ICsgICAgICAgICAgICAgICB9IGVsc2Ugew0KPj4gPiA+ID4gKyAgICAgICAgICAgICAg
ICAgICAgICAgcmV0dXJuIHNyaW92X2luaXQoZGV2LCBwb3MpOw0KPj4gPiA+ID4gKyAgICAgICAg
ICAgICAgIH0NCj4+ID4gPiA+ICsgICAgICAgfQ0KPj4gPiA+ID4gICAgICAgICByZXR1cm4gLUVO
T0RFVjsNCj4+ID4gPiA+ICB9DQo+PiA+ID4gPg0KPj4gPiA+DQo+PiA+DQo+Pg0KPg0K

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Zytaruk, Kelly]

>-----Original Message-----
>From: Cheng, Collins
>Sent: Saturday, May 20, 2017 12:53 AM
>To: Alexander Duyck; Alex Williamson
>Cc: Bjorn Helgaas; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
>Deucher, Alexander; Zytaruk, Kelly; Yinghai Lu
>Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>incapable platform
>
>Hi Alex,
>
>Yes, I hope kernel can disable SR-IOV and related VF resource allocation if the
>system BIOS is not SR-IOV capable.
>
>Adding the parameter "pci=nosriov" sounds a doable solution, but it would need
>user to add this parameter manually, right? I think an automatic detection would
>be better. My patch is trying to auto detect and bypass VF resource allocation.
>
>
>-Collins Cheng
>

Collins, be careful about this.  I don't think that this is what we want.  If you add "pci=nosriov" then you are globally disabling SRIOV for all devices.  This is not the solution that we are looking for.
Remember that there are 3 types of SBIOS; 
"not SR-IOV capable", 
"SR-IOV capable but does not support large resources", 
"Complete SR-IOV support".

The problem is that we are trying to find a fix for "broken" SBIOS that does support SR-IOV but does not support the full SR-IOV capabilities that devices with large resources require.

Thanks,
Kelly

>
>-----Original Message-----
>From: Alexander Duyck [mailto:alexander.duyck@gmail.com]
>Sent: Friday, May 19, 2017 11:44 PM
>To: Alex Williamson <alex.williamson@redhat.com>
>Cc: Cheng, Collins <Collins.Cheng@amd.com>; Bjorn Helgaas
><bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-
>kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>;
>Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
>Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>incapable platform
>
>On Mon, May 15, 2017 at 10:53 AM, Alex Williamson
><alex.williamson@redhat.com> wrote:
>> On Mon, 15 May 2017 08:19:28 +0000
>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>
>>> Hi Williamson,
>>>
>>> We cannot assume BIOS supports SR-IOV, actually only newer server
>motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older
>server motherboard BIOS doesn't support SR-IOV. This issue would happen if an
>user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.
>>
>> Servers should be supporting SR-IOV for a long time now.  What really
>> is there to a BIOS supporting SR-IOV anyway, it's simply reserving
>> sufficient bus number and MMIO resources such that we can enable the
>> VFs.  This process isn't exclusively reserved for the BIOS.  Some
>> platforms may choose to only initialize boot devices, leaving the rest
>> for the OS to program.  The initial proposal here to disable SR-IOV if
>> not programmed at OS hand-off disables even the possibility of the OS
>> reallocating resources for this device.
>
>There are differences between supporting SR-IOV and supporting SR-IOV on
>devices with massive resources. I know I have seen NICs that will keep a system
>from completing POST if SR-IOV is enabled, and MMIO beyond 4G is not. My
>guess would be that the issues being seen are probably that they disable SR-IOV in
>the BIOS in such a setup and end up running into issues when they try to boot into
>the Linux kernel as it goes through and tries to allocate resources for SR-IOV even
>though it was disabled in the BIOS.
>
>It might make sense to add a kernel parameter something like a "pci=nosriov"
>that would allow for disabling SR-IOV and related resource allocation if that is
>what we are talking about. That way you could plug in these types of devices into
>a system with a legacy bios or that doesn't wan to allocate addresses above 32b
>for MMIO, and this parameter would be all that is needed to disable SR-IOV so
>you could plug in a NIC that has SR-IOV associated with it.
>
>>> I agree that failure to allocate VF resources should leave the device in no
>worse condition than before it tried. I hope kernel could allocate PF device
>resource before allocating VF device resource, and keep PF device resource valid
>and functional if failed to allocate VF device resource.
>>>
>>> I will send out dmesg log lspci info tomorrow. Thanks.
>>
>> Thanks,
>> Alex
>>
>>> -----Original Message-----
>>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> Sent: Friday, May 12, 2017 10:43 PM
>>> To: Cheng, Collins <Collins.Cheng@amd.com>
>>> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>>> linux-kernel@vger.kernel.org; Deucher, Alexander
>>> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>;
>>> Yinghai Lu <yinghai@kernel.org>
>>> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>>> SR-IOV incapable platform
>>>
>>> On Fri, 12 May 2017 04:51:43 +0000
>>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>>
>>> > Hi Williamson,
>>> >
>>> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV
>NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(),
>it is safe and saving boot time, also there is no a better method to know if system
>BIOS has correctly initialized the SR-IOV capability or not.
>>>
>>> It also masks an underlying bug and creates a maintenance issue that we won't
>know when it's safe to remove this workaround.  I don't think faster boot is valid
>rationale, in one case SR-IOV is completely disabled, the other we attempt to
>allocate the resources the BIOS failed to provide.  I expect this is also a corner
>case, the BIOS should typically support SR-IOV, therefore this situation should be
>an exception.
>>>
>>> > I did not try to fix the issue from the kernel resource allocation perspective,
>it is because:
>>> >     1. I am not very familiar with the PCI resource allocation scheme in kernel.
>For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and
>PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is
>PF firstly, then this issue could be avoided.
>>> >     2. I am not sure if kernel has error handler if PCI resource allocation failed.
>In this case, kernel cannot allocate enough resource to PF. It should trigger some
>error handler to either just keep original BAR values set by system BIOS, or disable
>this device and log errors.
>>>
>>> I think these are the issues we should be trying to solve and I'm
>>> sure folks on the linux-pci list can help us identify the bug.
>>> Minimally, failure to allocate VF resources should leave the device
>>> in no worse condition than before it tried.  Perhaps you could post
>>> more details about the issue, boot with pci=earlydump, post dmesg of
>>> a boot where the PF resources are incorrectly re-allocated, and
>>> include lspci -vvv for the SR-IOV device.  Also, please test with the
>>> latest upstream kernel, upstream only patches old kernels through
>>> stable backports of commits to the latest kernel.  Adding Yinghai as
>>> a resource allocation expert. Thanks,
>>>
>>> Alex
>>>
>>> > -----Original Message-----
>>> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> > Sent: Friday, May 12, 2017 12:01 PM
>>> > To: Cheng, Collins <Collins.Cheng@amd.com>
>>> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>>> > linux-kernel@vger.kernel.org; Deucher, Alexander
>>> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
>>> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>>> > SR-IOV incapable platform
>>> >
>>> > On Fri, 12 May 2017 03:42:46 +0000
>>> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>> >
>>> > > Hi Williamson,
>>> > >
>>> > > GPU card needs more BAR aperture resource than other PCI devices. For
>example, Intel SR-IOV network card only require 512KB memory resource for all
>VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for
>frame buffer BAR aperture.
>>> > >
>>> > > If the system BIOS supports SR-IOV, it will reserve enough resource for all
>VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the
>enough resource for VF BARs, only PF BAR will be assigned and VF BARs are
>empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs
>are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The
>problem I saw is that kernel will fail to allocate PF BAR resource because some
>resources are assigned to VF, this is not expected. So kernel might need to do
>some check before re-assign the PF/VF resource, so that PF device will be
>correctly assigned BAR resource and user can use PF device.
>>> >
>>> > So the problem is that something bad happens when the kernel is
>>> > trying to reallocate resources in order to fulfill the requirements
>>> > of the VFs, leaving the PF resources incorrectly programmed?  Why
>>> > not just fix that bug rather than creating special handling for
>>> > this vendor/class of device which disables any attempt to fixup
>>> > resources for SR-IOV?  IOW, this patch just avoids the problem for
>>> > your devices rather than fixing the bug.  I'd suggest fixing the
>>> > bug such that the PF is left in a functional state if the kernel is
>>> > unable to allocate sufficient resources for the VFs.  Thanks,
>>> >
>>> > Alex
>>> >
>>> > > -----Original Message-----
>>> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> > > Sent: Friday, May 12, 2017 11:21 AM
>>> > > To: Cheng, Collins <Collins.Cheng@amd.com>
>>> > > Cc: Bjorn Helgaas <bhelgaas@google.com>;
>>> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher,
>>> > > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>>> > > <Kelly.Zytaruk@amd.com>
>>> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>>> > > SR-IOV incapable platform
>>> > >
>>> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins"
>>> > > <Collins.Cheng@amd.com> wrote:
>>> > >
>>> > > > Hi Helgaas,
>>> > > >
>>> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
>>> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable
>>> > > > platform. It would cause a problem on PCI resource allocation
>>> > > > in current Linux kernel.
>>> > > >
>>> > > > Therefore in order to allow the PF (Physical Function) device
>>> > > > of SR-IOV capable GPU to work on the SR-IOV incapable platform,
>>> > > > it is required to verify conditions for initializing BAR
>>> > > > resources on AMD SR-IOV capable GPUs.
>>> > > >
>>> > > > If the device is an AMD graphics device and it supports SR-IOV
>>> > > > it will require a large amount of resources.
>>> > > > Before calling sriov_init() must ensure that the system BIOS
>>> > > > also supports SR-IOV and that system BIOS has been able to
>>> > > > allocate enough resources.
>>> > > > If the VF BARs are zero then the system BIOS does not support
>>> > > > SR-IOV or it could not allocate the resources and this platform
>>> > > > will not support AMD graphics SR-IOV.
>>> > > > Therefore do not call sriov_init().
>>> > > > If the system BIOS does support SR-IOV then the VF BARs will be
>>> > > > properly initialized to non-zero values.
>>> > > >
>>> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
>>> > > >
>>> > > > I checked the drivers/pci/quirks.c, it looks the
>>> > > > workarounds/fixes in quirks.c are for specific devices and one
>>> > > > or more device ID are defined for the specific devices. However
>>> > > > my patch is for all AMD SR-IOV capable GPUs, that includes all existing
>and future AMD server GPUs.
>>> > > > So it doesn't seem like a good fit to put the fix in quirks.c.
>>> > >
>>> > >
>>> > > Why is an AMD graphics card unique here?  Doesn't sriov_init()
>>> > > always need to be able to deal with devices of any type where the
>>> > > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV
>>> > > devices can fit their VFs within a minimum bridge aperture, most
>>> > > cannot.  I don't understand why the VF resource requirements
>>> > > being exceptionally large dictates that they receive special handling.
>>> > > Thanks,
>>> > >
>>> > > Alex
>>> > >
>>> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>>> > > > ---
>>> > > >  drivers/pci/iov.c | 63
>>> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>>> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
>>> > > >
>>> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>>> > > > e30f05c..e4f1405 100644
>>> > > > --- a/drivers/pci/iov.c
>>> > > > +++ b/drivers/pci/iov.c
>>> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev
>*dev)
>>> > > >                 msleep(100);
>>> > > >  }
>>> > > >
>>> > > > +/*
>>> > > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
>>> > > > + * @dev: the PCI device
>>> > > > + * @pos: register offset of SR-IOV capability in PCI config
>>> > > > +space
>>> > > > + * Returns true any VF BAR has resource allocated, false
>>> > > > + * if all VF BARs are empty.
>>> > > > + */
>>> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>>> > > > +       int i;
>>> > > > +       u32 bar_value;
>>> > > > +       u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
>>> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
>>> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
>>> > > > +
>>> > > > +       for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>>> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4,
>&bar_value);
>>> > > > +               if (bar_value & bar_size_mask)
>>> > > > +                       return true;
>>> > > > +       }
>>> > > > +
>>> > > > +       return false;
>>> > > > +}
>>> > > > +
>>> > > > +/*
>>> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU
>>> > > > +device
>>> > > > + * @dev: the PCI device
>>> > > > + *
>>> > > > + * Returns true if device is an AMD/ATI display adapter,
>>> > > > + * otherwise return false.
>>> > > > + */
>>> > > > +
>>> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
>>> > > > +       return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
>>> > > > +               (dev->vendor == PCI_VENDOR_ID_ATI ||
>>> > > > +               dev->vendor == PCI_VENDOR_ID_AMD)); }
>>> > > > +
>>> > > >  /**
>>> > > >   * pci_iov_init - initialize the IOV capability
>>> > > >   * @dev: the PCI device
>>> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>>> > > >                 return -ENODEV;
>>> > > >
>>> > > >         pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
>>> > > > -       if (pos)
>>> > > > -               return sriov_init(dev, pos);
>>> > > > -
>>> > > > +       if (pos) {
>>> > > > +       /*
>>> > > > +        * If the device is an AMD graphics device and it supports
>>> > > > +        * SR-IOV it will require a large amount of resources.
>>> > > > +        * Before calling sriov_init() must ensure that the system
>>> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
>>> > > > +        * able to allocate enough resources.
>>> > > > +        * If the VF BARs are zero then the system BIOS does not
>>> > > > +        * support SR-IOV or it could not allocate the resources
>>> > > > +        * and this platform will not support AMD graphics SR-IOV.
>>> > > > +        * Therefore do not call sriov_init().
>>> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
>>> > > > +        * will be properly initialized to non-zero values.
>>> > > > +        */
>>> > > > +               if (is_amd_display_adapter(dev)) {
>>> > > > +                       if (pci_vf_bar_valid(dev, pos))
>>> > > > +                               return sriov_init(dev, pos);
>>> > > > +               } else {
>>> > > > +                       return sriov_init(dev, pos);
>>> > > > +               }
>>> > > > +       }
>>> > > >         return -ENODEV;
>>> > > >  }
>>> > > >
>>> > >
>>> >
>>>
>>

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Zytaruk, Kelly]

DQoNCj4tLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPkZyb206IENoZW5nLCBDb2xsaW5zDQo+
U2VudDogU2F0dXJkYXksIE1heSAyMCwgMjAxNyAxMjo1MyBBTQ0KPlRvOiBBbGV4YW5kZXIgRHV5
Y2s7IEFsZXggV2lsbGlhbXNvbg0KPkNjOiBCam9ybiBIZWxnYWFzOyBsaW51eC1wY2lAdmdlci5r
ZXJuZWwub3JnOyBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnOw0KPkRldWNoZXIsIEFsZXhh
bmRlcjsgWnl0YXJ1aywgS2VsbHk7IFlpbmdoYWkgTHUNCj5TdWJqZWN0OiBSRTogW1BBVENIXSBQ
Q0k6IE1ha2UgU1ItSU9WIGNhcGFibGUgR1BVIHdvcmtpbmcgb24gdGhlIFNSLUlPVg0KPmluY2Fw
YWJsZSBwbGF0Zm9ybQ0KPg0KPkhpIEFsZXgsDQo+DQo+WWVzLCBJIGhvcGUga2VybmVsIGNhbiBk
aXNhYmxlIFNSLUlPViBhbmQgcmVsYXRlZCBWRiByZXNvdXJjZSBhbGxvY2F0aW9uIGlmIHRoZQ0K
PnN5c3RlbSBCSU9TIGlzIG5vdCBTUi1JT1YgY2FwYWJsZS4NCj4NCj5BZGRpbmcgdGhlIHBhcmFt
ZXRlciAicGNpPW5vc3Jpb3YiIHNvdW5kcyBhIGRvYWJsZSBzb2x1dGlvbiwgYnV0IGl0IHdvdWxk
IG5lZWQNCj51c2VyIHRvIGFkZCB0aGlzIHBhcmFtZXRlciBtYW51YWxseSwgcmlnaHQ/IEkgdGhp
bmsgYW4gYXV0b21hdGljIGRldGVjdGlvbiB3b3VsZA0KPmJlIGJldHRlci4gTXkgcGF0Y2ggaXMg
dHJ5aW5nIHRvIGF1dG8gZGV0ZWN0IGFuZCBieXBhc3MgVkYgcmVzb3VyY2UgYWxsb2NhdGlvbi4N
Cj4NCj4NCj4tQ29sbGlucyBDaGVuZw0KPg0KDQpDb2xsaW5zLCBiZSBjYXJlZnVsIGFib3V0IHRo
aXMuICBJIGRvbid0IHRoaW5rIHRoYXQgdGhpcyBpcyB3aGF0IHdlIHdhbnQuICBJZiB5b3UgYWRk
ICJwY2k9bm9zcmlvdiIgdGhlbiB5b3UgYXJlIGdsb2JhbGx5IGRpc2FibGluZyBTUklPViBmb3Ig
YWxsIGRldmljZXMuICBUaGlzIGlzIG5vdCB0aGUgc29sdXRpb24gdGhhdCB3ZSBhcmUgbG9va2lu
ZyBmb3IuDQpSZW1lbWJlciB0aGF0IHRoZXJlIGFyZSAzIHR5cGVzIG9mIFNCSU9TOyANCiJub3Qg
U1ItSU9WIGNhcGFibGUiLCANCiJTUi1JT1YgY2FwYWJsZSBidXQgZG9lcyBub3Qgc3VwcG9ydCBs
YXJnZSByZXNvdXJjZXMiLCANCiJDb21wbGV0ZSBTUi1JT1Ygc3VwcG9ydCIuDQoNClRoZSBwcm9i
bGVtIGlzIHRoYXQgd2UgYXJlIHRyeWluZyB0byBmaW5kIGEgZml4IGZvciAiYnJva2VuIiBTQklP
UyB0aGF0IGRvZXMgc3VwcG9ydCBTUi1JT1YgYnV0IGRvZXMgbm90IHN1cHBvcnQgdGhlIGZ1bGwg
U1ItSU9WIGNhcGFiaWxpdGllcyB0aGF0IGRldmljZXMgd2l0aCBsYXJnZSByZXNvdXJjZXMgcmVx
dWlyZS4NCg0KVGhhbmtzLA0KS2VsbHkNCg0KPg0KPi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0t
DQo+RnJvbTogQWxleGFuZGVyIER1eWNrIFttYWlsdG86YWxleGFuZGVyLmR1eWNrQGdtYWlsLmNv
bV0NCj5TZW50OiBGcmlkYXksIE1heSAxOSwgMjAxNyAxMTo0NCBQTQ0KPlRvOiBBbGV4IFdpbGxp
YW1zb24gPGFsZXgud2lsbGlhbXNvbkByZWRoYXQuY29tPg0KPkNjOiBDaGVuZywgQ29sbGlucyA8
Q29sbGlucy5DaGVuZ0BhbWQuY29tPjsgQmpvcm4gSGVsZ2Fhcw0KPjxiaGVsZ2Fhc0Bnb29nbGUu
Y29tPjsgbGludXgtcGNpQHZnZXIua2VybmVsLm9yZzsgbGludXgtDQo+a2VybmVsQHZnZXIua2Vy
bmVsLm9yZzsgRGV1Y2hlciwgQWxleGFuZGVyIDxBbGV4YW5kZXIuRGV1Y2hlckBhbWQuY29tPjsN
Cj5aeXRhcnVrLCBLZWxseSA8S2VsbHkuWnl0YXJ1a0BhbWQuY29tPjsgWWluZ2hhaSBMdSA8eWlu
Z2hhaUBrZXJuZWwub3JnPg0KPlN1YmplY3Q6IFJlOiBbUEFUQ0hdIFBDSTogTWFrZSBTUi1JT1Yg
Y2FwYWJsZSBHUFUgd29ya2luZyBvbiB0aGUgU1ItSU9WDQo+aW5jYXBhYmxlIHBsYXRmb3JtDQo+
DQo+T24gTW9uLCBNYXkgMTUsIDIwMTcgYXQgMTA6NTMgQU0sIEFsZXggV2lsbGlhbXNvbg0KPjxh
bGV4LndpbGxpYW1zb25AcmVkaGF0LmNvbT4gd3JvdGU6DQo+PiBPbiBNb24sIDE1IE1heSAyMDE3
IDA4OjE5OjI4ICswMDAwDQo+PiAiQ2hlbmcsIENvbGxpbnMiIDxDb2xsaW5zLkNoZW5nQGFtZC5j
b20+IHdyb3RlOg0KPj4NCj4+PiBIaSBXaWxsaWFtc29uLA0KPj4+DQo+Pj4gV2UgY2Fubm90IGFz
c3VtZSBCSU9TIHN1cHBvcnRzIFNSLUlPViwgYWN0dWFsbHkgb25seSBuZXdlciBzZXJ2ZXINCj5t
b3RoZXJib2FyZCBCSU9TIHN1cHBvcnRzIFNSLUlPVi4gTm9ybWFsIGRlc2t0b3AgbW90aGVyYm9h
cmQgQklPUyBvciBvbGRlcg0KPnNlcnZlciBtb3RoZXJib2FyZCBCSU9TIGRvZXNuJ3Qgc3VwcG9y
dCBTUi1JT1YuIFRoaXMgaXNzdWUgd291bGQgaGFwcGVuIGlmIGFuDQo+dXNlciBwbHVncyBvdXIg
QU1EIFNSLUlPViBjYXBhYmxlIEdQVSBjYXJkIHRvIGEgbm9ybWFsIGRlc2t0b3AgbW90aGVyYm9h
cmQuDQo+Pg0KPj4gU2VydmVycyBzaG91bGQgYmUgc3VwcG9ydGluZyBTUi1JT1YgZm9yIGEgbG9u
ZyB0aW1lIG5vdy4gIFdoYXQgcmVhbGx5DQo+PiBpcyB0aGVyZSB0byBhIEJJT1Mgc3VwcG9ydGlu
ZyBTUi1JT1YgYW55d2F5LCBpdCdzIHNpbXBseSByZXNlcnZpbmcNCj4+IHN1ZmZpY2llbnQgYnVz
IG51bWJlciBhbmQgTU1JTyByZXNvdXJjZXMgc3VjaCB0aGF0IHdlIGNhbiBlbmFibGUgdGhlDQo+
PiBWRnMuICBUaGlzIHByb2Nlc3MgaXNuJ3QgZXhjbHVzaXZlbHkgcmVzZXJ2ZWQgZm9yIHRoZSBC
SU9TLiAgU29tZQ0KPj4gcGxhdGZvcm1zIG1heSBjaG9vc2UgdG8gb25seSBpbml0aWFsaXplIGJv
b3QgZGV2aWNlcywgbGVhdmluZyB0aGUgcmVzdA0KPj4gZm9yIHRoZSBPUyB0byBwcm9ncmFtLiAg
VGhlIGluaXRpYWwgcHJvcG9zYWwgaGVyZSB0byBkaXNhYmxlIFNSLUlPViBpZg0KPj4gbm90IHBy
b2dyYW1tZWQgYXQgT1MgaGFuZC1vZmYgZGlzYWJsZXMgZXZlbiB0aGUgcG9zc2liaWxpdHkgb2Yg
dGhlIE9TDQo+PiByZWFsbG9jYXRpbmcgcmVzb3VyY2VzIGZvciB0aGlzIGRldmljZS4NCj4NCj5U
aGVyZSBhcmUgZGlmZmVyZW5jZXMgYmV0d2VlbiBzdXBwb3J0aW5nIFNSLUlPViBhbmQgc3VwcG9y
dGluZyBTUi1JT1Ygb24NCj5kZXZpY2VzIHdpdGggbWFzc2l2ZSByZXNvdXJjZXMuIEkga25vdyBJ
IGhhdmUgc2VlbiBOSUNzIHRoYXQgd2lsbCBrZWVwIGEgc3lzdGVtDQo+ZnJvbSBjb21wbGV0aW5n
IFBPU1QgaWYgU1ItSU9WIGlzIGVuYWJsZWQsIGFuZCBNTUlPIGJleW9uZCA0RyBpcyBub3QuIE15
DQo+Z3Vlc3Mgd291bGQgYmUgdGhhdCB0aGUgaXNzdWVzIGJlaW5nIHNlZW4gYXJlIHByb2JhYmx5
IHRoYXQgdGhleSBkaXNhYmxlIFNSLUlPViBpbg0KPnRoZSBCSU9TIGluIHN1Y2ggYSBzZXR1cCBh
bmQgZW5kIHVwIHJ1bm5pbmcgaW50byBpc3N1ZXMgd2hlbiB0aGV5IHRyeSB0byBib290IGludG8N
Cj50aGUgTGludXgga2VybmVsIGFzIGl0IGdvZXMgdGhyb3VnaCBhbmQgdHJpZXMgdG8gYWxsb2Nh
dGUgcmVzb3VyY2VzIGZvciBTUi1JT1YgZXZlbg0KPnRob3VnaCBpdCB3YXMgZGlzYWJsZWQgaW4g
dGhlIEJJT1MuDQo+DQo+SXQgbWlnaHQgbWFrZSBzZW5zZSB0byBhZGQgYSBrZXJuZWwgcGFyYW1l
dGVyIHNvbWV0aGluZyBsaWtlIGEgInBjaT1ub3NyaW92Ig0KPnRoYXQgd291bGQgYWxsb3cgZm9y
IGRpc2FibGluZyBTUi1JT1YgYW5kIHJlbGF0ZWQgcmVzb3VyY2UgYWxsb2NhdGlvbiBpZiB0aGF0
IGlzDQo+d2hhdCB3ZSBhcmUgdGFsa2luZyBhYm91dC4gVGhhdCB3YXkgeW91IGNvdWxkIHBsdWcg
aW4gdGhlc2UgdHlwZXMgb2YgZGV2aWNlcyBpbnRvDQo+YSBzeXN0ZW0gd2l0aCBhIGxlZ2FjeSBi
aW9zIG9yIHRoYXQgZG9lc24ndCB3YW4gdG8gYWxsb2NhdGUgYWRkcmVzc2VzIGFib3ZlIDMyYg0K
PmZvciBNTUlPLCBhbmQgdGhpcyBwYXJhbWV0ZXIgd291bGQgYmUgYWxsIHRoYXQgaXMgbmVlZGVk
IHRvIGRpc2FibGUgU1ItSU9WIHNvDQo+eW91IGNvdWxkIHBsdWcgaW4gYSBOSUMgdGhhdCBoYXMg
U1ItSU9WIGFzc29jaWF0ZWQgd2l0aCBpdC4NCj4NCj4+PiBJIGFncmVlIHRoYXQgZmFpbHVyZSB0
byBhbGxvY2F0ZSBWRiByZXNvdXJjZXMgc2hvdWxkIGxlYXZlIHRoZSBkZXZpY2UgaW4gbm8NCj53
b3JzZSBjb25kaXRpb24gdGhhbiBiZWZvcmUgaXQgdHJpZWQuIEkgaG9wZSBrZXJuZWwgY291bGQg
YWxsb2NhdGUgUEYgZGV2aWNlDQo+cmVzb3VyY2UgYmVmb3JlIGFsbG9jYXRpbmcgVkYgZGV2aWNl
IHJlc291cmNlLCBhbmQga2VlcCBQRiBkZXZpY2UgcmVzb3VyY2UgdmFsaWQNCj5hbmQgZnVuY3Rp
b25hbCBpZiBmYWlsZWQgdG8gYWxsb2NhdGUgVkYgZGV2aWNlIHJlc291cmNlLg0KPj4+DQo+Pj4g
SSB3aWxsIHNlbmQgb3V0IGRtZXNnIGxvZyBsc3BjaSBpbmZvIHRvbW9ycm93LiBUaGFua3MuDQo+
Pg0KPj4gVGhhbmtzLA0KPj4gQWxleA0KPj4NCj4+PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0t
LQ0KPj4+IEZyb206IEFsZXggV2lsbGlhbXNvbiBbbWFpbHRvOmFsZXgud2lsbGlhbXNvbkByZWRo
YXQuY29tXQ0KPj4+IFNlbnQ6IEZyaWRheSwgTWF5IDEyLCAyMDE3IDEwOjQzIFBNDQo+Pj4gVG86
IENoZW5nLCBDb2xsaW5zIDxDb2xsaW5zLkNoZW5nQGFtZC5jb20+DQo+Pj4gQ2M6IEJqb3JuIEhl
bGdhYXMgPGJoZWxnYWFzQGdvb2dsZS5jb20+OyBsaW51eC1wY2lAdmdlci5rZXJuZWwub3JnOw0K
Pj4+IGxpbnV4LWtlcm5lbEB2Z2VyLmtlcm5lbC5vcmc7IERldWNoZXIsIEFsZXhhbmRlcg0KPj4+
IDxBbGV4YW5kZXIuRGV1Y2hlckBhbWQuY29tPjsgWnl0YXJ1aywgS2VsbHkgPEtlbGx5Llp5dGFy
dWtAYW1kLmNvbT47DQo+Pj4gWWluZ2hhaSBMdSA8eWluZ2hhaUBrZXJuZWwub3JnPg0KPj4+IFN1
YmplY3Q6IFJlOiBbUEFUQ0hdIFBDSTogTWFrZSBTUi1JT1YgY2FwYWJsZSBHUFUgd29ya2luZyBv
biB0aGUNCj4+PiBTUi1JT1YgaW5jYXBhYmxlIHBsYXRmb3JtDQo+Pj4NCj4+PiBPbiBGcmksIDEy
IE1heSAyMDE3IDA0OjUxOjQzICswMDAwDQo+Pj4gIkNoZW5nLCBDb2xsaW5zIiA8Q29sbGlucy5D
aGVuZ0BhbWQuY29tPiB3cm90ZToNCj4+Pg0KPj4+ID4gSGkgV2lsbGlhbXNvbiwNCj4+PiA+DQo+
Pj4gPiBJIHZlcmlmaWVkIHRoZSBwYXRjaCBpcyB3b3JraW5nIGZvciBib3RoIEFNRCBTUi1JT1Yg
R1BVIGFuZCBJbnRlbCBTUi1JT1YNCj5OSUMuIEkgZG9uJ3QgdGhpbmsgaXQgaXMgcmVkdW5kYW50
IHRvIGNoZWNrIHRoZSBWRiBCQVIgdmFsaWQgYmVmb3JlIGNhbGwgc3Jpb3ZfaW5pdCgpLA0KPml0
IGlzIHNhZmUgYW5kIHNhdmluZyBib290IHRpbWUsIGFsc28gdGhlcmUgaXMgbm8gYSBiZXR0ZXIg
bWV0aG9kIHRvIGtub3cgaWYgc3lzdGVtDQo+QklPUyBoYXMgY29ycmVjdGx5IGluaXRpYWxpemVk
IHRoZSBTUi1JT1YgY2FwYWJpbGl0eSBvciBub3QuDQo+Pj4NCj4+PiBJdCBhbHNvIG1hc2tzIGFu
IHVuZGVybHlpbmcgYnVnIGFuZCBjcmVhdGVzIGEgbWFpbnRlbmFuY2UgaXNzdWUgdGhhdCB3ZSB3
b24ndA0KPmtub3cgd2hlbiBpdCdzIHNhZmUgdG8gcmVtb3ZlIHRoaXMgd29ya2Fyb3VuZC4gIEkg
ZG9uJ3QgdGhpbmsgZmFzdGVyIGJvb3QgaXMgdmFsaWQNCj5yYXRpb25hbGUsIGluIG9uZSBjYXNl
IFNSLUlPViBpcyBjb21wbGV0ZWx5IGRpc2FibGVkLCB0aGUgb3RoZXIgd2UgYXR0ZW1wdCB0bw0K
PmFsbG9jYXRlIHRoZSByZXNvdXJjZXMgdGhlIEJJT1MgZmFpbGVkIHRvIHByb3ZpZGUuICBJIGV4
cGVjdCB0aGlzIGlzIGFsc28gYSBjb3JuZXINCj5jYXNlLCB0aGUgQklPUyBzaG91bGQgdHlwaWNh
bGx5IHN1cHBvcnQgU1ItSU9WLCB0aGVyZWZvcmUgdGhpcyBzaXR1YXRpb24gc2hvdWxkIGJlDQo+
YW4gZXhjZXB0aW9uLg0KPj4+DQo+Pj4gPiBJIGRpZCBub3QgdHJ5IHRvIGZpeCB0aGUgaXNzdWUg
ZnJvbSB0aGUga2VybmVsIHJlc291cmNlIGFsbG9jYXRpb24gcGVyc3BlY3RpdmUsDQo+aXQgaXMg
YmVjYXVzZToNCj4+PiA+ICAgICAxLiBJIGFtIG5vdCB2ZXJ5IGZhbWlsaWFyIHdpdGggdGhlIFBD
SSByZXNvdXJjZSBhbGxvY2F0aW9uIHNjaGVtZSBpbiBrZXJuZWwuDQo+Rm9yIGV4YW1wbGUsIGlu
IHNyaW92X2luaXQoKSwga2VybmVsIHdpbGwgcmUtYXNzaWduIHRoZSBQQ0kgcmVzb3VyY2UgZm9y
IGJvdGggVkYgYW5kDQo+UEYuIEkgZG9uJ3QgdW5kZXJzdGFuZCB3aHkga2VybmVsIGFsbG9jYXRl
cyByZXNvdXJjZSBmb3IgVkYgZmlyc3RseSwgdGhlbiBQRi4gSWYgaXQgaXMNCj5QRiBmaXJzdGx5
LCB0aGVuIHRoaXMgaXNzdWUgY291bGQgYmUgYXZvaWRlZC4NCj4+PiA+ICAgICAyLiBJIGFtIG5v
dCBzdXJlIGlmIGtlcm5lbCBoYXMgZXJyb3IgaGFuZGxlciBpZiBQQ0kgcmVzb3VyY2UgYWxsb2Nh
dGlvbiBmYWlsZWQuDQo+SW4gdGhpcyBjYXNlLCBrZXJuZWwgY2Fubm90IGFsbG9jYXRlIGVub3Vn
aCByZXNvdXJjZSB0byBQRi4gSXQgc2hvdWxkIHRyaWdnZXIgc29tZQ0KPmVycm9yIGhhbmRsZXIg
dG8gZWl0aGVyIGp1c3Qga2VlcCBvcmlnaW5hbCBCQVIgdmFsdWVzIHNldCBieSBzeXN0ZW0gQklP
Uywgb3IgZGlzYWJsZQ0KPnRoaXMgZGV2aWNlIGFuZCBsb2cgZXJyb3JzLg0KPj4+DQo+Pj4gSSB0
aGluayB0aGVzZSBhcmUgdGhlIGlzc3VlcyB3ZSBzaG91bGQgYmUgdHJ5aW5nIHRvIHNvbHZlIGFu
ZCBJJ20NCj4+PiBzdXJlIGZvbGtzIG9uIHRoZSBsaW51eC1wY2kgbGlzdCBjYW4gaGVscCB1cyBp
ZGVudGlmeSB0aGUgYnVnLg0KPj4+IE1pbmltYWxseSwgZmFpbHVyZSB0byBhbGxvY2F0ZSBWRiBy
ZXNvdXJjZXMgc2hvdWxkIGxlYXZlIHRoZSBkZXZpY2UNCj4+PiBpbiBubyB3b3JzZSBjb25kaXRp
b24gdGhhbiBiZWZvcmUgaXQgdHJpZWQuICBQZXJoYXBzIHlvdSBjb3VsZCBwb3N0DQo+Pj4gbW9y
ZSBkZXRhaWxzIGFib3V0IHRoZSBpc3N1ZSwgYm9vdCB3aXRoIHBjaT1lYXJseWR1bXAsIHBvc3Qg
ZG1lc2cgb2YNCj4+PiBhIGJvb3Qgd2hlcmUgdGhlIFBGIHJlc291cmNlcyBhcmUgaW5jb3JyZWN0
bHkgcmUtYWxsb2NhdGVkLCBhbmQNCj4+PiBpbmNsdWRlIGxzcGNpIC12dnYgZm9yIHRoZSBTUi1J
T1YgZGV2aWNlLiAgQWxzbywgcGxlYXNlIHRlc3Qgd2l0aCB0aGUNCj4+PiBsYXRlc3QgdXBzdHJl
YW0ga2VybmVsLCB1cHN0cmVhbSBvbmx5IHBhdGNoZXMgb2xkIGtlcm5lbHMgdGhyb3VnaA0KPj4+
IHN0YWJsZSBiYWNrcG9ydHMgb2YgY29tbWl0cyB0byB0aGUgbGF0ZXN0IGtlcm5lbC4gIEFkZGlu
ZyBZaW5naGFpIGFzDQo+Pj4gYSByZXNvdXJjZSBhbGxvY2F0aW9uIGV4cGVydC4gVGhhbmtzLA0K
Pj4+DQo+Pj4gQWxleA0KPj4+DQo+Pj4gPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPj4+
ID4gRnJvbTogQWxleCBXaWxsaWFtc29uIFttYWlsdG86YWxleC53aWxsaWFtc29uQHJlZGhhdC5j
b21dDQo+Pj4gPiBTZW50OiBGcmlkYXksIE1heSAxMiwgMjAxNyAxMjowMSBQTQ0KPj4+ID4gVG86
IENoZW5nLCBDb2xsaW5zIDxDb2xsaW5zLkNoZW5nQGFtZC5jb20+DQo+Pj4gPiBDYzogQmpvcm4g
SGVsZ2FhcyA8YmhlbGdhYXNAZ29vZ2xlLmNvbT47IGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc7
DQo+Pj4gPiBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnOyBEZXVjaGVyLCBBbGV4YW5kZXIN
Cj4+PiA+IDxBbGV4YW5kZXIuRGV1Y2hlckBhbWQuY29tPjsgWnl0YXJ1aywgS2VsbHkgPEtlbGx5
Llp5dGFydWtAYW1kLmNvbT4NCj4+PiA+IFN1YmplY3Q6IFJlOiBbUEFUQ0hdIFBDSTogTWFrZSBT
Ui1JT1YgY2FwYWJsZSBHUFUgd29ya2luZyBvbiB0aGUNCj4+PiA+IFNSLUlPViBpbmNhcGFibGUg
cGxhdGZvcm0NCj4+PiA+DQo+Pj4gPiBPbiBGcmksIDEyIE1heSAyMDE3IDAzOjQyOjQ2ICswMDAw
DQo+Pj4gPiAiQ2hlbmcsIENvbGxpbnMiIDxDb2xsaW5zLkNoZW5nQGFtZC5jb20+IHdyb3RlOg0K
Pj4+ID4NCj4+PiA+ID4gSGkgV2lsbGlhbXNvbiwNCj4+PiA+ID4NCj4+PiA+ID4gR1BVIGNhcmQg
bmVlZHMgbW9yZSBCQVIgYXBlcnR1cmUgcmVzb3VyY2UgdGhhbiBvdGhlciBQQ0kgZGV2aWNlcy4g
Rm9yDQo+ZXhhbXBsZSwgSW50ZWwgU1ItSU9WIG5ldHdvcmsgY2FyZCBvbmx5IHJlcXVpcmUgNTEy
S0IgbWVtb3J5IHJlc291cmNlIGZvciBhbGwNCj5WRnMuIEFNRCBTUi1JT1YgR1BVIGNhcmQgbmVl
ZHMgMjU2TUIgeDE2IFZGID0gNEdCIG1lbW9yeSByZXNvdXJjZSBmb3INCj5mcmFtZSBidWZmZXIg
QkFSIGFwZXJ0dXJlLg0KPj4+ID4gPg0KPj4+ID4gPiBJZiB0aGUgc3lzdGVtIEJJT1Mgc3VwcG9y
dHMgU1ItSU9WLCBpdCB3aWxsIHJlc2VydmUgZW5vdWdoIHJlc291cmNlIGZvciBhbGwNCj5WRiBC
QVJzLiBJZiB0aGUgc3lzdGVtIEJJT1MgZG9lc24ndCBzdXBwb3J0IFNSLUlPViBvciBjYW5ub3Qg
YWxsb2NhdGUgdGhlDQo+ZW5vdWdoIHJlc291cmNlIGZvciBWRiBCQVJzLCBvbmx5IFBGIEJBUiB3
aWxsIGJlIGFzc2lnbmVkIGFuZCBWRiBCQVJzIGFyZQ0KPmVtcHR5LiBUaGVuIHN5c3RlbSBib290
cyB0byBMaW51eCBrZXJuZWwgYW5kIGtlcm5lbCBkb2Vzbid0IGNoZWNrIGlmIHRoZSBWRiBCQVJz
DQo+YXJlIGVtcHR5IG9yIHZhbGlkLiBLZXJuZWwgd2lsbCByZS1hc3NpZ24gdGhlIEJBUiByZXNv
dXJjZXMgZm9yIFBGIGFuZCBhbGwgVkZzLiBUaGUNCj5wcm9ibGVtIEkgc2F3IGlzIHRoYXQga2Vy
bmVsIHdpbGwgZmFpbCB0byBhbGxvY2F0ZSBQRiBCQVIgcmVzb3VyY2UgYmVjYXVzZSBzb21lDQo+
cmVzb3VyY2VzIGFyZSBhc3NpZ25lZCB0byBWRiwgdGhpcyBpcyBub3QgZXhwZWN0ZWQuIFNvIGtl
cm5lbCBtaWdodCBuZWVkIHRvIGRvDQo+c29tZSBjaGVjayBiZWZvcmUgcmUtYXNzaWduIHRoZSBQ
Ri9WRiByZXNvdXJjZSwgc28gdGhhdCBQRiBkZXZpY2Ugd2lsbCBiZQ0KPmNvcnJlY3RseSBhc3Np
Z25lZCBCQVIgcmVzb3VyY2UgYW5kIHVzZXIgY2FuIHVzZSBQRiBkZXZpY2UuDQo+Pj4gPg0KPj4+
ID4gU28gdGhlIHByb2JsZW0gaXMgdGhhdCBzb21ldGhpbmcgYmFkIGhhcHBlbnMgd2hlbiB0aGUg
a2VybmVsIGlzDQo+Pj4gPiB0cnlpbmcgdG8gcmVhbGxvY2F0ZSByZXNvdXJjZXMgaW4gb3JkZXIg
dG8gZnVsZmlsbCB0aGUgcmVxdWlyZW1lbnRzDQo+Pj4gPiBvZiB0aGUgVkZzLCBsZWF2aW5nIHRo
ZSBQRiByZXNvdXJjZXMgaW5jb3JyZWN0bHkgcHJvZ3JhbW1lZD8gIFdoeQ0KPj4+ID4gbm90IGp1
c3QgZml4IHRoYXQgYnVnIHJhdGhlciB0aGFuIGNyZWF0aW5nIHNwZWNpYWwgaGFuZGxpbmcgZm9y
DQo+Pj4gPiB0aGlzIHZlbmRvci9jbGFzcyBvZiBkZXZpY2Ugd2hpY2ggZGlzYWJsZXMgYW55IGF0
dGVtcHQgdG8gZml4dXANCj4+PiA+IHJlc291cmNlcyBmb3IgU1ItSU9WPyAgSU9XLCB0aGlzIHBh
dGNoIGp1c3QgYXZvaWRzIHRoZSBwcm9ibGVtIGZvcg0KPj4+ID4geW91ciBkZXZpY2VzIHJhdGhl
ciB0aGFuIGZpeGluZyB0aGUgYnVnLiAgSSdkIHN1Z2dlc3QgZml4aW5nIHRoZQ0KPj4+ID4gYnVn
IHN1Y2ggdGhhdCB0aGUgUEYgaXMgbGVmdCBpbiBhIGZ1bmN0aW9uYWwgc3RhdGUgaWYgdGhlIGtl
cm5lbCBpcw0KPj4+ID4gdW5hYmxlIHRvIGFsbG9jYXRlIHN1ZmZpY2llbnQgcmVzb3VyY2VzIGZv
ciB0aGUgVkZzLiAgVGhhbmtzLA0KPj4+ID4NCj4+PiA+IEFsZXgNCj4+PiA+DQo+Pj4gPiA+IC0t
LS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+Pj4gPiA+IEZyb206IEFsZXggV2lsbGlhbXNvbiBb
bWFpbHRvOmFsZXgud2lsbGlhbXNvbkByZWRoYXQuY29tXQ0KPj4+ID4gPiBTZW50OiBGcmlkYXks
IE1heSAxMiwgMjAxNyAxMToyMSBBTQ0KPj4+ID4gPiBUbzogQ2hlbmcsIENvbGxpbnMgPENvbGxp
bnMuQ2hlbmdAYW1kLmNvbT4NCj4+PiA+ID4gQ2M6IEJqb3JuIEhlbGdhYXMgPGJoZWxnYWFzQGdv
b2dsZS5jb20+Ow0KPj4+ID4gPiBsaW51eC1wY2lAdmdlci5rZXJuZWwub3JnOyBsaW51eC1rZXJu
ZWxAdmdlci5rZXJuZWwub3JnOyBEZXVjaGVyLA0KPj4+ID4gPiBBbGV4YW5kZXIgPEFsZXhhbmRl
ci5EZXVjaGVyQGFtZC5jb20+OyBaeXRhcnVrLCBLZWxseQ0KPj4+ID4gPiA8S2VsbHkuWnl0YXJ1
a0BhbWQuY29tPg0KPj4+ID4gPiBTdWJqZWN0OiBSZTogW1BBVENIXSBQQ0k6IE1ha2UgU1ItSU9W
IGNhcGFibGUgR1BVIHdvcmtpbmcgb24gdGhlDQo+Pj4gPiA+IFNSLUlPViBpbmNhcGFibGUgcGxh
dGZvcm0NCj4+PiA+ID4NCj4+PiA+ID4gT24gRnJpLCAxMiBNYXkgMjAxNyAwMjo1MDozMiArMDAw
MCAiQ2hlbmcsIENvbGxpbnMiDQo+Pj4gPiA+IDxDb2xsaW5zLkNoZW5nQGFtZC5jb20+IHdyb3Rl
Og0KPj4+ID4gPg0KPj4+ID4gPiA+IEhpIEhlbGdhYXMsDQo+Pj4gPiA+ID4NCj4+PiA+ID4gPiBT
b21lIEFNRCBHUFVzIGhhdmUgaGFyZHdhcmUgc3VwcG9ydCBmb3IgZ3JhcGhpY3MgU1ItSU9WLg0K
Pj4+ID4gPiA+IElmIHRoZSBTUi1JT1YgY2FwYWJsZSBHUFUgaXMgcGx1Z2dlZCBpbnRvIHRoZSBT
Ui1JT1YgaW5jYXBhYmxlDQo+Pj4gPiA+ID4gcGxhdGZvcm0uIEl0IHdvdWxkIGNhdXNlIGEgcHJv
YmxlbSBvbiBQQ0kgcmVzb3VyY2UgYWxsb2NhdGlvbg0KPj4+ID4gPiA+IGluIGN1cnJlbnQgTGlu
dXgga2VybmVsLg0KPj4+ID4gPiA+DQo+Pj4gPiA+ID4gVGhlcmVmb3JlIGluIG9yZGVyIHRvIGFs
bG93IHRoZSBQRiAoUGh5c2ljYWwgRnVuY3Rpb24pIGRldmljZQ0KPj4+ID4gPiA+IG9mIFNSLUlP
ViBjYXBhYmxlIEdQVSB0byB3b3JrIG9uIHRoZSBTUi1JT1YgaW5jYXBhYmxlIHBsYXRmb3JtLA0K
Pj4+ID4gPiA+IGl0IGlzIHJlcXVpcmVkIHRvIHZlcmlmeSBjb25kaXRpb25zIGZvciBpbml0aWFs
aXppbmcgQkFSDQo+Pj4gPiA+ID4gcmVzb3VyY2VzIG9uIEFNRCBTUi1JT1YgY2FwYWJsZSBHUFVz
Lg0KPj4+ID4gPiA+DQo+Pj4gPiA+ID4gSWYgdGhlIGRldmljZSBpcyBhbiBBTUQgZ3JhcGhpY3Mg
ZGV2aWNlIGFuZCBpdCBzdXBwb3J0cyBTUi1JT1YNCj4+PiA+ID4gPiBpdCB3aWxsIHJlcXVpcmUg
YSBsYXJnZSBhbW91bnQgb2YgcmVzb3VyY2VzLg0KPj4+ID4gPiA+IEJlZm9yZSBjYWxsaW5nIHNy
aW92X2luaXQoKSBtdXN0IGVuc3VyZSB0aGF0IHRoZSBzeXN0ZW0gQklPUw0KPj4+ID4gPiA+IGFs
c28gc3VwcG9ydHMgU1ItSU9WIGFuZCB0aGF0IHN5c3RlbSBCSU9TIGhhcyBiZWVuIGFibGUgdG8N
Cj4+PiA+ID4gPiBhbGxvY2F0ZSBlbm91Z2ggcmVzb3VyY2VzLg0KPj4+ID4gPiA+IElmIHRoZSBW
RiBCQVJzIGFyZSB6ZXJvIHRoZW4gdGhlIHN5c3RlbSBCSU9TIGRvZXMgbm90IHN1cHBvcnQNCj4+
PiA+ID4gPiBTUi1JT1Ygb3IgaXQgY291bGQgbm90IGFsbG9jYXRlIHRoZSByZXNvdXJjZXMgYW5k
IHRoaXMgcGxhdGZvcm0NCj4+PiA+ID4gPiB3aWxsIG5vdCBzdXBwb3J0IEFNRCBncmFwaGljcyBT
Ui1JT1YuDQo+Pj4gPiA+ID4gVGhlcmVmb3JlIGRvIG5vdCBjYWxsIHNyaW92X2luaXQoKS4NCj4+
PiA+ID4gPiBJZiB0aGUgc3lzdGVtIEJJT1MgZG9lcyBzdXBwb3J0IFNSLUlPViB0aGVuIHRoZSBW
RiBCQVJzIHdpbGwgYmUNCj4+PiA+ID4gPiBwcm9wZXJseSBpbml0aWFsaXplZCB0byBub24temVy
byB2YWx1ZXMuDQo+Pj4gPiA+ID4NCj4+PiA+ID4gPiBCZWxvdyBpcyB0aGUgcGF0Y2ggYWdhaW5z
dCB0byBLZXJuZWwgNC44ICYgNC45LiBQbGVhc2UgcmV2aWV3Lg0KPj4+ID4gPiA+DQo+Pj4gPiA+
ID4gSSBjaGVja2VkIHRoZSBkcml2ZXJzL3BjaS9xdWlya3MuYywgaXQgbG9va3MgdGhlDQo+Pj4g
PiA+ID4gd29ya2Fyb3VuZHMvZml4ZXMgaW4gcXVpcmtzLmMgYXJlIGZvciBzcGVjaWZpYyBkZXZp
Y2VzIGFuZCBvbmUNCj4+PiA+ID4gPiBvciBtb3JlIGRldmljZSBJRCBhcmUgZGVmaW5lZCBmb3Ig
dGhlIHNwZWNpZmljIGRldmljZXMuIEhvd2V2ZXINCj4+PiA+ID4gPiBteSBwYXRjaCBpcyBmb3Ig
YWxsIEFNRCBTUi1JT1YgY2FwYWJsZSBHUFVzLCB0aGF0IGluY2x1ZGVzIGFsbCBleGlzdGluZw0K
PmFuZCBmdXR1cmUgQU1EIHNlcnZlciBHUFVzLg0KPj4+ID4gPiA+IFNvIGl0IGRvZXNuJ3Qgc2Vl
bSBsaWtlIGEgZ29vZCBmaXQgdG8gcHV0IHRoZSBmaXggaW4gcXVpcmtzLmMuDQo+Pj4gPiA+DQo+
Pj4gPiA+DQo+Pj4gPiA+IFdoeSBpcyBhbiBBTUQgZ3JhcGhpY3MgY2FyZCB1bmlxdWUgaGVyZT8g
IERvZXNuJ3Qgc3Jpb3ZfaW5pdCgpDQo+Pj4gPiA+IGFsd2F5cyBuZWVkIHRvIGJlIGFibGUgdG8g
ZGVhbCB3aXRoIGRldmljZXMgb2YgYW55IHR5cGUgd2hlcmUgdGhlDQo+Pj4gPiA+IEJJT1MgaGFz
bid0IGluaXRpYWxpemVkIHRoZSBTUi1JT1YgY2FwYWJpbGl0eT8gIFNvbWUgU1ItSU9WDQo+Pj4g
PiA+IGRldmljZXMgY2FuIGZpdCB0aGVpciBWRnMgd2l0aGluIGEgbWluaW11bSBicmlkZ2UgYXBl
cnR1cmUsIG1vc3QNCj4+PiA+ID4gY2Fubm90LiAgSSBkb24ndCB1bmRlcnN0YW5kIHdoeSB0aGUg
VkYgcmVzb3VyY2UgcmVxdWlyZW1lbnRzDQo+Pj4gPiA+IGJlaW5nIGV4Y2VwdGlvbmFsbHkgbGFy
Z2UgZGljdGF0ZXMgdGhhdCB0aGV5IHJlY2VpdmUgc3BlY2lhbCBoYW5kbGluZy4NCj4+PiA+ID4g
VGhhbmtzLA0KPj4+ID4gPg0KPj4+ID4gPiBBbGV4DQo+Pj4gPiA+DQo+Pj4gPiA+ID4gU2lnbmVk
LW9mZi1ieTogQ29sbGlucyBDaGVuZyA8Y29sbGlucy5jaGVuZ0BhbWQuY29tPg0KPj4+ID4gPiA+
IC0tLQ0KPj4+ID4gPiA+ICBkcml2ZXJzL3BjaS9pb3YuYyB8IDYzDQo+Pj4gPiA+ID4gKysrKysr
KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKy0tLQ0KPj4+ID4g
PiA+ICAxIGZpbGUgY2hhbmdlZCwgNjAgaW5zZXJ0aW9ucygrKSwgMyBkZWxldGlvbnMoLSkNCj4+
PiA+ID4gPg0KPj4+ID4gPiA+IGRpZmYgLS1naXQgYS9kcml2ZXJzL3BjaS9pb3YuYyBiL2RyaXZl
cnMvcGNpL2lvdi5jIGluZGV4DQo+Pj4gPiA+ID4gZTMwZjA1Yy4uZTRmMTQwNSAxMDA2NDQNCj4+
PiA+ID4gPiAtLS0gYS9kcml2ZXJzL3BjaS9pb3YuYw0KPj4+ID4gPiA+ICsrKyBiL2RyaXZlcnMv
cGNpL2lvdi5jDQo+Pj4gPiA+ID4gQEAgLTUyMyw2ICs1MjMsNDUgQEAgc3RhdGljIHZvaWQgc3Jp
b3ZfcmVzdG9yZV9zdGF0ZShzdHJ1Y3QgcGNpX2Rldg0KPipkZXYpDQo+Pj4gPiA+ID4gICAgICAg
ICAgICAgICAgIG1zbGVlcCgxMDApOw0KPj4+ID4gPiA+ICB9DQo+Pj4gPiA+ID4NCj4+PiA+ID4g
PiArLyoNCj4+PiA+ID4gPiArICogcGNpX3ZmX2Jhcl92YWxpZCAtIGNoZWNrIGlmIFZGIEJBUnMg
aGF2ZSByZXNvdXJjZSBhbGxvY2F0ZWQNCj4+PiA+ID4gPiArICogQGRldjogdGhlIFBDSSBkZXZp
Y2UNCj4+PiA+ID4gPiArICogQHBvczogcmVnaXN0ZXIgb2Zmc2V0IG9mIFNSLUlPViBjYXBhYmls
aXR5IGluIFBDSSBjb25maWcNCj4+PiA+ID4gPiArc3BhY2UNCj4+PiA+ID4gPiArICogUmV0dXJu
cyB0cnVlIGFueSBWRiBCQVIgaGFzIHJlc291cmNlIGFsbG9jYXRlZCwgZmFsc2UNCj4+PiA+ID4g
PiArICogaWYgYWxsIFZGIEJBUnMgYXJlIGVtcHR5Lg0KPj4+ID4gPiA+ICsgKi8NCj4+PiA+ID4g
PiArc3RhdGljIGJvb2wgcGNpX3ZmX2Jhcl92YWxpZChzdHJ1Y3QgcGNpX2RldiAqZGV2LCBpbnQg
cG9zKSB7DQo+Pj4gPiA+ID4gKyAgICAgICBpbnQgaTsNCj4+PiA+ID4gPiArICAgICAgIHUzMiBi
YXJfdmFsdWU7DQo+Pj4gPiA+ID4gKyAgICAgICB1MzIgYmFyX3NpemVfbWFzayA9IH4oUENJX0JB
U0VfQUREUkVTU19TUEFDRSB8DQo+Pj4gPiA+ID4gKyAgICAgICAgICAgICAgICAgICAgICAgUENJ
X0JBU0VfQUREUkVTU19NRU1fVFlQRV82NCB8DQo+Pj4gPiA+ID4gKyAgICAgICAgICAgICAgICAg
ICAgICAgUENJX0JBU0VfQUREUkVTU19NRU1fUFJFRkVUQ0gpOw0KPj4+ID4gPiA+ICsNCj4+PiA+
ID4gPiArICAgICAgIGZvciAoaSA9IDA7IGkgPCBQQ0lfU1JJT1ZfTlVNX0JBUlM7IGkrKykgew0K
Pj4+ID4gPiA+ICsgICAgICAgICAgICAgICBwY2lfcmVhZF9jb25maWdfZHdvcmQoZGV2LCBwb3Mg
KyBQQ0lfU1JJT1ZfQkFSICsgaSAqIDQsDQo+JmJhcl92YWx1ZSk7DQo+Pj4gPiA+ID4gKyAgICAg
ICAgICAgICAgIGlmIChiYXJfdmFsdWUgJiBiYXJfc2l6ZV9tYXNrKQ0KPj4+ID4gPiA+ICsgICAg
ICAgICAgICAgICAgICAgICAgIHJldHVybiB0cnVlOw0KPj4+ID4gPiA+ICsgICAgICAgfQ0KPj4+
ID4gPiA+ICsNCj4+PiA+ID4gPiArICAgICAgIHJldHVybiBmYWxzZTsNCj4+PiA+ID4gPiArfQ0K
Pj4+ID4gPiA+ICsNCj4+PiA+ID4gPiArLyoNCj4+PiA+ID4gPiArICogaXNfYW1kX2Rpc3BsYXlf
YWRhcHRlciAtIGNoZWNrIGlmIGl0IGlzIGFuIEFNRC9BVEkgR1BVDQo+Pj4gPiA+ID4gK2Rldmlj
ZQ0KPj4+ID4gPiA+ICsgKiBAZGV2OiB0aGUgUENJIGRldmljZQ0KPj4+ID4gPiA+ICsgKg0KPj4+
ID4gPiA+ICsgKiBSZXR1cm5zIHRydWUgaWYgZGV2aWNlIGlzIGFuIEFNRC9BVEkgZGlzcGxheSBh
ZGFwdGVyLA0KPj4+ID4gPiA+ICsgKiBvdGhlcndpc2UgcmV0dXJuIGZhbHNlLg0KPj4+ID4gPiA+
ICsgKi8NCj4+PiA+ID4gPiArDQo+Pj4gPiA+ID4gK3N0YXRpYyBib29sIGlzX2FtZF9kaXNwbGF5
X2FkYXB0ZXIoc3RydWN0IHBjaV9kZXYgKmRldikgew0KPj4+ID4gPiA+ICsgICAgICAgcmV0dXJu
ICgoKGRldi0+Y2xhc3MgPj4gMTYpID09IFBDSV9CQVNFX0NMQVNTX0RJU1BMQVkpICYmDQo+Pj4g
PiA+ID4gKyAgICAgICAgICAgICAgIChkZXYtPnZlbmRvciA9PSBQQ0lfVkVORE9SX0lEX0FUSSB8
fA0KPj4+ID4gPiA+ICsgICAgICAgICAgICAgICBkZXYtPnZlbmRvciA9PSBQQ0lfVkVORE9SX0lE
X0FNRCkpOyB9DQo+Pj4gPiA+ID4gKw0KPj4+ID4gPiA+ICAvKioNCj4+PiA+ID4gPiAgICogcGNp
X2lvdl9pbml0IC0gaW5pdGlhbGl6ZSB0aGUgSU9WIGNhcGFiaWxpdHkNCj4+PiA+ID4gPiAgICog
QGRldjogdGhlIFBDSSBkZXZpY2UNCj4+PiA+ID4gPiBAQCAtNTM3LDkgKzU3NiwyNyBAQCBpbnQg
cGNpX2lvdl9pbml0KHN0cnVjdCBwY2lfZGV2ICpkZXYpDQo+Pj4gPiA+ID4gICAgICAgICAgICAg
ICAgIHJldHVybiAtRU5PREVWOw0KPj4+ID4gPiA+DQo+Pj4gPiA+ID4gICAgICAgICBwb3MgPSBw
Y2lfZmluZF9leHRfY2FwYWJpbGl0eShkZXYsIFBDSV9FWFRfQ0FQX0lEX1NSSU9WKTsNCj4+PiA+
ID4gPiAtICAgICAgIGlmIChwb3MpDQo+Pj4gPiA+ID4gLSAgICAgICAgICAgICAgIHJldHVybiBz
cmlvdl9pbml0KGRldiwgcG9zKTsNCj4+PiA+ID4gPiAtDQo+Pj4gPiA+ID4gKyAgICAgICBpZiAo
cG9zKSB7DQo+Pj4gPiA+ID4gKyAgICAgICAvKg0KPj4+ID4gPiA+ICsgICAgICAgICogSWYgdGhl
IGRldmljZSBpcyBhbiBBTUQgZ3JhcGhpY3MgZGV2aWNlIGFuZCBpdCBzdXBwb3J0cw0KPj4+ID4g
PiA+ICsgICAgICAgICogU1ItSU9WIGl0IHdpbGwgcmVxdWlyZSBhIGxhcmdlIGFtb3VudCBvZiBy
ZXNvdXJjZXMuDQo+Pj4gPiA+ID4gKyAgICAgICAgKiBCZWZvcmUgY2FsbGluZyBzcmlvdl9pbml0
KCkgbXVzdCBlbnN1cmUgdGhhdCB0aGUgc3lzdGVtDQo+Pj4gPiA+ID4gKyAgICAgICAgKiBCSU9T
IGFsc28gc3VwcG9ydHMgU1ItSU9WIGFuZCB0aGF0IHN5c3RlbSBCSU9TIGhhcyBiZWVuDQo+Pj4g
PiA+ID4gKyAgICAgICAgKiBhYmxlIHRvIGFsbG9jYXRlIGVub3VnaCByZXNvdXJjZXMuDQo+Pj4g
PiA+ID4gKyAgICAgICAgKiBJZiB0aGUgVkYgQkFScyBhcmUgemVybyB0aGVuIHRoZSBzeXN0ZW0g
QklPUyBkb2VzIG5vdA0KPj4+ID4gPiA+ICsgICAgICAgICogc3VwcG9ydCBTUi1JT1Ygb3IgaXQg
Y291bGQgbm90IGFsbG9jYXRlIHRoZSByZXNvdXJjZXMNCj4+PiA+ID4gPiArICAgICAgICAqIGFu
ZCB0aGlzIHBsYXRmb3JtIHdpbGwgbm90IHN1cHBvcnQgQU1EIGdyYXBoaWNzIFNSLUlPVi4NCj4+
PiA+ID4gPiArICAgICAgICAqIFRoZXJlZm9yZSBkbyBub3QgY2FsbCBzcmlvdl9pbml0KCkuDQo+
Pj4gPiA+ID4gKyAgICAgICAgKiBJZiB0aGUgc3lzdGVtIEJJT1MgZG9lcyBzdXBwb3J0IFNSLUlP
ViB0aGVuIHRoZSBWRiBCQVJzDQo+Pj4gPiA+ID4gKyAgICAgICAgKiB3aWxsIGJlIHByb3Blcmx5
IGluaXRpYWxpemVkIHRvIG5vbi16ZXJvIHZhbHVlcy4NCj4+PiA+ID4gPiArICAgICAgICAqLw0K
Pj4+ID4gPiA+ICsgICAgICAgICAgICAgICBpZiAoaXNfYW1kX2Rpc3BsYXlfYWRhcHRlcihkZXYp
KSB7DQo+Pj4gPiA+ID4gKyAgICAgICAgICAgICAgICAgICAgICAgaWYgKHBjaV92Zl9iYXJfdmFs
aWQoZGV2LCBwb3MpKQ0KPj4+ID4gPiA+ICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
cmV0dXJuIHNyaW92X2luaXQoZGV2LCBwb3MpOw0KPj4+ID4gPiA+ICsgICAgICAgICAgICAgICB9
IGVsc2Ugew0KPj4+ID4gPiA+ICsgICAgICAgICAgICAgICAgICAgICAgIHJldHVybiBzcmlvdl9p
bml0KGRldiwgcG9zKTsNCj4+PiA+ID4gPiArICAgICAgICAgICAgICAgfQ0KPj4+ID4gPiA+ICsg
ICAgICAgfQ0KPj4+ID4gPiA+ICAgICAgICAgcmV0dXJuIC1FTk9ERVY7DQo+Pj4gPiA+ID4gIH0N
Cj4+PiA+ID4gPg0KPj4+ID4gPg0KPj4+ID4NCj4+Pg0KPj4NCg==

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Kelly,

This issue also happens in "not SR-IOV capable" SBIOS. It seems some "not SR-IOV capable" SBIOS will directly report error in system BIOS boot stage and doesn't boot to OS. But other "not SR-IOV capable" SBIOS would not report error and boot to Linux.

-Collins Cheng


-----Original Message-----
From: Zytaruk, Kelly 
Sent: Saturday, May 20, 2017 6:28 PM
To: Cheng, Collins <Collins.Cheng@amd.com>; Alexander Duyck <alexander.duyck@gmail.com>; Alex Williamson <alex.williamson@redhat.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform



>-----Original Message-----
>From: Cheng, Collins
>Sent: Saturday, May 20, 2017 12:53 AM
>To: Alexander Duyck; Alex Williamson
>Cc: Bjorn Helgaas; linux-pci@vger.kernel.org; 
>linux-kernel@vger.kernel.org; Deucher, Alexander; Zytaruk, Kelly; 
>Yinghai Lu
>Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV 
>incapable platform
>
>Hi Alex,
>
>Yes, I hope kernel can disable SR-IOV and related VF resource 
>allocation if the system BIOS is not SR-IOV capable.
>
>Adding the parameter "pci=nosriov" sounds a doable solution, but it 
>would need user to add this parameter manually, right? I think an 
>automatic detection would be better. My patch is trying to auto detect and bypass VF resource allocation.
>
>
>-Collins Cheng
>

Collins, be careful about this.  I don't think that this is what we want.  If you add "pci=nosriov" then you are globally disabling SRIOV for all devices.  This is not the solution that we are looking for.
Remember that there are 3 types of SBIOS; "not SR-IOV capable", "SR-IOV capable but does not support large resources", "Complete SR-IOV support".

The problem is that we are trying to find a fix for "broken" SBIOS that does support SR-IOV but does not support the full SR-IOV capabilities that devices with large resources require.

Thanks,
Kelly

>
>-----Original Message-----
>From: Alexander Duyck [mailto:alexander.duyck@gmail.com]
>Sent: Friday, May 19, 2017 11:44 PM
>To: Alex Williamson <alex.williamson@redhat.com>
>Cc: Cheng, Collins <Collins.Cheng@amd.com>; Bjorn Helgaas 
><bhelgaas@google.com>; linux-pci@vger.kernel.org; linux- 
>kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; 
>Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
>Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV 
>incapable platform
>
>On Mon, May 15, 2017 at 10:53 AM, Alex Williamson 
><alex.williamson@redhat.com> wrote:
>> On Mon, 15 May 2017 08:19:28 +0000
>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>
>>> Hi Williamson,
>>>
>>> We cannot assume BIOS supports SR-IOV, actually only newer server
>motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or 
>older server motherboard BIOS doesn't support SR-IOV. This issue would 
>happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.
>>
>> Servers should be supporting SR-IOV for a long time now.  What really 
>> is there to a BIOS supporting SR-IOV anyway, it's simply reserving 
>> sufficient bus number and MMIO resources such that we can enable the 
>> VFs.  This process isn't exclusively reserved for the BIOS.  Some 
>> platforms may choose to only initialize boot devices, leaving the 
>> rest for the OS to program.  The initial proposal here to disable 
>> SR-IOV if not programmed at OS hand-off disables even the possibility 
>> of the OS reallocating resources for this device.
>
>There are differences between supporting SR-IOV and supporting SR-IOV 
>on devices with massive resources. I know I have seen NICs that will 
>keep a system from completing POST if SR-IOV is enabled, and MMIO 
>beyond 4G is not. My guess would be that the issues being seen are 
>probably that they disable SR-IOV in the BIOS in such a setup and end 
>up running into issues when they try to boot into the Linux kernel as 
>it goes through and tries to allocate resources for SR-IOV even though it was disabled in the BIOS.
>
>It might make sense to add a kernel parameter something like a "pci=nosriov"
>that would allow for disabling SR-IOV and related resource allocation 
>if that is what we are talking about. That way you could plug in these 
>types of devices into a system with a legacy bios or that doesn't wan 
>to allocate addresses above 32b for MMIO, and this parameter would be 
>all that is needed to disable SR-IOV so you could plug in a NIC that has SR-IOV associated with it.
>
>>> I agree that failure to allocate VF resources should leave the 
>>> device in no
>worse condition than before it tried. I hope kernel could allocate PF 
>device resource before allocating VF device resource, and keep PF 
>device resource valid and functional if failed to allocate VF device resource.
>>>
>>> I will send out dmesg log lspci info tomorrow. Thanks.
>>
>> Thanks,
>> Alex
>>
>>> -----Original Message-----
>>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> Sent: Friday, May 12, 2017 10:43 PM
>>> To: Cheng, Collins <Collins.Cheng@amd.com>
>>> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
>>> linux-kernel@vger.kernel.org; Deucher, Alexander 
>>> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; 
>>> Yinghai Lu <yinghai@kernel.org>
>>> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
>>> SR-IOV incapable platform
>>>
>>> On Fri, 12 May 2017 04:51:43 +0000
>>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>>
>>> > Hi Williamson,
>>> >
>>> > I verified the patch is working for both AMD SR-IOV GPU and Intel 
>>> > SR-IOV
>NIC. I don't think it is redundant to check the VF BAR valid before 
>call sriov_init(), it is safe and saving boot time, also there is no a 
>better method to know if system BIOS has correctly initialized the SR-IOV capability or not.
>>>
>>> It also masks an underlying bug and creates a maintenance issue that 
>>> we won't
>know when it's safe to remove this workaround.  I don't think faster 
>boot is valid rationale, in one case SR-IOV is completely disabled, the 
>other we attempt to allocate the resources the BIOS failed to provide.  
>I expect this is also a corner case, the BIOS should typically support 
>SR-IOV, therefore this situation should be an exception.
>>>
>>> > I did not try to fix the issue from the kernel resource allocation 
>>> > perspective,
>it is because:
>>> >     1. I am not very familiar with the PCI resource allocation scheme in kernel.
>For example, in sriov_init(), kernel will re-assign the PCI resource 
>for both VF and PF. I don't understand why kernel allocates resource 
>for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
>>> >     2. I am not sure if kernel has error handler if PCI resource allocation failed.
>In this case, kernel cannot allocate enough resource to PF. It should 
>trigger some error handler to either just keep original BAR values set 
>by system BIOS, or disable this device and log errors.
>>>
>>> I think these are the issues we should be trying to solve and I'm 
>>> sure folks on the linux-pci list can help us identify the bug.
>>> Minimally, failure to allocate VF resources should leave the device 
>>> in no worse condition than before it tried.  Perhaps you could post 
>>> more details about the issue, boot with pci=earlydump, post dmesg of 
>>> a boot where the PF resources are incorrectly re-allocated, and 
>>> include lspci -vvv for the SR-IOV device.  Also, please test with 
>>> the latest upstream kernel, upstream only patches old kernels 
>>> through stable backports of commits to the latest kernel.  Adding 
>>> Yinghai as a resource allocation expert. Thanks,
>>>
>>> Alex
>>>
>>> > -----Original Message-----
>>> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> > Sent: Friday, May 12, 2017 12:01 PM
>>> > To: Cheng, Collins <Collins.Cheng@amd.com>
>>> > Cc: Bjorn Helgaas <bhelgaas@google.com>; 
>>> > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, 
>>> > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly 
>>> > <Kelly.Zytaruk@amd.com>
>>> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
>>> > SR-IOV incapable platform
>>> >
>>> > On Fri, 12 May 2017 03:42:46 +0000 "Cheng, Collins" 
>>> > <Collins.Cheng@amd.com> wrote:
>>> >
>>> > > Hi Williamson,
>>> > >
>>> > > GPU card needs more BAR aperture resource than other PCI 
>>> > > devices. For
>example, Intel SR-IOV network card only require 512KB memory resource 
>for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory 
>resource for frame buffer BAR aperture.
>>> > >
>>> > > If the system BIOS supports SR-IOV, it will reserve enough 
>>> > > resource for all
>VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate 
>the enough resource for VF BARs, only PF BAR will be assigned and VF 
>BARs are empty. Then system boots to Linux kernel and kernel doesn't 
>check if the VF BARs are empty or valid. Kernel will re-assign the BAR 
>resources for PF and all VFs. The problem I saw is that kernel will 
>fail to allocate PF BAR resource because some resources are assigned to 
>VF, this is not expected. So kernel might need to do some check before 
>re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.
>>> >
>>> > So the problem is that something bad happens when the kernel is 
>>> > trying to reallocate resources in order to fulfill the 
>>> > requirements of the VFs, leaving the PF resources incorrectly 
>>> > programmed?  Why not just fix that bug rather than creating 
>>> > special handling for this vendor/class of device which disables 
>>> > any attempt to fixup resources for SR-IOV?  IOW, this patch just 
>>> > avoids the problem for your devices rather than fixing the bug.  
>>> > I'd suggest fixing the bug such that the PF is left in a 
>>> > functional state if the kernel is unable to allocate sufficient 
>>> > resources for the VFs.  Thanks,
>>> >
>>> > Alex
>>> >
>>> > > -----Original Message-----
>>> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>> > > Sent: Friday, May 12, 2017 11:21 AM
>>> > > To: Cheng, Collins <Collins.Cheng@amd.com>
>>> > > Cc: Bjorn Helgaas <bhelgaas@google.com>; 
>>> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; 
>>> > > Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly 
>>> > > <Kelly.Zytaruk@amd.com>
>>> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
>>> > > SR-IOV incapable platform
>>> > >
>>> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins"
>>> > > <Collins.Cheng@amd.com> wrote:
>>> > >
>>> > > > Hi Helgaas,
>>> > > >
>>> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
>>> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable 
>>> > > > platform. It would cause a problem on PCI resource allocation 
>>> > > > in current Linux kernel.
>>> > > >
>>> > > > Therefore in order to allow the PF (Physical Function) device 
>>> > > > of SR-IOV capable GPU to work on the SR-IOV incapable 
>>> > > > platform, it is required to verify conditions for initializing 
>>> > > > BAR resources on AMD SR-IOV capable GPUs.
>>> > > >
>>> > > > If the device is an AMD graphics device and it supports SR-IOV 
>>> > > > it will require a large amount of resources.
>>> > > > Before calling sriov_init() must ensure that the system BIOS 
>>> > > > also supports SR-IOV and that system BIOS has been able to 
>>> > > > allocate enough resources.
>>> > > > If the VF BARs are zero then the system BIOS does not support 
>>> > > > SR-IOV or it could not allocate the resources and this 
>>> > > > platform will not support AMD graphics SR-IOV.
>>> > > > Therefore do not call sriov_init().
>>> > > > If the system BIOS does support SR-IOV then the VF BARs will 
>>> > > > be properly initialized to non-zero values.
>>> > > >
>>> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
>>> > > >
>>> > > > I checked the drivers/pci/quirks.c, it looks the 
>>> > > > workarounds/fixes in quirks.c are for specific devices and one 
>>> > > > or more device ID are defined for the specific devices. 
>>> > > > However my patch is for all AMD SR-IOV capable GPUs, that 
>>> > > > includes all existing
>and future AMD server GPUs.
>>> > > > So it doesn't seem like a good fit to put the fix in quirks.c.
>>> > >
>>> > >
>>> > > Why is an AMD graphics card unique here?  Doesn't sriov_init() 
>>> > > always need to be able to deal with devices of any type where 
>>> > > the BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV 
>>> > > devices can fit their VFs within a minimum bridge aperture, most 
>>> > > cannot.  I don't understand why the VF resource requirements 
>>> > > being exceptionally large dictates that they receive special handling.
>>> > > Thanks,
>>> > >
>>> > > Alex
>>> > >
>>> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>>> > > > ---
>>> > > >  drivers/pci/iov.c | 63
>>> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>>> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
>>> > > >
>>> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>>> > > > e30f05c..e4f1405 100644
>>> > > > --- a/drivers/pci/iov.c
>>> > > > +++ b/drivers/pci/iov.c
>>> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct 
>>> > > > pci_dev
>*dev)
>>> > > >                 msleep(100);
>>> > > >  }
>>> > > >
>>> > > > +/*
>>> > > > + * pci_vf_bar_valid - check if VF BARs have resource 
>>> > > > +allocated
>>> > > > + * @dev: the PCI device
>>> > > > + * @pos: register offset of SR-IOV capability in PCI config 
>>> > > > +space
>>> > > > + * Returns true any VF BAR has resource allocated, false
>>> > > > + * if all VF BARs are empty.
>>> > > > + */
>>> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>>> > > > +       int i;
>>> > > > +       u32 bar_value;
>>> > > > +       u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
>>> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
>>> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
>>> > > > +
>>> > > > +       for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>>> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR 
>>> > > > + + i * 4,
>&bar_value);
>>> > > > +               if (bar_value & bar_size_mask)
>>> > > > +                       return true;
>>> > > > +       }
>>> > > > +
>>> > > > +       return false;
>>> > > > +}
>>> > > > +
>>> > > > +/*
>>> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU 
>>> > > > +device
>>> > > > + * @dev: the PCI device
>>> > > > + *
>>> > > > + * Returns true if device is an AMD/ATI display adapter,
>>> > > > + * otherwise return false.
>>> > > > + */
>>> > > > +
>>> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
>>> > > > +       return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
>>> > > > +               (dev->vendor == PCI_VENDOR_ID_ATI ||
>>> > > > +               dev->vendor == PCI_VENDOR_ID_AMD)); }
>>> > > > +
>>> > > >  /**
>>> > > >   * pci_iov_init - initialize the IOV capability
>>> > > >   * @dev: the PCI device
>>> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>>> > > >                 return -ENODEV;
>>> > > >
>>> > > >         pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
>>> > > > -       if (pos)
>>> > > > -               return sriov_init(dev, pos);
>>> > > > -
>>> > > > +       if (pos) {
>>> > > > +       /*
>>> > > > +        * If the device is an AMD graphics device and it supports
>>> > > > +        * SR-IOV it will require a large amount of resources.
>>> > > > +        * Before calling sriov_init() must ensure that the system
>>> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
>>> > > > +        * able to allocate enough resources.
>>> > > > +        * If the VF BARs are zero then the system BIOS does not
>>> > > > +        * support SR-IOV or it could not allocate the resources
>>> > > > +        * and this platform will not support AMD graphics SR-IOV.
>>> > > > +        * Therefore do not call sriov_init().
>>> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
>>> > > > +        * will be properly initialized to non-zero values.
>>> > > > +        */
>>> > > > +               if (is_amd_display_adapter(dev)) {
>>> > > > +                       if (pci_vf_bar_valid(dev, pos))
>>> > > > +                               return sriov_init(dev, pos);
>>> > > > +               } else {
>>> > > > +                       return sriov_init(dev, pos);
>>> > > > +               }
>>> > > > +       }
>>> > > >         return -ENODEV;
>>> > > >  }
>>> > > >
>>> > >
>>> >
>>>
>>

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

SGkgS2VsbHksDQoNClRoaXMgaXNzdWUgYWxzbyBoYXBwZW5zIGluICJub3QgU1ItSU9WIGNhcGFi
bGUiIFNCSU9TLiBJdCBzZWVtcyBzb21lICJub3QgU1ItSU9WIGNhcGFibGUiIFNCSU9TIHdpbGwg
ZGlyZWN0bHkgcmVwb3J0IGVycm9yIGluIHN5c3RlbSBCSU9TIGJvb3Qgc3RhZ2UgYW5kIGRvZXNu
J3QgYm9vdCB0byBPUy4gQnV0IG90aGVyICJub3QgU1ItSU9WIGNhcGFibGUiIFNCSU9TIHdvdWxk
IG5vdCByZXBvcnQgZXJyb3IgYW5kIGJvb3QgdG8gTGludXguDQoNCi1Db2xsaW5zIENoZW5nDQoN
Cg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IFp5dGFydWssIEtlbGx5IA0KU2Vu
dDogU2F0dXJkYXksIE1heSAyMCwgMjAxNyA2OjI4IFBNDQpUbzogQ2hlbmcsIENvbGxpbnMgPENv
bGxpbnMuQ2hlbmdAYW1kLmNvbT47IEFsZXhhbmRlciBEdXljayA8YWxleGFuZGVyLmR1eWNrQGdt
YWlsLmNvbT47IEFsZXggV2lsbGlhbXNvbiA8YWxleC53aWxsaWFtc29uQHJlZGhhdC5jb20+DQpD
YzogQmpvcm4gSGVsZ2FhcyA8YmhlbGdhYXNAZ29vZ2xlLmNvbT47IGxpbnV4LXBjaUB2Z2VyLmtl
cm5lbC5vcmc7IGxpbnV4LWtlcm5lbEB2Z2VyLmtlcm5lbC5vcmc7IERldWNoZXIsIEFsZXhhbmRl
ciA8QWxleGFuZGVyLkRldWNoZXJAYW1kLmNvbT47IFlpbmdoYWkgTHUgPHlpbmdoYWlAa2VybmVs
Lm9yZz4NClN1YmplY3Q6IFJFOiBbUEFUQ0hdIFBDSTogTWFrZSBTUi1JT1YgY2FwYWJsZSBHUFUg
d29ya2luZyBvbiB0aGUgU1ItSU9WIGluY2FwYWJsZSBwbGF0Zm9ybQ0KDQoNCg0KPi0tLS0tT3Jp
Z2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogQ2hlbmcsIENvbGxpbnMNCj5TZW50OiBTYXR1cmRh
eSwgTWF5IDIwLCAyMDE3IDEyOjUzIEFNDQo+VG86IEFsZXhhbmRlciBEdXljazsgQWxleCBXaWxs
aWFtc29uDQo+Q2M6IEJqb3JuIEhlbGdhYXM7IGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc7IA0K
PmxpbnV4LWtlcm5lbEB2Z2VyLmtlcm5lbC5vcmc7IERldWNoZXIsIEFsZXhhbmRlcjsgWnl0YXJ1
aywgS2VsbHk7IA0KPllpbmdoYWkgTHUNCj5TdWJqZWN0OiBSRTogW1BBVENIXSBQQ0k6IE1ha2Ug
U1ItSU9WIGNhcGFibGUgR1BVIHdvcmtpbmcgb24gdGhlIFNSLUlPViANCj5pbmNhcGFibGUgcGxh
dGZvcm0NCj4NCj5IaSBBbGV4LA0KPg0KPlllcywgSSBob3BlIGtlcm5lbCBjYW4gZGlzYWJsZSBT
Ui1JT1YgYW5kIHJlbGF0ZWQgVkYgcmVzb3VyY2UgDQo+YWxsb2NhdGlvbiBpZiB0aGUgc3lzdGVt
IEJJT1MgaXMgbm90IFNSLUlPViBjYXBhYmxlLg0KPg0KPkFkZGluZyB0aGUgcGFyYW1ldGVyICJw
Y2k9bm9zcmlvdiIgc291bmRzIGEgZG9hYmxlIHNvbHV0aW9uLCBidXQgaXQgDQo+d291bGQgbmVl
ZCB1c2VyIHRvIGFkZCB0aGlzIHBhcmFtZXRlciBtYW51YWxseSwgcmlnaHQ/IEkgdGhpbmsgYW4g
DQo+YXV0b21hdGljIGRldGVjdGlvbiB3b3VsZCBiZSBiZXR0ZXIuIE15IHBhdGNoIGlzIHRyeWlu
ZyB0byBhdXRvIGRldGVjdCBhbmQgYnlwYXNzIFZGIHJlc291cmNlIGFsbG9jYXRpb24uDQo+DQo+
DQo+LUNvbGxpbnMgQ2hlbmcNCj4NCg0KQ29sbGlucywgYmUgY2FyZWZ1bCBhYm91dCB0aGlzLiAg
SSBkb24ndCB0aGluayB0aGF0IHRoaXMgaXMgd2hhdCB3ZSB3YW50LiAgSWYgeW91IGFkZCAicGNp
PW5vc3Jpb3YiIHRoZW4geW91IGFyZSBnbG9iYWxseSBkaXNhYmxpbmcgU1JJT1YgZm9yIGFsbCBk
ZXZpY2VzLiAgVGhpcyBpcyBub3QgdGhlIHNvbHV0aW9uIHRoYXQgd2UgYXJlIGxvb2tpbmcgZm9y
Lg0KUmVtZW1iZXIgdGhhdCB0aGVyZSBhcmUgMyB0eXBlcyBvZiBTQklPUzsgIm5vdCBTUi1JT1Yg
Y2FwYWJsZSIsICJTUi1JT1YgY2FwYWJsZSBidXQgZG9lcyBub3Qgc3VwcG9ydCBsYXJnZSByZXNv
dXJjZXMiLCAiQ29tcGxldGUgU1ItSU9WIHN1cHBvcnQiLg0KDQpUaGUgcHJvYmxlbSBpcyB0aGF0
IHdlIGFyZSB0cnlpbmcgdG8gZmluZCBhIGZpeCBmb3IgImJyb2tlbiIgU0JJT1MgdGhhdCBkb2Vz
IHN1cHBvcnQgU1ItSU9WIGJ1dCBkb2VzIG5vdCBzdXBwb3J0IHRoZSBmdWxsIFNSLUlPViBjYXBh
YmlsaXRpZXMgdGhhdCBkZXZpY2VzIHdpdGggbGFyZ2UgcmVzb3VyY2VzIHJlcXVpcmUuDQoNClRo
YW5rcywNCktlbGx5DQoNCj4NCj4tLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPkZyb206IEFs
ZXhhbmRlciBEdXljayBbbWFpbHRvOmFsZXhhbmRlci5kdXlja0BnbWFpbC5jb21dDQo+U2VudDog
RnJpZGF5LCBNYXkgMTksIDIwMTcgMTE6NDQgUE0NCj5UbzogQWxleCBXaWxsaWFtc29uIDxhbGV4
LndpbGxpYW1zb25AcmVkaGF0LmNvbT4NCj5DYzogQ2hlbmcsIENvbGxpbnMgPENvbGxpbnMuQ2hl
bmdAYW1kLmNvbT47IEJqb3JuIEhlbGdhYXMgDQo+PGJoZWxnYWFzQGdvb2dsZS5jb20+OyBsaW51
eC1wY2lAdmdlci5rZXJuZWwub3JnOyBsaW51eC0gDQo+a2VybmVsQHZnZXIua2VybmVsLm9yZzsg
RGV1Y2hlciwgQWxleGFuZGVyIDxBbGV4YW5kZXIuRGV1Y2hlckBhbWQuY29tPjsgDQo+Wnl0YXJ1
aywgS2VsbHkgPEtlbGx5Llp5dGFydWtAYW1kLmNvbT47IFlpbmdoYWkgTHUgPHlpbmdoYWlAa2Vy
bmVsLm9yZz4NCj5TdWJqZWN0OiBSZTogW1BBVENIXSBQQ0k6IE1ha2UgU1ItSU9WIGNhcGFibGUg
R1BVIHdvcmtpbmcgb24gdGhlIFNSLUlPViANCj5pbmNhcGFibGUgcGxhdGZvcm0NCj4NCj5PbiBN
b24sIE1heSAxNSwgMjAxNyBhdCAxMDo1MyBBTSwgQWxleCBXaWxsaWFtc29uIA0KPjxhbGV4Lndp
bGxpYW1zb25AcmVkaGF0LmNvbT4gd3JvdGU6DQo+PiBPbiBNb24sIDE1IE1heSAyMDE3IDA4OjE5
OjI4ICswMDAwDQo+PiAiQ2hlbmcsIENvbGxpbnMiIDxDb2xsaW5zLkNoZW5nQGFtZC5jb20+IHdy
b3RlOg0KPj4NCj4+PiBIaSBXaWxsaWFtc29uLA0KPj4+DQo+Pj4gV2UgY2Fubm90IGFzc3VtZSBC
SU9TIHN1cHBvcnRzIFNSLUlPViwgYWN0dWFsbHkgb25seSBuZXdlciBzZXJ2ZXINCj5tb3RoZXJi
b2FyZCBCSU9TIHN1cHBvcnRzIFNSLUlPVi4gTm9ybWFsIGRlc2t0b3AgbW90aGVyYm9hcmQgQklP
UyBvciANCj5vbGRlciBzZXJ2ZXIgbW90aGVyYm9hcmQgQklPUyBkb2Vzbid0IHN1cHBvcnQgU1It
SU9WLiBUaGlzIGlzc3VlIHdvdWxkIA0KPmhhcHBlbiBpZiBhbiB1c2VyIHBsdWdzIG91ciBBTUQg
U1ItSU9WIGNhcGFibGUgR1BVIGNhcmQgdG8gYSBub3JtYWwgZGVza3RvcCBtb3RoZXJib2FyZC4N
Cj4+DQo+PiBTZXJ2ZXJzIHNob3VsZCBiZSBzdXBwb3J0aW5nIFNSLUlPViBmb3IgYSBsb25nIHRp
bWUgbm93LiAgV2hhdCByZWFsbHkgDQo+PiBpcyB0aGVyZSB0byBhIEJJT1Mgc3VwcG9ydGluZyBT
Ui1JT1YgYW55d2F5LCBpdCdzIHNpbXBseSByZXNlcnZpbmcgDQo+PiBzdWZmaWNpZW50IGJ1cyBu
dW1iZXIgYW5kIE1NSU8gcmVzb3VyY2VzIHN1Y2ggdGhhdCB3ZSBjYW4gZW5hYmxlIHRoZSANCj4+
IFZGcy4gIFRoaXMgcHJvY2VzcyBpc24ndCBleGNsdXNpdmVseSByZXNlcnZlZCBmb3IgdGhlIEJJ
T1MuICBTb21lIA0KPj4gcGxhdGZvcm1zIG1heSBjaG9vc2UgdG8gb25seSBpbml0aWFsaXplIGJv
b3QgZGV2aWNlcywgbGVhdmluZyB0aGUgDQo+PiByZXN0IGZvciB0aGUgT1MgdG8gcHJvZ3JhbS4g
IFRoZSBpbml0aWFsIHByb3Bvc2FsIGhlcmUgdG8gZGlzYWJsZSANCj4+IFNSLUlPViBpZiBub3Qg
cHJvZ3JhbW1lZCBhdCBPUyBoYW5kLW9mZiBkaXNhYmxlcyBldmVuIHRoZSBwb3NzaWJpbGl0eSAN
Cj4+IG9mIHRoZSBPUyByZWFsbG9jYXRpbmcgcmVzb3VyY2VzIGZvciB0aGlzIGRldmljZS4NCj4N
Cj5UaGVyZSBhcmUgZGlmZmVyZW5jZXMgYmV0d2VlbiBzdXBwb3J0aW5nIFNSLUlPViBhbmQgc3Vw
cG9ydGluZyBTUi1JT1YgDQo+b24gZGV2aWNlcyB3aXRoIG1hc3NpdmUgcmVzb3VyY2VzLiBJIGtu
b3cgSSBoYXZlIHNlZW4gTklDcyB0aGF0IHdpbGwgDQo+a2VlcCBhIHN5c3RlbSBmcm9tIGNvbXBs
ZXRpbmcgUE9TVCBpZiBTUi1JT1YgaXMgZW5hYmxlZCwgYW5kIE1NSU8gDQo+YmV5b25kIDRHIGlz
IG5vdC4gTXkgZ3Vlc3Mgd291bGQgYmUgdGhhdCB0aGUgaXNzdWVzIGJlaW5nIHNlZW4gYXJlIA0K
PnByb2JhYmx5IHRoYXQgdGhleSBkaXNhYmxlIFNSLUlPViBpbiB0aGUgQklPUyBpbiBzdWNoIGEg
c2V0dXAgYW5kIGVuZCANCj51cCBydW5uaW5nIGludG8gaXNzdWVzIHdoZW4gdGhleSB0cnkgdG8g
Ym9vdCBpbnRvIHRoZSBMaW51eCBrZXJuZWwgYXMgDQo+aXQgZ29lcyB0aHJvdWdoIGFuZCB0cmll
cyB0byBhbGxvY2F0ZSByZXNvdXJjZXMgZm9yIFNSLUlPViBldmVuIHRob3VnaCBpdCB3YXMgZGlz
YWJsZWQgaW4gdGhlIEJJT1MuDQo+DQo+SXQgbWlnaHQgbWFrZSBzZW5zZSB0byBhZGQgYSBrZXJu
ZWwgcGFyYW1ldGVyIHNvbWV0aGluZyBsaWtlIGEgInBjaT1ub3NyaW92Ig0KPnRoYXQgd291bGQg
YWxsb3cgZm9yIGRpc2FibGluZyBTUi1JT1YgYW5kIHJlbGF0ZWQgcmVzb3VyY2UgYWxsb2NhdGlv
biANCj5pZiB0aGF0IGlzIHdoYXQgd2UgYXJlIHRhbGtpbmcgYWJvdXQuIFRoYXQgd2F5IHlvdSBj
b3VsZCBwbHVnIGluIHRoZXNlIA0KPnR5cGVzIG9mIGRldmljZXMgaW50byBhIHN5c3RlbSB3aXRo
IGEgbGVnYWN5IGJpb3Mgb3IgdGhhdCBkb2Vzbid0IHdhbiANCj50byBhbGxvY2F0ZSBhZGRyZXNz
ZXMgYWJvdmUgMzJiIGZvciBNTUlPLCBhbmQgdGhpcyBwYXJhbWV0ZXIgd291bGQgYmUgDQo+YWxs
IHRoYXQgaXMgbmVlZGVkIHRvIGRpc2FibGUgU1ItSU9WIHNvIHlvdSBjb3VsZCBwbHVnIGluIGEg
TklDIHRoYXQgaGFzIFNSLUlPViBhc3NvY2lhdGVkIHdpdGggaXQuDQo+DQo+Pj4gSSBhZ3JlZSB0
aGF0IGZhaWx1cmUgdG8gYWxsb2NhdGUgVkYgcmVzb3VyY2VzIHNob3VsZCBsZWF2ZSB0aGUgDQo+
Pj4gZGV2aWNlIGluIG5vDQo+d29yc2UgY29uZGl0aW9uIHRoYW4gYmVmb3JlIGl0IHRyaWVkLiBJ
IGhvcGUga2VybmVsIGNvdWxkIGFsbG9jYXRlIFBGIA0KPmRldmljZSByZXNvdXJjZSBiZWZvcmUg
YWxsb2NhdGluZyBWRiBkZXZpY2UgcmVzb3VyY2UsIGFuZCBrZWVwIFBGIA0KPmRldmljZSByZXNv
dXJjZSB2YWxpZCBhbmQgZnVuY3Rpb25hbCBpZiBmYWlsZWQgdG8gYWxsb2NhdGUgVkYgZGV2aWNl
IHJlc291cmNlLg0KPj4+DQo+Pj4gSSB3aWxsIHNlbmQgb3V0IGRtZXNnIGxvZyBsc3BjaSBpbmZv
IHRvbW9ycm93LiBUaGFua3MuDQo+Pg0KPj4gVGhhbmtzLA0KPj4gQWxleA0KPj4NCj4+PiAtLS0t
LU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPj4+IEZyb206IEFsZXggV2lsbGlhbXNvbiBbbWFpbHRv
OmFsZXgud2lsbGlhbXNvbkByZWRoYXQuY29tXQ0KPj4+IFNlbnQ6IEZyaWRheSwgTWF5IDEyLCAy
MDE3IDEwOjQzIFBNDQo+Pj4gVG86IENoZW5nLCBDb2xsaW5zIDxDb2xsaW5zLkNoZW5nQGFtZC5j
b20+DQo+Pj4gQ2M6IEJqb3JuIEhlbGdhYXMgPGJoZWxnYWFzQGdvb2dsZS5jb20+OyBsaW51eC1w
Y2lAdmdlci5rZXJuZWwub3JnOyANCj4+PiBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnOyBE
ZXVjaGVyLCBBbGV4YW5kZXIgDQo+Pj4gPEFsZXhhbmRlci5EZXVjaGVyQGFtZC5jb20+OyBaeXRh
cnVrLCBLZWxseSA8S2VsbHkuWnl0YXJ1a0BhbWQuY29tPjsgDQo+Pj4gWWluZ2hhaSBMdSA8eWlu
Z2hhaUBrZXJuZWwub3JnPg0KPj4+IFN1YmplY3Q6IFJlOiBbUEFUQ0hdIFBDSTogTWFrZSBTUi1J
T1YgY2FwYWJsZSBHUFUgd29ya2luZyBvbiB0aGUgDQo+Pj4gU1ItSU9WIGluY2FwYWJsZSBwbGF0
Zm9ybQ0KPj4+DQo+Pj4gT24gRnJpLCAxMiBNYXkgMjAxNyAwNDo1MTo0MyArMDAwMA0KPj4+ICJD
aGVuZywgQ29sbGlucyIgPENvbGxpbnMuQ2hlbmdAYW1kLmNvbT4gd3JvdGU6DQo+Pj4NCj4+PiA+
IEhpIFdpbGxpYW1zb24sDQo+Pj4gPg0KPj4+ID4gSSB2ZXJpZmllZCB0aGUgcGF0Y2ggaXMgd29y
a2luZyBmb3IgYm90aCBBTUQgU1ItSU9WIEdQVSBhbmQgSW50ZWwgDQo+Pj4gPiBTUi1JT1YNCj5O
SUMuIEkgZG9uJ3QgdGhpbmsgaXQgaXMgcmVkdW5kYW50IHRvIGNoZWNrIHRoZSBWRiBCQVIgdmFs
aWQgYmVmb3JlIA0KPmNhbGwgc3Jpb3ZfaW5pdCgpLCBpdCBpcyBzYWZlIGFuZCBzYXZpbmcgYm9v
dCB0aW1lLCBhbHNvIHRoZXJlIGlzIG5vIGEgDQo+YmV0dGVyIG1ldGhvZCB0byBrbm93IGlmIHN5
c3RlbSBCSU9TIGhhcyBjb3JyZWN0bHkgaW5pdGlhbGl6ZWQgdGhlIFNSLUlPViBjYXBhYmlsaXR5
IG9yIG5vdC4NCj4+Pg0KPj4+IEl0IGFsc28gbWFza3MgYW4gdW5kZXJseWluZyBidWcgYW5kIGNy
ZWF0ZXMgYSBtYWludGVuYW5jZSBpc3N1ZSB0aGF0IA0KPj4+IHdlIHdvbid0DQo+a25vdyB3aGVu
IGl0J3Mgc2FmZSB0byByZW1vdmUgdGhpcyB3b3JrYXJvdW5kLiAgSSBkb24ndCB0aGluayBmYXN0
ZXIgDQo+Ym9vdCBpcyB2YWxpZCByYXRpb25hbGUsIGluIG9uZSBjYXNlIFNSLUlPViBpcyBjb21w
bGV0ZWx5IGRpc2FibGVkLCB0aGUgDQo+b3RoZXIgd2UgYXR0ZW1wdCB0byBhbGxvY2F0ZSB0aGUg
cmVzb3VyY2VzIHRoZSBCSU9TIGZhaWxlZCB0byBwcm92aWRlLiAgDQo+SSBleHBlY3QgdGhpcyBp
cyBhbHNvIGEgY29ybmVyIGNhc2UsIHRoZSBCSU9TIHNob3VsZCB0eXBpY2FsbHkgc3VwcG9ydCAN
Cj5TUi1JT1YsIHRoZXJlZm9yZSB0aGlzIHNpdHVhdGlvbiBzaG91bGQgYmUgYW4gZXhjZXB0aW9u
Lg0KPj4+DQo+Pj4gPiBJIGRpZCBub3QgdHJ5IHRvIGZpeCB0aGUgaXNzdWUgZnJvbSB0aGUga2Vy
bmVsIHJlc291cmNlIGFsbG9jYXRpb24gDQo+Pj4gPiBwZXJzcGVjdGl2ZSwNCj5pdCBpcyBiZWNh
dXNlOg0KPj4+ID4gICAgIDEuIEkgYW0gbm90IHZlcnkgZmFtaWxpYXIgd2l0aCB0aGUgUENJIHJl
c291cmNlIGFsbG9jYXRpb24gc2NoZW1lIGluIGtlcm5lbC4NCj5Gb3IgZXhhbXBsZSwgaW4gc3Jp
b3ZfaW5pdCgpLCBrZXJuZWwgd2lsbCByZS1hc3NpZ24gdGhlIFBDSSByZXNvdXJjZSANCj5mb3Ig
Ym90aCBWRiBhbmQgUEYuIEkgZG9uJ3QgdW5kZXJzdGFuZCB3aHkga2VybmVsIGFsbG9jYXRlcyBy
ZXNvdXJjZSANCj5mb3IgVkYgZmlyc3RseSwgdGhlbiBQRi4gSWYgaXQgaXMgUEYgZmlyc3RseSwg
dGhlbiB0aGlzIGlzc3VlIGNvdWxkIGJlIGF2b2lkZWQuDQo+Pj4gPiAgICAgMi4gSSBhbSBub3Qg
c3VyZSBpZiBrZXJuZWwgaGFzIGVycm9yIGhhbmRsZXIgaWYgUENJIHJlc291cmNlIGFsbG9jYXRp
b24gZmFpbGVkLg0KPkluIHRoaXMgY2FzZSwga2VybmVsIGNhbm5vdCBhbGxvY2F0ZSBlbm91Z2gg
cmVzb3VyY2UgdG8gUEYuIEl0IHNob3VsZCANCj50cmlnZ2VyIHNvbWUgZXJyb3IgaGFuZGxlciB0
byBlaXRoZXIganVzdCBrZWVwIG9yaWdpbmFsIEJBUiB2YWx1ZXMgc2V0IA0KPmJ5IHN5c3RlbSBC
SU9TLCBvciBkaXNhYmxlIHRoaXMgZGV2aWNlIGFuZCBsb2cgZXJyb3JzLg0KPj4+DQo+Pj4gSSB0
aGluayB0aGVzZSBhcmUgdGhlIGlzc3VlcyB3ZSBzaG91bGQgYmUgdHJ5aW5nIHRvIHNvbHZlIGFu
ZCBJJ20gDQo+Pj4gc3VyZSBmb2xrcyBvbiB0aGUgbGludXgtcGNpIGxpc3QgY2FuIGhlbHAgdXMg
aWRlbnRpZnkgdGhlIGJ1Zy4NCj4+PiBNaW5pbWFsbHksIGZhaWx1cmUgdG8gYWxsb2NhdGUgVkYg
cmVzb3VyY2VzIHNob3VsZCBsZWF2ZSB0aGUgZGV2aWNlIA0KPj4+IGluIG5vIHdvcnNlIGNvbmRp
dGlvbiB0aGFuIGJlZm9yZSBpdCB0cmllZC4gIFBlcmhhcHMgeW91IGNvdWxkIHBvc3QgDQo+Pj4g
bW9yZSBkZXRhaWxzIGFib3V0IHRoZSBpc3N1ZSwgYm9vdCB3aXRoIHBjaT1lYXJseWR1bXAsIHBv
c3QgZG1lc2cgb2YgDQo+Pj4gYSBib290IHdoZXJlIHRoZSBQRiByZXNvdXJjZXMgYXJlIGluY29y
cmVjdGx5IHJlLWFsbG9jYXRlZCwgYW5kIA0KPj4+IGluY2x1ZGUgbHNwY2kgLXZ2diBmb3IgdGhl
IFNSLUlPViBkZXZpY2UuICBBbHNvLCBwbGVhc2UgdGVzdCB3aXRoIA0KPj4+IHRoZSBsYXRlc3Qg
dXBzdHJlYW0ga2VybmVsLCB1cHN0cmVhbSBvbmx5IHBhdGNoZXMgb2xkIGtlcm5lbHMgDQo+Pj4g
dGhyb3VnaCBzdGFibGUgYmFja3BvcnRzIG9mIGNvbW1pdHMgdG8gdGhlIGxhdGVzdCBrZXJuZWwu
ICBBZGRpbmcgDQo+Pj4gWWluZ2hhaSBhcyBhIHJlc291cmNlIGFsbG9jYXRpb24gZXhwZXJ0LiBU
aGFua3MsDQo+Pj4NCj4+PiBBbGV4DQo+Pj4NCj4+PiA+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0t
LS0tDQo+Pj4gPiBGcm9tOiBBbGV4IFdpbGxpYW1zb24gW21haWx0bzphbGV4LndpbGxpYW1zb25A
cmVkaGF0LmNvbV0NCj4+PiA+IFNlbnQ6IEZyaWRheSwgTWF5IDEyLCAyMDE3IDEyOjAxIFBNDQo+
Pj4gPiBUbzogQ2hlbmcsIENvbGxpbnMgPENvbGxpbnMuQ2hlbmdAYW1kLmNvbT4NCj4+PiA+IENj
OiBCam9ybiBIZWxnYWFzIDxiaGVsZ2Fhc0Bnb29nbGUuY29tPjsgDQo+Pj4gPiBsaW51eC1wY2lA
dmdlci5rZXJuZWwub3JnOyBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnOyBEZXVjaGVyLCAN
Cj4+PiA+IEFsZXhhbmRlciA8QWxleGFuZGVyLkRldWNoZXJAYW1kLmNvbT47IFp5dGFydWssIEtl
bGx5IA0KPj4+ID4gPEtlbGx5Llp5dGFydWtAYW1kLmNvbT4NCj4+PiA+IFN1YmplY3Q6IFJlOiBb
UEFUQ0hdIFBDSTogTWFrZSBTUi1JT1YgY2FwYWJsZSBHUFUgd29ya2luZyBvbiB0aGUgDQo+Pj4g
PiBTUi1JT1YgaW5jYXBhYmxlIHBsYXRmb3JtDQo+Pj4gPg0KPj4+ID4gT24gRnJpLCAxMiBNYXkg
MjAxNyAwMzo0Mjo0NiArMDAwMCAiQ2hlbmcsIENvbGxpbnMiIA0KPj4+ID4gPENvbGxpbnMuQ2hl
bmdAYW1kLmNvbT4gd3JvdGU6DQo+Pj4gPg0KPj4+ID4gPiBIaSBXaWxsaWFtc29uLA0KPj4+ID4g
Pg0KPj4+ID4gPiBHUFUgY2FyZCBuZWVkcyBtb3JlIEJBUiBhcGVydHVyZSByZXNvdXJjZSB0aGFu
IG90aGVyIFBDSSANCj4+PiA+ID4gZGV2aWNlcy4gRm9yDQo+ZXhhbXBsZSwgSW50ZWwgU1ItSU9W
IG5ldHdvcmsgY2FyZCBvbmx5IHJlcXVpcmUgNTEyS0IgbWVtb3J5IHJlc291cmNlIA0KPmZvciBh
bGwgVkZzLiBBTUQgU1ItSU9WIEdQVSBjYXJkIG5lZWRzIDI1Nk1CIHgxNiBWRiA9IDRHQiBtZW1v
cnkgDQo+cmVzb3VyY2UgZm9yIGZyYW1lIGJ1ZmZlciBCQVIgYXBlcnR1cmUuDQo+Pj4gPiA+DQo+
Pj4gPiA+IElmIHRoZSBzeXN0ZW0gQklPUyBzdXBwb3J0cyBTUi1JT1YsIGl0IHdpbGwgcmVzZXJ2
ZSBlbm91Z2ggDQo+Pj4gPiA+IHJlc291cmNlIGZvciBhbGwNCj5WRiBCQVJzLiBJZiB0aGUgc3lz
dGVtIEJJT1MgZG9lc24ndCBzdXBwb3J0IFNSLUlPViBvciBjYW5ub3QgYWxsb2NhdGUgDQo+dGhl
IGVub3VnaCByZXNvdXJjZSBmb3IgVkYgQkFScywgb25seSBQRiBCQVIgd2lsbCBiZSBhc3NpZ25l
ZCBhbmQgVkYgDQo+QkFScyBhcmUgZW1wdHkuIFRoZW4gc3lzdGVtIGJvb3RzIHRvIExpbnV4IGtl
cm5lbCBhbmQga2VybmVsIGRvZXNuJ3QgDQo+Y2hlY2sgaWYgdGhlIFZGIEJBUnMgYXJlIGVtcHR5
IG9yIHZhbGlkLiBLZXJuZWwgd2lsbCByZS1hc3NpZ24gdGhlIEJBUiANCj5yZXNvdXJjZXMgZm9y
IFBGIGFuZCBhbGwgVkZzLiBUaGUgcHJvYmxlbSBJIHNhdyBpcyB0aGF0IGtlcm5lbCB3aWxsIA0K
PmZhaWwgdG8gYWxsb2NhdGUgUEYgQkFSIHJlc291cmNlIGJlY2F1c2Ugc29tZSByZXNvdXJjZXMg
YXJlIGFzc2lnbmVkIHRvIA0KPlZGLCB0aGlzIGlzIG5vdCBleHBlY3RlZC4gU28ga2VybmVsIG1p
Z2h0IG5lZWQgdG8gZG8gc29tZSBjaGVjayBiZWZvcmUgDQo+cmUtYXNzaWduIHRoZSBQRi9WRiBy
ZXNvdXJjZSwgc28gdGhhdCBQRiBkZXZpY2Ugd2lsbCBiZSBjb3JyZWN0bHkgYXNzaWduZWQgQkFS
IHJlc291cmNlIGFuZCB1c2VyIGNhbiB1c2UgUEYgZGV2aWNlLg0KPj4+ID4NCj4+PiA+IFNvIHRo
ZSBwcm9ibGVtIGlzIHRoYXQgc29tZXRoaW5nIGJhZCBoYXBwZW5zIHdoZW4gdGhlIGtlcm5lbCBp
cyANCj4+PiA+IHRyeWluZyB0byByZWFsbG9jYXRlIHJlc291cmNlcyBpbiBvcmRlciB0byBmdWxm
aWxsIHRoZSANCj4+PiA+IHJlcXVpcmVtZW50cyBvZiB0aGUgVkZzLCBsZWF2aW5nIHRoZSBQRiBy
ZXNvdXJjZXMgaW5jb3JyZWN0bHkgDQo+Pj4gPiBwcm9ncmFtbWVkPyAgV2h5IG5vdCBqdXN0IGZp
eCB0aGF0IGJ1ZyByYXRoZXIgdGhhbiBjcmVhdGluZyANCj4+PiA+IHNwZWNpYWwgaGFuZGxpbmcg
Zm9yIHRoaXMgdmVuZG9yL2NsYXNzIG9mIGRldmljZSB3aGljaCBkaXNhYmxlcyANCj4+PiA+IGFu
eSBhdHRlbXB0IHRvIGZpeHVwIHJlc291cmNlcyBmb3IgU1ItSU9WPyAgSU9XLCB0aGlzIHBhdGNo
IGp1c3QgDQo+Pj4gPiBhdm9pZHMgdGhlIHByb2JsZW0gZm9yIHlvdXIgZGV2aWNlcyByYXRoZXIg
dGhhbiBmaXhpbmcgdGhlIGJ1Zy4gIA0KPj4+ID4gSSdkIHN1Z2dlc3QgZml4aW5nIHRoZSBidWcg
c3VjaCB0aGF0IHRoZSBQRiBpcyBsZWZ0IGluIGEgDQo+Pj4gPiBmdW5jdGlvbmFsIHN0YXRlIGlm
IHRoZSBrZXJuZWwgaXMgdW5hYmxlIHRvIGFsbG9jYXRlIHN1ZmZpY2llbnQgDQo+Pj4gPiByZXNv
dXJjZXMgZm9yIHRoZSBWRnMuICBUaGFua3MsDQo+Pj4gPg0KPj4+ID4gQWxleA0KPj4+ID4NCj4+
PiA+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4+PiA+ID4gRnJvbTogQWxleCBXaWxs
aWFtc29uIFttYWlsdG86YWxleC53aWxsaWFtc29uQHJlZGhhdC5jb21dDQo+Pj4gPiA+IFNlbnQ6
IEZyaWRheSwgTWF5IDEyLCAyMDE3IDExOjIxIEFNDQo+Pj4gPiA+IFRvOiBDaGVuZywgQ29sbGlu
cyA8Q29sbGlucy5DaGVuZ0BhbWQuY29tPg0KPj4+ID4gPiBDYzogQmpvcm4gSGVsZ2FhcyA8Ymhl
bGdhYXNAZ29vZ2xlLmNvbT47IA0KPj4+ID4gPiBsaW51eC1wY2lAdmdlci5rZXJuZWwub3JnOyBs
aW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnOyANCj4+PiA+ID4gRGV1Y2hlciwgQWxleGFuZGVy
IDxBbGV4YW5kZXIuRGV1Y2hlckBhbWQuY29tPjsgWnl0YXJ1aywgS2VsbHkgDQo+Pj4gPiA+IDxL
ZWxseS5aeXRhcnVrQGFtZC5jb20+DQo+Pj4gPiA+IFN1YmplY3Q6IFJlOiBbUEFUQ0hdIFBDSTog
TWFrZSBTUi1JT1YgY2FwYWJsZSBHUFUgd29ya2luZyBvbiB0aGUgDQo+Pj4gPiA+IFNSLUlPViBp
bmNhcGFibGUgcGxhdGZvcm0NCj4+PiA+ID4NCj4+PiA+ID4gT24gRnJpLCAxMiBNYXkgMjAxNyAw
Mjo1MDozMiArMDAwMCAiQ2hlbmcsIENvbGxpbnMiDQo+Pj4gPiA+IDxDb2xsaW5zLkNoZW5nQGFt
ZC5jb20+IHdyb3RlOg0KPj4+ID4gPg0KPj4+ID4gPiA+IEhpIEhlbGdhYXMsDQo+Pj4gPiA+ID4N
Cj4+PiA+ID4gPiBTb21lIEFNRCBHUFVzIGhhdmUgaGFyZHdhcmUgc3VwcG9ydCBmb3IgZ3JhcGhp
Y3MgU1ItSU9WLg0KPj4+ID4gPiA+IElmIHRoZSBTUi1JT1YgY2FwYWJsZSBHUFUgaXMgcGx1Z2dl
ZCBpbnRvIHRoZSBTUi1JT1YgaW5jYXBhYmxlIA0KPj4+ID4gPiA+IHBsYXRmb3JtLiBJdCB3b3Vs
ZCBjYXVzZSBhIHByb2JsZW0gb24gUENJIHJlc291cmNlIGFsbG9jYXRpb24gDQo+Pj4gPiA+ID4g
aW4gY3VycmVudCBMaW51eCBrZXJuZWwuDQo+Pj4gPiA+ID4NCj4+PiA+ID4gPiBUaGVyZWZvcmUg
aW4gb3JkZXIgdG8gYWxsb3cgdGhlIFBGIChQaHlzaWNhbCBGdW5jdGlvbikgZGV2aWNlIA0KPj4+
ID4gPiA+IG9mIFNSLUlPViBjYXBhYmxlIEdQVSB0byB3b3JrIG9uIHRoZSBTUi1JT1YgaW5jYXBh
YmxlIA0KPj4+ID4gPiA+IHBsYXRmb3JtLCBpdCBpcyByZXF1aXJlZCB0byB2ZXJpZnkgY29uZGl0
aW9ucyBmb3IgaW5pdGlhbGl6aW5nIA0KPj4+ID4gPiA+IEJBUiByZXNvdXJjZXMgb24gQU1EIFNS
LUlPViBjYXBhYmxlIEdQVXMuDQo+Pj4gPiA+ID4NCj4+PiA+ID4gPiBJZiB0aGUgZGV2aWNlIGlz
IGFuIEFNRCBncmFwaGljcyBkZXZpY2UgYW5kIGl0IHN1cHBvcnRzIFNSLUlPViANCj4+PiA+ID4g
PiBpdCB3aWxsIHJlcXVpcmUgYSBsYXJnZSBhbW91bnQgb2YgcmVzb3VyY2VzLg0KPj4+ID4gPiA+
IEJlZm9yZSBjYWxsaW5nIHNyaW92X2luaXQoKSBtdXN0IGVuc3VyZSB0aGF0IHRoZSBzeXN0ZW0g
QklPUyANCj4+PiA+ID4gPiBhbHNvIHN1cHBvcnRzIFNSLUlPViBhbmQgdGhhdCBzeXN0ZW0gQklP
UyBoYXMgYmVlbiBhYmxlIHRvIA0KPj4+ID4gPiA+IGFsbG9jYXRlIGVub3VnaCByZXNvdXJjZXMu
DQo+Pj4gPiA+ID4gSWYgdGhlIFZGIEJBUnMgYXJlIHplcm8gdGhlbiB0aGUgc3lzdGVtIEJJT1Mg
ZG9lcyBub3Qgc3VwcG9ydCANCj4+PiA+ID4gPiBTUi1JT1Ygb3IgaXQgY291bGQgbm90IGFsbG9j
YXRlIHRoZSByZXNvdXJjZXMgYW5kIHRoaXMgDQo+Pj4gPiA+ID4gcGxhdGZvcm0gd2lsbCBub3Qg
c3VwcG9ydCBBTUQgZ3JhcGhpY3MgU1ItSU9WLg0KPj4+ID4gPiA+IFRoZXJlZm9yZSBkbyBub3Qg
Y2FsbCBzcmlvdl9pbml0KCkuDQo+Pj4gPiA+ID4gSWYgdGhlIHN5c3RlbSBCSU9TIGRvZXMgc3Vw
cG9ydCBTUi1JT1YgdGhlbiB0aGUgVkYgQkFScyB3aWxsIA0KPj4+ID4gPiA+IGJlIHByb3Blcmx5
IGluaXRpYWxpemVkIHRvIG5vbi16ZXJvIHZhbHVlcy4NCj4+PiA+ID4gPg0KPj4+ID4gPiA+IEJl
bG93IGlzIHRoZSBwYXRjaCBhZ2FpbnN0IHRvIEtlcm5lbCA0LjggJiA0LjkuIFBsZWFzZSByZXZp
ZXcuDQo+Pj4gPiA+ID4NCj4+PiA+ID4gPiBJIGNoZWNrZWQgdGhlIGRyaXZlcnMvcGNpL3F1aXJr
cy5jLCBpdCBsb29rcyB0aGUgDQo+Pj4gPiA+ID4gd29ya2Fyb3VuZHMvZml4ZXMgaW4gcXVpcmtz
LmMgYXJlIGZvciBzcGVjaWZpYyBkZXZpY2VzIGFuZCBvbmUgDQo+Pj4gPiA+ID4gb3IgbW9yZSBk
ZXZpY2UgSUQgYXJlIGRlZmluZWQgZm9yIHRoZSBzcGVjaWZpYyBkZXZpY2VzLiANCj4+PiA+ID4g
PiBIb3dldmVyIG15IHBhdGNoIGlzIGZvciBhbGwgQU1EIFNSLUlPViBjYXBhYmxlIEdQVXMsIHRo
YXQgDQo+Pj4gPiA+ID4gaW5jbHVkZXMgYWxsIGV4aXN0aW5nDQo+YW5kIGZ1dHVyZSBBTUQgc2Vy
dmVyIEdQVXMuDQo+Pj4gPiA+ID4gU28gaXQgZG9lc24ndCBzZWVtIGxpa2UgYSBnb29kIGZpdCB0
byBwdXQgdGhlIGZpeCBpbiBxdWlya3MuYy4NCj4+PiA+ID4NCj4+PiA+ID4NCj4+PiA+ID4gV2h5
IGlzIGFuIEFNRCBncmFwaGljcyBjYXJkIHVuaXF1ZSBoZXJlPyAgRG9lc24ndCBzcmlvdl9pbml0
KCkgDQo+Pj4gPiA+IGFsd2F5cyBuZWVkIHRvIGJlIGFibGUgdG8gZGVhbCB3aXRoIGRldmljZXMg
b2YgYW55IHR5cGUgd2hlcmUgDQo+Pj4gPiA+IHRoZSBCSU9TIGhhc24ndCBpbml0aWFsaXplZCB0
aGUgU1ItSU9WIGNhcGFiaWxpdHk/ICBTb21lIFNSLUlPViANCj4+PiA+ID4gZGV2aWNlcyBjYW4g
Zml0IHRoZWlyIFZGcyB3aXRoaW4gYSBtaW5pbXVtIGJyaWRnZSBhcGVydHVyZSwgbW9zdCANCj4+
PiA+ID4gY2Fubm90LiAgSSBkb24ndCB1bmRlcnN0YW5kIHdoeSB0aGUgVkYgcmVzb3VyY2UgcmVx
dWlyZW1lbnRzIA0KPj4+ID4gPiBiZWluZyBleGNlcHRpb25hbGx5IGxhcmdlIGRpY3RhdGVzIHRo
YXQgdGhleSByZWNlaXZlIHNwZWNpYWwgaGFuZGxpbmcuDQo+Pj4gPiA+IFRoYW5rcywNCj4+PiA+
ID4NCj4+PiA+ID4gQWxleA0KPj4+ID4gPg0KPj4+ID4gPiA+IFNpZ25lZC1vZmYtYnk6IENvbGxp
bnMgQ2hlbmcgPGNvbGxpbnMuY2hlbmdAYW1kLmNvbT4NCj4+PiA+ID4gPiAtLS0NCj4+PiA+ID4g
PiAgZHJpdmVycy9wY2kvaW92LmMgfCA2Mw0KPj4+ID4gPiA+ICsrKysrKysrKysrKysrKysrKysr
KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKystLS0NCj4+PiA+ID4gPiAgMSBmaWxlIGNo
YW5nZWQsIDYwIGluc2VydGlvbnMoKyksIDMgZGVsZXRpb25zKC0pDQo+Pj4gPiA+ID4NCj4+PiA+
ID4gPiBkaWZmIC0tZ2l0IGEvZHJpdmVycy9wY2kvaW92LmMgYi9kcml2ZXJzL3BjaS9pb3YuYyBp
bmRleA0KPj4+ID4gPiA+IGUzMGYwNWMuLmU0ZjE0MDUgMTAwNjQ0DQo+Pj4gPiA+ID4gLS0tIGEv
ZHJpdmVycy9wY2kvaW92LmMNCj4+PiA+ID4gPiArKysgYi9kcml2ZXJzL3BjaS9pb3YuYw0KPj4+
ID4gPiA+IEBAIC01MjMsNiArNTIzLDQ1IEBAIHN0YXRpYyB2b2lkIHNyaW92X3Jlc3RvcmVfc3Rh
dGUoc3RydWN0IA0KPj4+ID4gPiA+IHBjaV9kZXYNCj4qZGV2KQ0KPj4+ID4gPiA+ICAgICAgICAg
ICAgICAgICBtc2xlZXAoMTAwKTsNCj4+PiA+ID4gPiAgfQ0KPj4+ID4gPiA+DQo+Pj4gPiA+ID4g
Ky8qDQo+Pj4gPiA+ID4gKyAqIHBjaV92Zl9iYXJfdmFsaWQgLSBjaGVjayBpZiBWRiBCQVJzIGhh
dmUgcmVzb3VyY2UgDQo+Pj4gPiA+ID4gK2FsbG9jYXRlZA0KPj4+ID4gPiA+ICsgKiBAZGV2OiB0
aGUgUENJIGRldmljZQ0KPj4+ID4gPiA+ICsgKiBAcG9zOiByZWdpc3RlciBvZmZzZXQgb2YgU1It
SU9WIGNhcGFiaWxpdHkgaW4gUENJIGNvbmZpZyANCj4+PiA+ID4gPiArc3BhY2UNCj4+PiA+ID4g
PiArICogUmV0dXJucyB0cnVlIGFueSBWRiBCQVIgaGFzIHJlc291cmNlIGFsbG9jYXRlZCwgZmFs
c2UNCj4+PiA+ID4gPiArICogaWYgYWxsIFZGIEJBUnMgYXJlIGVtcHR5Lg0KPj4+ID4gPiA+ICsg
Ki8NCj4+PiA+ID4gPiArc3RhdGljIGJvb2wgcGNpX3ZmX2Jhcl92YWxpZChzdHJ1Y3QgcGNpX2Rl
diAqZGV2LCBpbnQgcG9zKSB7DQo+Pj4gPiA+ID4gKyAgICAgICBpbnQgaTsNCj4+PiA+ID4gPiAr
ICAgICAgIHUzMiBiYXJfdmFsdWU7DQo+Pj4gPiA+ID4gKyAgICAgICB1MzIgYmFyX3NpemVfbWFz
ayA9IH4oUENJX0JBU0VfQUREUkVTU19TUEFDRSB8DQo+Pj4gPiA+ID4gKyAgICAgICAgICAgICAg
ICAgICAgICAgUENJX0JBU0VfQUREUkVTU19NRU1fVFlQRV82NCB8DQo+Pj4gPiA+ID4gKyAgICAg
ICAgICAgICAgICAgICAgICAgUENJX0JBU0VfQUREUkVTU19NRU1fUFJFRkVUQ0gpOw0KPj4+ID4g
PiA+ICsNCj4+PiA+ID4gPiArICAgICAgIGZvciAoaSA9IDA7IGkgPCBQQ0lfU1JJT1ZfTlVNX0JB
UlM7IGkrKykgew0KPj4+ID4gPiA+ICsgICAgICAgICAgICAgICBwY2lfcmVhZF9jb25maWdfZHdv
cmQoZGV2LCBwb3MgKyBQQ0lfU1JJT1ZfQkFSIA0KPj4+ID4gPiA+ICsgKyBpICogNCwNCj4mYmFy
X3ZhbHVlKTsNCj4+PiA+ID4gPiArICAgICAgICAgICAgICAgaWYgKGJhcl92YWx1ZSAmIGJhcl9z
aXplX21hc2spDQo+Pj4gPiA+ID4gKyAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIHRydWU7
DQo+Pj4gPiA+ID4gKyAgICAgICB9DQo+Pj4gPiA+ID4gKw0KPj4+ID4gPiA+ICsgICAgICAgcmV0
dXJuIGZhbHNlOw0KPj4+ID4gPiA+ICt9DQo+Pj4gPiA+ID4gKw0KPj4+ID4gPiA+ICsvKg0KPj4+
ID4gPiA+ICsgKiBpc19hbWRfZGlzcGxheV9hZGFwdGVyIC0gY2hlY2sgaWYgaXQgaXMgYW4gQU1E
L0FUSSBHUFUgDQo+Pj4gPiA+ID4gK2RldmljZQ0KPj4+ID4gPiA+ICsgKiBAZGV2OiB0aGUgUENJ
IGRldmljZQ0KPj4+ID4gPiA+ICsgKg0KPj4+ID4gPiA+ICsgKiBSZXR1cm5zIHRydWUgaWYgZGV2
aWNlIGlzIGFuIEFNRC9BVEkgZGlzcGxheSBhZGFwdGVyLA0KPj4+ID4gPiA+ICsgKiBvdGhlcndp
c2UgcmV0dXJuIGZhbHNlLg0KPj4+ID4gPiA+ICsgKi8NCj4+PiA+ID4gPiArDQo+Pj4gPiA+ID4g
K3N0YXRpYyBib29sIGlzX2FtZF9kaXNwbGF5X2FkYXB0ZXIoc3RydWN0IHBjaV9kZXYgKmRldikg
ew0KPj4+ID4gPiA+ICsgICAgICAgcmV0dXJuICgoKGRldi0+Y2xhc3MgPj4gMTYpID09IFBDSV9C
QVNFX0NMQVNTX0RJU1BMQVkpICYmDQo+Pj4gPiA+ID4gKyAgICAgICAgICAgICAgIChkZXYtPnZl
bmRvciA9PSBQQ0lfVkVORE9SX0lEX0FUSSB8fA0KPj4+ID4gPiA+ICsgICAgICAgICAgICAgICBk
ZXYtPnZlbmRvciA9PSBQQ0lfVkVORE9SX0lEX0FNRCkpOyB9DQo+Pj4gPiA+ID4gKw0KPj4+ID4g
PiA+ICAvKioNCj4+PiA+ID4gPiAgICogcGNpX2lvdl9pbml0IC0gaW5pdGlhbGl6ZSB0aGUgSU9W
IGNhcGFiaWxpdHkNCj4+PiA+ID4gPiAgICogQGRldjogdGhlIFBDSSBkZXZpY2UNCj4+PiA+ID4g
PiBAQCAtNTM3LDkgKzU3NiwyNyBAQCBpbnQgcGNpX2lvdl9pbml0KHN0cnVjdCBwY2lfZGV2ICpk
ZXYpDQo+Pj4gPiA+ID4gICAgICAgICAgICAgICAgIHJldHVybiAtRU5PREVWOw0KPj4+ID4gPiA+
DQo+Pj4gPiA+ID4gICAgICAgICBwb3MgPSBwY2lfZmluZF9leHRfY2FwYWJpbGl0eShkZXYsIFBD
SV9FWFRfQ0FQX0lEX1NSSU9WKTsNCj4+PiA+ID4gPiAtICAgICAgIGlmIChwb3MpDQo+Pj4gPiA+
ID4gLSAgICAgICAgICAgICAgIHJldHVybiBzcmlvdl9pbml0KGRldiwgcG9zKTsNCj4+PiA+ID4g
PiAtDQo+Pj4gPiA+ID4gKyAgICAgICBpZiAocG9zKSB7DQo+Pj4gPiA+ID4gKyAgICAgICAvKg0K
Pj4+ID4gPiA+ICsgICAgICAgICogSWYgdGhlIGRldmljZSBpcyBhbiBBTUQgZ3JhcGhpY3MgZGV2
aWNlIGFuZCBpdCBzdXBwb3J0cw0KPj4+ID4gPiA+ICsgICAgICAgICogU1ItSU9WIGl0IHdpbGwg
cmVxdWlyZSBhIGxhcmdlIGFtb3VudCBvZiByZXNvdXJjZXMuDQo+Pj4gPiA+ID4gKyAgICAgICAg
KiBCZWZvcmUgY2FsbGluZyBzcmlvdl9pbml0KCkgbXVzdCBlbnN1cmUgdGhhdCB0aGUgc3lzdGVt
DQo+Pj4gPiA+ID4gKyAgICAgICAgKiBCSU9TIGFsc28gc3VwcG9ydHMgU1ItSU9WIGFuZCB0aGF0
IHN5c3RlbSBCSU9TIGhhcyBiZWVuDQo+Pj4gPiA+ID4gKyAgICAgICAgKiBhYmxlIHRvIGFsbG9j
YXRlIGVub3VnaCByZXNvdXJjZXMuDQo+Pj4gPiA+ID4gKyAgICAgICAgKiBJZiB0aGUgVkYgQkFS
cyBhcmUgemVybyB0aGVuIHRoZSBzeXN0ZW0gQklPUyBkb2VzIG5vdA0KPj4+ID4gPiA+ICsgICAg
ICAgICogc3VwcG9ydCBTUi1JT1Ygb3IgaXQgY291bGQgbm90IGFsbG9jYXRlIHRoZSByZXNvdXJj
ZXMNCj4+PiA+ID4gPiArICAgICAgICAqIGFuZCB0aGlzIHBsYXRmb3JtIHdpbGwgbm90IHN1cHBv
cnQgQU1EIGdyYXBoaWNzIFNSLUlPVi4NCj4+PiA+ID4gPiArICAgICAgICAqIFRoZXJlZm9yZSBk
byBub3QgY2FsbCBzcmlvdl9pbml0KCkuDQo+Pj4gPiA+ID4gKyAgICAgICAgKiBJZiB0aGUgc3lz
dGVtIEJJT1MgZG9lcyBzdXBwb3J0IFNSLUlPViB0aGVuIHRoZSBWRiBCQVJzDQo+Pj4gPiA+ID4g
KyAgICAgICAgKiB3aWxsIGJlIHByb3Blcmx5IGluaXRpYWxpemVkIHRvIG5vbi16ZXJvIHZhbHVl
cy4NCj4+PiA+ID4gPiArICAgICAgICAqLw0KPj4+ID4gPiA+ICsgICAgICAgICAgICAgICBpZiAo
aXNfYW1kX2Rpc3BsYXlfYWRhcHRlcihkZXYpKSB7DQo+Pj4gPiA+ID4gKyAgICAgICAgICAgICAg
ICAgICAgICAgaWYgKHBjaV92Zl9iYXJfdmFsaWQoZGV2LCBwb3MpKQ0KPj4+ID4gPiA+ICsgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIHNyaW92X2luaXQoZGV2LCBwb3MpOw0K
Pj4+ID4gPiA+ICsgICAgICAgICAgICAgICB9IGVsc2Ugew0KPj4+ID4gPiA+ICsgICAgICAgICAg
ICAgICAgICAgICAgIHJldHVybiBzcmlvdl9pbml0KGRldiwgcG9zKTsNCj4+PiA+ID4gPiArICAg
ICAgICAgICAgICAgfQ0KPj4+ID4gPiA+ICsgICAgICAgfQ0KPj4+ID4gPiA+ICAgICAgICAgcmV0
dXJuIC1FTk9ERVY7DQo+Pj4gPiA+ID4gIH0NCj4+PiA+ID4gPg0KPj4+ID4gPg0KPj4+ID4NCj4+
Pg0KPj4NCg==

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Zytaruk, Kelly]

Collins,

Okay, good to know.
Is there a common solution that can handle all cases?

Thanks,
Kelly

>-----Original Message-----
>From: Cheng, Collins
>Sent: Saturday, May 20, 2017 6:38 AM
>To: Zytaruk, Kelly; Alexander Duyck; Alex Williamson
>Cc: Bjorn Helgaas; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
>Deucher, Alexander; Yinghai Lu
>Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>incapable platform
>
>Hi Kelly,
>
>This issue also happens in "not SR-IOV capable" SBIOS. It seems some "not SR-IOV
>capable" SBIOS will directly report error in system BIOS boot stage and doesn't
>boot to OS. But other "not SR-IOV capable" SBIOS would not report error and
>boot to Linux.
>
>-Collins Cheng
>
>
>-----Original Message-----
>From: Zytaruk, Kelly
>Sent: Saturday, May 20, 2017 6:28 PM
>To: Cheng, Collins <Collins.Cheng@amd.com>; Alexander Duyck
><alexander.duyck@gmail.com>; Alex Williamson <alex.williamson@redhat.com>
>Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-
>kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>;
>Yinghai Lu <yinghai@kernel.org>
>Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>incapable platform
>
>
>
>>-----Original Message-----
>>From: Cheng, Collins
>>Sent: Saturday, May 20, 2017 12:53 AM
>>To: Alexander Duyck; Alex Williamson
>>Cc: Bjorn Helgaas; linux-pci@vger.kernel.org;
>>linux-kernel@vger.kernel.org; Deucher, Alexander; Zytaruk, Kelly;
>>Yinghai Lu
>>Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>>incapable platform
>>
>>Hi Alex,
>>
>>Yes, I hope kernel can disable SR-IOV and related VF resource
>>allocation if the system BIOS is not SR-IOV capable.
>>
>>Adding the parameter "pci=nosriov" sounds a doable solution, but it
>>would need user to add this parameter manually, right? I think an
>>automatic detection would be better. My patch is trying to auto detect and
>bypass VF resource allocation.
>>
>>
>>-Collins Cheng
>>
>
>Collins, be careful about this.  I don't think that this is what we want.  If you add
>"pci=nosriov" then you are globally disabling SRIOV for all devices.  This is not the
>solution that we are looking for.
>Remember that there are 3 types of SBIOS; "not SR-IOV capable", "SR-IOV
>capable but does not support large resources", "Complete SR-IOV support".
>
>The problem is that we are trying to find a fix for "broken" SBIOS that does
>support SR-IOV but does not support the full SR-IOV capabilities that devices with
>large resources require.
>
>Thanks,
>Kelly
>
>>
>>-----Original Message-----
>>From: Alexander Duyck [mailto:alexander.duyck@gmail.com]
>>Sent: Friday, May 19, 2017 11:44 PM
>>To: Alex Williamson <alex.williamson@redhat.com>
>>Cc: Cheng, Collins <Collins.Cheng@amd.com>; Bjorn Helgaas
>><bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-
>>kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>;
>>Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
>>Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>>incapable platform
>>
>>On Mon, May 15, 2017 at 10:53 AM, Alex Williamson
>><alex.williamson@redhat.com> wrote:
>>> On Mon, 15 May 2017 08:19:28 +0000
>>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>>
>>>> Hi Williamson,
>>>>
>>>> We cannot assume BIOS supports SR-IOV, actually only newer server
>>motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or
>>older server motherboard BIOS doesn't support SR-IOV. This issue would
>>happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop
>motherboard.
>>>
>>> Servers should be supporting SR-IOV for a long time now.  What really
>>> is there to a BIOS supporting SR-IOV anyway, it's simply reserving
>>> sufficient bus number and MMIO resources such that we can enable the
>>> VFs.  This process isn't exclusively reserved for the BIOS.  Some
>>> platforms may choose to only initialize boot devices, leaving the
>>> rest for the OS to program.  The initial proposal here to disable
>>> SR-IOV if not programmed at OS hand-off disables even the possibility
>>> of the OS reallocating resources for this device.
>>
>>There are differences between supporting SR-IOV and supporting SR-IOV
>>on devices with massive resources. I know I have seen NICs that will
>>keep a system from completing POST if SR-IOV is enabled, and MMIO
>>beyond 4G is not. My guess would be that the issues being seen are
>>probably that they disable SR-IOV in the BIOS in such a setup and end
>>up running into issues when they try to boot into the Linux kernel as
>>it goes through and tries to allocate resources for SR-IOV even though it was
>disabled in the BIOS.
>>
>>It might make sense to add a kernel parameter something like a "pci=nosriov"
>>that would allow for disabling SR-IOV and related resource allocation
>>if that is what we are talking about. That way you could plug in these
>>types of devices into a system with a legacy bios or that doesn't wan
>>to allocate addresses above 32b for MMIO, and this parameter would be
>>all that is needed to disable SR-IOV so you could plug in a NIC that has SR-IOV
>associated with it.
>>
>>>> I agree that failure to allocate VF resources should leave the
>>>> device in no
>>worse condition than before it tried. I hope kernel could allocate PF
>>device resource before allocating VF device resource, and keep PF
>>device resource valid and functional if failed to allocate VF device resource.
>>>>
>>>> I will send out dmesg log lspci info tomorrow. Thanks.
>>>
>>> Thanks,
>>> Alex
>>>
>>>> -----Original Message-----
>>>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>>> Sent: Friday, May 12, 2017 10:43 PM
>>>> To: Cheng, Collins <Collins.Cheng@amd.com>
>>>> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>>>> linux-kernel@vger.kernel.org; Deucher, Alexander
>>>> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>;
>>>> Yinghai Lu <yinghai@kernel.org>
>>>> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>>>> SR-IOV incapable platform
>>>>
>>>> On Fri, 12 May 2017 04:51:43 +0000
>>>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>>>
>>>> > Hi Williamson,
>>>> >
>>>> > I verified the patch is working for both AMD SR-IOV GPU and Intel
>>>> > SR-IOV
>>NIC. I don't think it is redundant to check the VF BAR valid before
>>call sriov_init(), it is safe and saving boot time, also there is no a
>>better method to know if system BIOS has correctly initialized the SR-IOV
>capability or not.
>>>>
>>>> It also masks an underlying bug and creates a maintenance issue that
>>>> we won't
>>know when it's safe to remove this workaround.  I don't think faster
>>boot is valid rationale, in one case SR-IOV is completely disabled, the
>>other we attempt to allocate the resources the BIOS failed to provide.
>>I expect this is also a corner case, the BIOS should typically support
>>SR-IOV, therefore this situation should be an exception.
>>>>
>>>> > I did not try to fix the issue from the kernel resource allocation
>>>> > perspective,
>>it is because:
>>>> >     1. I am not very familiar with the PCI resource allocation scheme in
>kernel.
>>For example, in sriov_init(), kernel will re-assign the PCI resource
>>for both VF and PF. I don't understand why kernel allocates resource
>>for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
>>>> >     2. I am not sure if kernel has error handler if PCI resource allocation
>failed.
>>In this case, kernel cannot allocate enough resource to PF. It should
>>trigger some error handler to either just keep original BAR values set
>>by system BIOS, or disable this device and log errors.
>>>>
>>>> I think these are the issues we should be trying to solve and I'm
>>>> sure folks on the linux-pci list can help us identify the bug.
>>>> Minimally, failure to allocate VF resources should leave the device
>>>> in no worse condition than before it tried.  Perhaps you could post
>>>> more details about the issue, boot with pci=earlydump, post dmesg of
>>>> a boot where the PF resources are incorrectly re-allocated, and
>>>> include lspci -vvv for the SR-IOV device.  Also, please test with
>>>> the latest upstream kernel, upstream only patches old kernels
>>>> through stable backports of commits to the latest kernel.  Adding
>>>> Yinghai as a resource allocation expert. Thanks,
>>>>
>>>> Alex
>>>>
>>>> > -----Original Message-----
>>>> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>>> > Sent: Friday, May 12, 2017 12:01 PM
>>>> > To: Cheng, Collins <Collins.Cheng@amd.com>
>>>> > Cc: Bjorn Helgaas <bhelgaas@google.com>;
>>>> > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher,
>>>> > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>>>> > <Kelly.Zytaruk@amd.com>
>>>> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>>>> > SR-IOV incapable platform
>>>> >
>>>> > On Fri, 12 May 2017 03:42:46 +0000 "Cheng, Collins"
>>>> > <Collins.Cheng@amd.com> wrote:
>>>> >
>>>> > > Hi Williamson,
>>>> > >
>>>> > > GPU card needs more BAR aperture resource than other PCI
>>>> > > devices. For
>>example, Intel SR-IOV network card only require 512KB memory resource
>>for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory
>>resource for frame buffer BAR aperture.
>>>> > >
>>>> > > If the system BIOS supports SR-IOV, it will reserve enough
>>>> > > resource for all
>>VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate
>>the enough resource for VF BARs, only PF BAR will be assigned and VF
>>BARs are empty. Then system boots to Linux kernel and kernel doesn't
>>check if the VF BARs are empty or valid. Kernel will re-assign the BAR
>>resources for PF and all VFs. The problem I saw is that kernel will
>>fail to allocate PF BAR resource because some resources are assigned to
>>VF, this is not expected. So kernel might need to do some check before
>>re-assign the PF/VF resource, so that PF device will be correctly assigned BAR
>resource and user can use PF device.
>>>> >
>>>> > So the problem is that something bad happens when the kernel is
>>>> > trying to reallocate resources in order to fulfill the
>>>> > requirements of the VFs, leaving the PF resources incorrectly
>>>> > programmed?  Why not just fix that bug rather than creating
>>>> > special handling for this vendor/class of device which disables
>>>> > any attempt to fixup resources for SR-IOV?  IOW, this patch just
>>>> > avoids the problem for your devices rather than fixing the bug.
>>>> > I'd suggest fixing the bug such that the PF is left in a
>>>> > functional state if the kernel is unable to allocate sufficient
>>>> > resources for the VFs.  Thanks,
>>>> >
>>>> > Alex
>>>> >
>>>> > > -----Original Message-----
>>>> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>>> > > Sent: Friday, May 12, 2017 11:21 AM
>>>> > > To: Cheng, Collins <Collins.Cheng@amd.com>
>>>> > > Cc: Bjorn Helgaas <bhelgaas@google.com>;
>>>> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
>>>> > > Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>>>> > > <Kelly.Zytaruk@amd.com>
>>>> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>>>> > > SR-IOV incapable platform
>>>> > >
>>>> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins"
>>>> > > <Collins.Cheng@amd.com> wrote:
>>>> > >
>>>> > > > Hi Helgaas,
>>>> > > >
>>>> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
>>>> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable
>>>> > > > platform. It would cause a problem on PCI resource allocation
>>>> > > > in current Linux kernel.
>>>> > > >
>>>> > > > Therefore in order to allow the PF (Physical Function) device
>>>> > > > of SR-IOV capable GPU to work on the SR-IOV incapable
>>>> > > > platform, it is required to verify conditions for initializing
>>>> > > > BAR resources on AMD SR-IOV capable GPUs.
>>>> > > >
>>>> > > > If the device is an AMD graphics device and it supports SR-IOV
>>>> > > > it will require a large amount of resources.
>>>> > > > Before calling sriov_init() must ensure that the system BIOS
>>>> > > > also supports SR-IOV and that system BIOS has been able to
>>>> > > > allocate enough resources.
>>>> > > > If the VF BARs are zero then the system BIOS does not support
>>>> > > > SR-IOV or it could not allocate the resources and this
>>>> > > > platform will not support AMD graphics SR-IOV.
>>>> > > > Therefore do not call sriov_init().
>>>> > > > If the system BIOS does support SR-IOV then the VF BARs will
>>>> > > > be properly initialized to non-zero values.
>>>> > > >
>>>> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
>>>> > > >
>>>> > > > I checked the drivers/pci/quirks.c, it looks the
>>>> > > > workarounds/fixes in quirks.c are for specific devices and one
>>>> > > > or more device ID are defined for the specific devices.
>>>> > > > However my patch is for all AMD SR-IOV capable GPUs, that
>>>> > > > includes all existing
>>and future AMD server GPUs.
>>>> > > > So it doesn't seem like a good fit to put the fix in quirks.c.
>>>> > >
>>>> > >
>>>> > > Why is an AMD graphics card unique here?  Doesn't sriov_init()
>>>> > > always need to be able to deal with devices of any type where
>>>> > > the BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV
>>>> > > devices can fit their VFs within a minimum bridge aperture, most
>>>> > > cannot.  I don't understand why the VF resource requirements
>>>> > > being exceptionally large dictates that they receive special handling.
>>>> > > Thanks,
>>>> > >
>>>> > > Alex
>>>> > >
>>>> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>>>> > > > ---
>>>> > > >  drivers/pci/iov.c | 63
>>>> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>>>> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
>>>> > > >
>>>> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>>>> > > > e30f05c..e4f1405 100644
>>>> > > > --- a/drivers/pci/iov.c
>>>> > > > +++ b/drivers/pci/iov.c
>>>> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct
>>>> > > > pci_dev
>>*dev)
>>>> > > >                 msleep(100);
>>>> > > >  }
>>>> > > >
>>>> > > > +/*
>>>> > > > + * pci_vf_bar_valid - check if VF BARs have resource
>>>> > > > +allocated
>>>> > > > + * @dev: the PCI device
>>>> > > > + * @pos: register offset of SR-IOV capability in PCI config
>>>> > > > +space
>>>> > > > + * Returns true any VF BAR has resource allocated, false
>>>> > > > + * if all VF BARs are empty.
>>>> > > > + */
>>>> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>>>> > > > +       int i;
>>>> > > > +       u32 bar_value;
>>>> > > > +       u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
>>>> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
>>>> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
>>>> > > > +
>>>> > > > +       for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>>>> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR
>>>> > > > + + i * 4,
>>&bar_value);
>>>> > > > +               if (bar_value & bar_size_mask)
>>>> > > > +                       return true;
>>>> > > > +       }
>>>> > > > +
>>>> > > > +       return false;
>>>> > > > +}
>>>> > > > +
>>>> > > > +/*
>>>> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU
>>>> > > > +device
>>>> > > > + * @dev: the PCI device
>>>> > > > + *
>>>> > > > + * Returns true if device is an AMD/ATI display adapter,
>>>> > > > + * otherwise return false.
>>>> > > > + */
>>>> > > > +
>>>> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
>>>> > > > +       return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
>>>> > > > +               (dev->vendor == PCI_VENDOR_ID_ATI ||
>>>> > > > +               dev->vendor == PCI_VENDOR_ID_AMD)); }
>>>> > > > +
>>>> > > >  /**
>>>> > > >   * pci_iov_init - initialize the IOV capability
>>>> > > >   * @dev: the PCI device
>>>> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>>>> > > >                 return -ENODEV;
>>>> > > >
>>>> > > >         pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
>>>> > > > -       if (pos)
>>>> > > > -               return sriov_init(dev, pos);
>>>> > > > -
>>>> > > > +       if (pos) {
>>>> > > > +       /*
>>>> > > > +        * If the device is an AMD graphics device and it supports
>>>> > > > +        * SR-IOV it will require a large amount of resources.
>>>> > > > +        * Before calling sriov_init() must ensure that the system
>>>> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
>>>> > > > +        * able to allocate enough resources.
>>>> > > > +        * If the VF BARs are zero then the system BIOS does not
>>>> > > > +        * support SR-IOV or it could not allocate the resources
>>>> > > > +        * and this platform will not support AMD graphics SR-IOV.
>>>> > > > +        * Therefore do not call sriov_init().
>>>> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
>>>> > > > +        * will be properly initialized to non-zero values.
>>>> > > > +        */
>>>> > > > +               if (is_amd_display_adapter(dev)) {
>>>> > > > +                       if (pci_vf_bar_valid(dev, pos))
>>>> > > > +                               return sriov_init(dev, pos);
>>>> > > > +               } else {
>>>> > > > +                       return sriov_init(dev, pos);
>>>> > > > +               }
>>>> > > > +       }
>>>> > > >         return -ENODEV;
>>>> > > >  }
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Zytaruk, Kelly]

Q29sbGlucywNCg0KT2theSwgZ29vZCB0byBrbm93Lg0KSXMgdGhlcmUgYSBjb21tb24gc29sdXRp
b24gdGhhdCBjYW4gaGFuZGxlIGFsbCBjYXNlcz8NCg0KVGhhbmtzLA0KS2VsbHkNCg0KPi0tLS0t
T3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogQ2hlbmcsIENvbGxpbnMNCj5TZW50OiBTYXR1
cmRheSwgTWF5IDIwLCAyMDE3IDY6MzggQU0NCj5UbzogWnl0YXJ1aywgS2VsbHk7IEFsZXhhbmRl
ciBEdXljazsgQWxleCBXaWxsaWFtc29uDQo+Q2M6IEJqb3JuIEhlbGdhYXM7IGxpbnV4LXBjaUB2
Z2VyLmtlcm5lbC5vcmc7IGxpbnV4LWtlcm5lbEB2Z2VyLmtlcm5lbC5vcmc7DQo+RGV1Y2hlciwg
QWxleGFuZGVyOyBZaW5naGFpIEx1DQo+U3ViamVjdDogUkU6IFtQQVRDSF0gUENJOiBNYWtlIFNS
LUlPViBjYXBhYmxlIEdQVSB3b3JraW5nIG9uIHRoZSBTUi1JT1YNCj5pbmNhcGFibGUgcGxhdGZv
cm0NCj4NCj5IaSBLZWxseSwNCj4NCj5UaGlzIGlzc3VlIGFsc28gaGFwcGVucyBpbiAibm90IFNS
LUlPViBjYXBhYmxlIiBTQklPUy4gSXQgc2VlbXMgc29tZSAibm90IFNSLUlPVg0KPmNhcGFibGUi
IFNCSU9TIHdpbGwgZGlyZWN0bHkgcmVwb3J0IGVycm9yIGluIHN5c3RlbSBCSU9TIGJvb3Qgc3Rh
Z2UgYW5kIGRvZXNuJ3QNCj5ib290IHRvIE9TLiBCdXQgb3RoZXIgIm5vdCBTUi1JT1YgY2FwYWJs
ZSIgU0JJT1Mgd291bGQgbm90IHJlcG9ydCBlcnJvciBhbmQNCj5ib290IHRvIExpbnV4Lg0KPg0K
Pi1Db2xsaW5zIENoZW5nDQo+DQo+DQo+LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj5Gcm9t
OiBaeXRhcnVrLCBLZWxseQ0KPlNlbnQ6IFNhdHVyZGF5LCBNYXkgMjAsIDIwMTcgNjoyOCBQTQ0K
PlRvOiBDaGVuZywgQ29sbGlucyA8Q29sbGlucy5DaGVuZ0BhbWQuY29tPjsgQWxleGFuZGVyIER1
eWNrDQo+PGFsZXhhbmRlci5kdXlja0BnbWFpbC5jb20+OyBBbGV4IFdpbGxpYW1zb24gPGFsZXgu
d2lsbGlhbXNvbkByZWRoYXQuY29tPg0KPkNjOiBCam9ybiBIZWxnYWFzIDxiaGVsZ2Fhc0Bnb29n
bGUuY29tPjsgbGludXgtcGNpQHZnZXIua2VybmVsLm9yZzsgbGludXgtDQo+a2VybmVsQHZnZXIu
a2VybmVsLm9yZzsgRGV1Y2hlciwgQWxleGFuZGVyIDxBbGV4YW5kZXIuRGV1Y2hlckBhbWQuY29t
PjsNCj5ZaW5naGFpIEx1IDx5aW5naGFpQGtlcm5lbC5vcmc+DQo+U3ViamVjdDogUkU6IFtQQVRD
SF0gUENJOiBNYWtlIFNSLUlPViBjYXBhYmxlIEdQVSB3b3JraW5nIG9uIHRoZSBTUi1JT1YNCj5p
bmNhcGFibGUgcGxhdGZvcm0NCj4NCj4NCj4NCj4+LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0N
Cj4+RnJvbTogQ2hlbmcsIENvbGxpbnMNCj4+U2VudDogU2F0dXJkYXksIE1heSAyMCwgMjAxNyAx
Mjo1MyBBTQ0KPj5UbzogQWxleGFuZGVyIER1eWNrOyBBbGV4IFdpbGxpYW1zb24NCj4+Q2M6IEJq
b3JuIEhlbGdhYXM7IGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc7DQo+PmxpbnV4LWtlcm5lbEB2
Z2VyLmtlcm5lbC5vcmc7IERldWNoZXIsIEFsZXhhbmRlcjsgWnl0YXJ1aywgS2VsbHk7DQo+Pllp
bmdoYWkgTHUNCj4+U3ViamVjdDogUkU6IFtQQVRDSF0gUENJOiBNYWtlIFNSLUlPViBjYXBhYmxl
IEdQVSB3b3JraW5nIG9uIHRoZSBTUi1JT1YNCj4+aW5jYXBhYmxlIHBsYXRmb3JtDQo+Pg0KPj5I
aSBBbGV4LA0KPj4NCj4+WWVzLCBJIGhvcGUga2VybmVsIGNhbiBkaXNhYmxlIFNSLUlPViBhbmQg
cmVsYXRlZCBWRiByZXNvdXJjZQ0KPj5hbGxvY2F0aW9uIGlmIHRoZSBzeXN0ZW0gQklPUyBpcyBu
b3QgU1ItSU9WIGNhcGFibGUuDQo+Pg0KPj5BZGRpbmcgdGhlIHBhcmFtZXRlciAicGNpPW5vc3Jp
b3YiIHNvdW5kcyBhIGRvYWJsZSBzb2x1dGlvbiwgYnV0IGl0DQo+PndvdWxkIG5lZWQgdXNlciB0
byBhZGQgdGhpcyBwYXJhbWV0ZXIgbWFudWFsbHksIHJpZ2h0PyBJIHRoaW5rIGFuDQo+PmF1dG9t
YXRpYyBkZXRlY3Rpb24gd291bGQgYmUgYmV0dGVyLiBNeSBwYXRjaCBpcyB0cnlpbmcgdG8gYXV0
byBkZXRlY3QgYW5kDQo+YnlwYXNzIFZGIHJlc291cmNlIGFsbG9jYXRpb24uDQo+Pg0KPj4NCj4+
LUNvbGxpbnMgQ2hlbmcNCj4+DQo+DQo+Q29sbGlucywgYmUgY2FyZWZ1bCBhYm91dCB0aGlzLiAg
SSBkb24ndCB0aGluayB0aGF0IHRoaXMgaXMgd2hhdCB3ZSB3YW50LiAgSWYgeW91IGFkZA0KPiJw
Y2k9bm9zcmlvdiIgdGhlbiB5b3UgYXJlIGdsb2JhbGx5IGRpc2FibGluZyBTUklPViBmb3IgYWxs
IGRldmljZXMuICBUaGlzIGlzIG5vdCB0aGUNCj5zb2x1dGlvbiB0aGF0IHdlIGFyZSBsb29raW5n
IGZvci4NCj5SZW1lbWJlciB0aGF0IHRoZXJlIGFyZSAzIHR5cGVzIG9mIFNCSU9TOyAibm90IFNS
LUlPViBjYXBhYmxlIiwgIlNSLUlPVg0KPmNhcGFibGUgYnV0IGRvZXMgbm90IHN1cHBvcnQgbGFy
Z2UgcmVzb3VyY2VzIiwgIkNvbXBsZXRlIFNSLUlPViBzdXBwb3J0Ii4NCj4NCj5UaGUgcHJvYmxl
bSBpcyB0aGF0IHdlIGFyZSB0cnlpbmcgdG8gZmluZCBhIGZpeCBmb3IgImJyb2tlbiIgU0JJT1Mg
dGhhdCBkb2VzDQo+c3VwcG9ydCBTUi1JT1YgYnV0IGRvZXMgbm90IHN1cHBvcnQgdGhlIGZ1bGwg
U1ItSU9WIGNhcGFiaWxpdGllcyB0aGF0IGRldmljZXMgd2l0aA0KPmxhcmdlIHJlc291cmNlcyBy
ZXF1aXJlLg0KPg0KPlRoYW5rcywNCj5LZWxseQ0KPg0KPj4NCj4+LS0tLS1PcmlnaW5hbCBNZXNz
YWdlLS0tLS0NCj4+RnJvbTogQWxleGFuZGVyIER1eWNrIFttYWlsdG86YWxleGFuZGVyLmR1eWNr
QGdtYWlsLmNvbV0NCj4+U2VudDogRnJpZGF5LCBNYXkgMTksIDIwMTcgMTE6NDQgUE0NCj4+VG86
IEFsZXggV2lsbGlhbXNvbiA8YWxleC53aWxsaWFtc29uQHJlZGhhdC5jb20+DQo+PkNjOiBDaGVu
ZywgQ29sbGlucyA8Q29sbGlucy5DaGVuZ0BhbWQuY29tPjsgQmpvcm4gSGVsZ2Fhcw0KPj48Ymhl
bGdhYXNAZ29vZ2xlLmNvbT47IGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc7IGxpbnV4LQ0KPj5r
ZXJuZWxAdmdlci5rZXJuZWwub3JnOyBEZXVjaGVyLCBBbGV4YW5kZXIgPEFsZXhhbmRlci5EZXVj
aGVyQGFtZC5jb20+Ow0KPj5aeXRhcnVrLCBLZWxseSA8S2VsbHkuWnl0YXJ1a0BhbWQuY29tPjsg
WWluZ2hhaSBMdSA8eWluZ2hhaUBrZXJuZWwub3JnPg0KPj5TdWJqZWN0OiBSZTogW1BBVENIXSBQ
Q0k6IE1ha2UgU1ItSU9WIGNhcGFibGUgR1BVIHdvcmtpbmcgb24gdGhlIFNSLUlPVg0KPj5pbmNh
cGFibGUgcGxhdGZvcm0NCj4+DQo+Pk9uIE1vbiwgTWF5IDE1LCAyMDE3IGF0IDEwOjUzIEFNLCBB
bGV4IFdpbGxpYW1zb24NCj4+PGFsZXgud2lsbGlhbXNvbkByZWRoYXQuY29tPiB3cm90ZToNCj4+
PiBPbiBNb24sIDE1IE1heSAyMDE3IDA4OjE5OjI4ICswMDAwDQo+Pj4gIkNoZW5nLCBDb2xsaW5z
IiA8Q29sbGlucy5DaGVuZ0BhbWQuY29tPiB3cm90ZToNCj4+Pg0KPj4+PiBIaSBXaWxsaWFtc29u
LA0KPj4+Pg0KPj4+PiBXZSBjYW5ub3QgYXNzdW1lIEJJT1Mgc3VwcG9ydHMgU1ItSU9WLCBhY3R1
YWxseSBvbmx5IG5ld2VyIHNlcnZlcg0KPj5tb3RoZXJib2FyZCBCSU9TIHN1cHBvcnRzIFNSLUlP
Vi4gTm9ybWFsIGRlc2t0b3AgbW90aGVyYm9hcmQgQklPUyBvcg0KPj5vbGRlciBzZXJ2ZXIgbW90
aGVyYm9hcmQgQklPUyBkb2Vzbid0IHN1cHBvcnQgU1ItSU9WLiBUaGlzIGlzc3VlIHdvdWxkDQo+
PmhhcHBlbiBpZiBhbiB1c2VyIHBsdWdzIG91ciBBTUQgU1ItSU9WIGNhcGFibGUgR1BVIGNhcmQg
dG8gYSBub3JtYWwgZGVza3RvcA0KPm1vdGhlcmJvYXJkLg0KPj4+DQo+Pj4gU2VydmVycyBzaG91
bGQgYmUgc3VwcG9ydGluZyBTUi1JT1YgZm9yIGEgbG9uZyB0aW1lIG5vdy4gIFdoYXQgcmVhbGx5
DQo+Pj4gaXMgdGhlcmUgdG8gYSBCSU9TIHN1cHBvcnRpbmcgU1ItSU9WIGFueXdheSwgaXQncyBz
aW1wbHkgcmVzZXJ2aW5nDQo+Pj4gc3VmZmljaWVudCBidXMgbnVtYmVyIGFuZCBNTUlPIHJlc291
cmNlcyBzdWNoIHRoYXQgd2UgY2FuIGVuYWJsZSB0aGUNCj4+PiBWRnMuICBUaGlzIHByb2Nlc3Mg
aXNuJ3QgZXhjbHVzaXZlbHkgcmVzZXJ2ZWQgZm9yIHRoZSBCSU9TLiAgU29tZQ0KPj4+IHBsYXRm
b3JtcyBtYXkgY2hvb3NlIHRvIG9ubHkgaW5pdGlhbGl6ZSBib290IGRldmljZXMsIGxlYXZpbmcg
dGhlDQo+Pj4gcmVzdCBmb3IgdGhlIE9TIHRvIHByb2dyYW0uICBUaGUgaW5pdGlhbCBwcm9wb3Nh
bCBoZXJlIHRvIGRpc2FibGUNCj4+PiBTUi1JT1YgaWYgbm90IHByb2dyYW1tZWQgYXQgT1MgaGFu
ZC1vZmYgZGlzYWJsZXMgZXZlbiB0aGUgcG9zc2liaWxpdHkNCj4+PiBvZiB0aGUgT1MgcmVhbGxv
Y2F0aW5nIHJlc291cmNlcyBmb3IgdGhpcyBkZXZpY2UuDQo+Pg0KPj5UaGVyZSBhcmUgZGlmZmVy
ZW5jZXMgYmV0d2VlbiBzdXBwb3J0aW5nIFNSLUlPViBhbmQgc3VwcG9ydGluZyBTUi1JT1YNCj4+
b24gZGV2aWNlcyB3aXRoIG1hc3NpdmUgcmVzb3VyY2VzLiBJIGtub3cgSSBoYXZlIHNlZW4gTklD
cyB0aGF0IHdpbGwNCj4+a2VlcCBhIHN5c3RlbSBmcm9tIGNvbXBsZXRpbmcgUE9TVCBpZiBTUi1J
T1YgaXMgZW5hYmxlZCwgYW5kIE1NSU8NCj4+YmV5b25kIDRHIGlzIG5vdC4gTXkgZ3Vlc3Mgd291
bGQgYmUgdGhhdCB0aGUgaXNzdWVzIGJlaW5nIHNlZW4gYXJlDQo+PnByb2JhYmx5IHRoYXQgdGhl
eSBkaXNhYmxlIFNSLUlPViBpbiB0aGUgQklPUyBpbiBzdWNoIGEgc2V0dXAgYW5kIGVuZA0KPj51
cCBydW5uaW5nIGludG8gaXNzdWVzIHdoZW4gdGhleSB0cnkgdG8gYm9vdCBpbnRvIHRoZSBMaW51
eCBrZXJuZWwgYXMNCj4+aXQgZ29lcyB0aHJvdWdoIGFuZCB0cmllcyB0byBhbGxvY2F0ZSByZXNv
dXJjZXMgZm9yIFNSLUlPViBldmVuIHRob3VnaCBpdCB3YXMNCj5kaXNhYmxlZCBpbiB0aGUgQklP
Uy4NCj4+DQo+Pkl0IG1pZ2h0IG1ha2Ugc2Vuc2UgdG8gYWRkIGEga2VybmVsIHBhcmFtZXRlciBz
b21ldGhpbmcgbGlrZSBhICJwY2k9bm9zcmlvdiINCj4+dGhhdCB3b3VsZCBhbGxvdyBmb3IgZGlz
YWJsaW5nIFNSLUlPViBhbmQgcmVsYXRlZCByZXNvdXJjZSBhbGxvY2F0aW9uDQo+PmlmIHRoYXQg
aXMgd2hhdCB3ZSBhcmUgdGFsa2luZyBhYm91dC4gVGhhdCB3YXkgeW91IGNvdWxkIHBsdWcgaW4g
dGhlc2UNCj4+dHlwZXMgb2YgZGV2aWNlcyBpbnRvIGEgc3lzdGVtIHdpdGggYSBsZWdhY3kgYmlv
cyBvciB0aGF0IGRvZXNuJ3Qgd2FuDQo+PnRvIGFsbG9jYXRlIGFkZHJlc3NlcyBhYm92ZSAzMmIg
Zm9yIE1NSU8sIGFuZCB0aGlzIHBhcmFtZXRlciB3b3VsZCBiZQ0KPj5hbGwgdGhhdCBpcyBuZWVk
ZWQgdG8gZGlzYWJsZSBTUi1JT1Ygc28geW91IGNvdWxkIHBsdWcgaW4gYSBOSUMgdGhhdCBoYXMg
U1ItSU9WDQo+YXNzb2NpYXRlZCB3aXRoIGl0Lg0KPj4NCj4+Pj4gSSBhZ3JlZSB0aGF0IGZhaWx1
cmUgdG8gYWxsb2NhdGUgVkYgcmVzb3VyY2VzIHNob3VsZCBsZWF2ZSB0aGUNCj4+Pj4gZGV2aWNl
IGluIG5vDQo+PndvcnNlIGNvbmRpdGlvbiB0aGFuIGJlZm9yZSBpdCB0cmllZC4gSSBob3BlIGtl
cm5lbCBjb3VsZCBhbGxvY2F0ZSBQRg0KPj5kZXZpY2UgcmVzb3VyY2UgYmVmb3JlIGFsbG9jYXRp
bmcgVkYgZGV2aWNlIHJlc291cmNlLCBhbmQga2VlcCBQRg0KPj5kZXZpY2UgcmVzb3VyY2UgdmFs
aWQgYW5kIGZ1bmN0aW9uYWwgaWYgZmFpbGVkIHRvIGFsbG9jYXRlIFZGIGRldmljZSByZXNvdXJj
ZS4NCj4+Pj4NCj4+Pj4gSSB3aWxsIHNlbmQgb3V0IGRtZXNnIGxvZyBsc3BjaSBpbmZvIHRvbW9y
cm93LiBUaGFua3MuDQo+Pj4NCj4+PiBUaGFua3MsDQo+Pj4gQWxleA0KPj4+DQo+Pj4+IC0tLS0t
T3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+Pj4+IEZyb206IEFsZXggV2lsbGlhbXNvbiBbbWFpbHRv
OmFsZXgud2lsbGlhbXNvbkByZWRoYXQuY29tXQ0KPj4+PiBTZW50OiBGcmlkYXksIE1heSAxMiwg
MjAxNyAxMDo0MyBQTQ0KPj4+PiBUbzogQ2hlbmcsIENvbGxpbnMgPENvbGxpbnMuQ2hlbmdAYW1k
LmNvbT4NCj4+Pj4gQ2M6IEJqb3JuIEhlbGdhYXMgPGJoZWxnYWFzQGdvb2dsZS5jb20+OyBsaW51
eC1wY2lAdmdlci5rZXJuZWwub3JnOw0KPj4+PiBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3Jn
OyBEZXVjaGVyLCBBbGV4YW5kZXINCj4+Pj4gPEFsZXhhbmRlci5EZXVjaGVyQGFtZC5jb20+OyBa
eXRhcnVrLCBLZWxseSA8S2VsbHkuWnl0YXJ1a0BhbWQuY29tPjsNCj4+Pj4gWWluZ2hhaSBMdSA8
eWluZ2hhaUBrZXJuZWwub3JnPg0KPj4+PiBTdWJqZWN0OiBSZTogW1BBVENIXSBQQ0k6IE1ha2Ug
U1ItSU9WIGNhcGFibGUgR1BVIHdvcmtpbmcgb24gdGhlDQo+Pj4+IFNSLUlPViBpbmNhcGFibGUg
cGxhdGZvcm0NCj4+Pj4NCj4+Pj4gT24gRnJpLCAxMiBNYXkgMjAxNyAwNDo1MTo0MyArMDAwMA0K
Pj4+PiAiQ2hlbmcsIENvbGxpbnMiIDxDb2xsaW5zLkNoZW5nQGFtZC5jb20+IHdyb3RlOg0KPj4+
Pg0KPj4+PiA+IEhpIFdpbGxpYW1zb24sDQo+Pj4+ID4NCj4+Pj4gPiBJIHZlcmlmaWVkIHRoZSBw
YXRjaCBpcyB3b3JraW5nIGZvciBib3RoIEFNRCBTUi1JT1YgR1BVIGFuZCBJbnRlbA0KPj4+PiA+
IFNSLUlPVg0KPj5OSUMuIEkgZG9uJ3QgdGhpbmsgaXQgaXMgcmVkdW5kYW50IHRvIGNoZWNrIHRo
ZSBWRiBCQVIgdmFsaWQgYmVmb3JlDQo+PmNhbGwgc3Jpb3ZfaW5pdCgpLCBpdCBpcyBzYWZlIGFu
ZCBzYXZpbmcgYm9vdCB0aW1lLCBhbHNvIHRoZXJlIGlzIG5vIGENCj4+YmV0dGVyIG1ldGhvZCB0
byBrbm93IGlmIHN5c3RlbSBCSU9TIGhhcyBjb3JyZWN0bHkgaW5pdGlhbGl6ZWQgdGhlIFNSLUlP
Vg0KPmNhcGFiaWxpdHkgb3Igbm90Lg0KPj4+Pg0KPj4+PiBJdCBhbHNvIG1hc2tzIGFuIHVuZGVy
bHlpbmcgYnVnIGFuZCBjcmVhdGVzIGEgbWFpbnRlbmFuY2UgaXNzdWUgdGhhdA0KPj4+PiB3ZSB3
b24ndA0KPj5rbm93IHdoZW4gaXQncyBzYWZlIHRvIHJlbW92ZSB0aGlzIHdvcmthcm91bmQuICBJ
IGRvbid0IHRoaW5rIGZhc3Rlcg0KPj5ib290IGlzIHZhbGlkIHJhdGlvbmFsZSwgaW4gb25lIGNh
c2UgU1ItSU9WIGlzIGNvbXBsZXRlbHkgZGlzYWJsZWQsIHRoZQ0KPj5vdGhlciB3ZSBhdHRlbXB0
IHRvIGFsbG9jYXRlIHRoZSByZXNvdXJjZXMgdGhlIEJJT1MgZmFpbGVkIHRvIHByb3ZpZGUuDQo+
PkkgZXhwZWN0IHRoaXMgaXMgYWxzbyBhIGNvcm5lciBjYXNlLCB0aGUgQklPUyBzaG91bGQgdHlw
aWNhbGx5IHN1cHBvcnQNCj4+U1ItSU9WLCB0aGVyZWZvcmUgdGhpcyBzaXR1YXRpb24gc2hvdWxk
IGJlIGFuIGV4Y2VwdGlvbi4NCj4+Pj4NCj4+Pj4gPiBJIGRpZCBub3QgdHJ5IHRvIGZpeCB0aGUg
aXNzdWUgZnJvbSB0aGUga2VybmVsIHJlc291cmNlIGFsbG9jYXRpb24NCj4+Pj4gPiBwZXJzcGVj
dGl2ZSwNCj4+aXQgaXMgYmVjYXVzZToNCj4+Pj4gPiAgICAgMS4gSSBhbSBub3QgdmVyeSBmYW1p
bGlhciB3aXRoIHRoZSBQQ0kgcmVzb3VyY2UgYWxsb2NhdGlvbiBzY2hlbWUgaW4NCj5rZXJuZWwu
DQo+PkZvciBleGFtcGxlLCBpbiBzcmlvdl9pbml0KCksIGtlcm5lbCB3aWxsIHJlLWFzc2lnbiB0
aGUgUENJIHJlc291cmNlDQo+PmZvciBib3RoIFZGIGFuZCBQRi4gSSBkb24ndCB1bmRlcnN0YW5k
IHdoeSBrZXJuZWwgYWxsb2NhdGVzIHJlc291cmNlDQo+PmZvciBWRiBmaXJzdGx5LCB0aGVuIFBG
LiBJZiBpdCBpcyBQRiBmaXJzdGx5LCB0aGVuIHRoaXMgaXNzdWUgY291bGQgYmUgYXZvaWRlZC4N
Cj4+Pj4gPiAgICAgMi4gSSBhbSBub3Qgc3VyZSBpZiBrZXJuZWwgaGFzIGVycm9yIGhhbmRsZXIg
aWYgUENJIHJlc291cmNlIGFsbG9jYXRpb24NCj5mYWlsZWQuDQo+PkluIHRoaXMgY2FzZSwga2Vy
bmVsIGNhbm5vdCBhbGxvY2F0ZSBlbm91Z2ggcmVzb3VyY2UgdG8gUEYuIEl0IHNob3VsZA0KPj50
cmlnZ2VyIHNvbWUgZXJyb3IgaGFuZGxlciB0byBlaXRoZXIganVzdCBrZWVwIG9yaWdpbmFsIEJB
UiB2YWx1ZXMgc2V0DQo+PmJ5IHN5c3RlbSBCSU9TLCBvciBkaXNhYmxlIHRoaXMgZGV2aWNlIGFu
ZCBsb2cgZXJyb3JzLg0KPj4+Pg0KPj4+PiBJIHRoaW5rIHRoZXNlIGFyZSB0aGUgaXNzdWVzIHdl
IHNob3VsZCBiZSB0cnlpbmcgdG8gc29sdmUgYW5kIEknbQ0KPj4+PiBzdXJlIGZvbGtzIG9uIHRo
ZSBsaW51eC1wY2kgbGlzdCBjYW4gaGVscCB1cyBpZGVudGlmeSB0aGUgYnVnLg0KPj4+PiBNaW5p
bWFsbHksIGZhaWx1cmUgdG8gYWxsb2NhdGUgVkYgcmVzb3VyY2VzIHNob3VsZCBsZWF2ZSB0aGUg
ZGV2aWNlDQo+Pj4+IGluIG5vIHdvcnNlIGNvbmRpdGlvbiB0aGFuIGJlZm9yZSBpdCB0cmllZC4g
IFBlcmhhcHMgeW91IGNvdWxkIHBvc3QNCj4+Pj4gbW9yZSBkZXRhaWxzIGFib3V0IHRoZSBpc3N1
ZSwgYm9vdCB3aXRoIHBjaT1lYXJseWR1bXAsIHBvc3QgZG1lc2cgb2YNCj4+Pj4gYSBib290IHdo
ZXJlIHRoZSBQRiByZXNvdXJjZXMgYXJlIGluY29ycmVjdGx5IHJlLWFsbG9jYXRlZCwgYW5kDQo+
Pj4+IGluY2x1ZGUgbHNwY2kgLXZ2diBmb3IgdGhlIFNSLUlPViBkZXZpY2UuICBBbHNvLCBwbGVh
c2UgdGVzdCB3aXRoDQo+Pj4+IHRoZSBsYXRlc3QgdXBzdHJlYW0ga2VybmVsLCB1cHN0cmVhbSBv
bmx5IHBhdGNoZXMgb2xkIGtlcm5lbHMNCj4+Pj4gdGhyb3VnaCBzdGFibGUgYmFja3BvcnRzIG9m
IGNvbW1pdHMgdG8gdGhlIGxhdGVzdCBrZXJuZWwuICBBZGRpbmcNCj4+Pj4gWWluZ2hhaSBhcyBh
IHJlc291cmNlIGFsbG9jYXRpb24gZXhwZXJ0LiBUaGFua3MsDQo+Pj4+DQo+Pj4+IEFsZXgNCj4+
Pj4NCj4+Pj4gPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPj4+PiA+IEZyb206IEFsZXgg
V2lsbGlhbXNvbiBbbWFpbHRvOmFsZXgud2lsbGlhbXNvbkByZWRoYXQuY29tXQ0KPj4+PiA+IFNl
bnQ6IEZyaWRheSwgTWF5IDEyLCAyMDE3IDEyOjAxIFBNDQo+Pj4+ID4gVG86IENoZW5nLCBDb2xs
aW5zIDxDb2xsaW5zLkNoZW5nQGFtZC5jb20+DQo+Pj4+ID4gQ2M6IEJqb3JuIEhlbGdhYXMgPGJo
ZWxnYWFzQGdvb2dsZS5jb20+Ow0KPj4+PiA+IGxpbnV4LXBjaUB2Z2VyLmtlcm5lbC5vcmc7IGxp
bnV4LWtlcm5lbEB2Z2VyLmtlcm5lbC5vcmc7IERldWNoZXIsDQo+Pj4+ID4gQWxleGFuZGVyIDxB
bGV4YW5kZXIuRGV1Y2hlckBhbWQuY29tPjsgWnl0YXJ1aywgS2VsbHkNCj4+Pj4gPiA8S2VsbHku
Wnl0YXJ1a0BhbWQuY29tPg0KPj4+PiA+IFN1YmplY3Q6IFJlOiBbUEFUQ0hdIFBDSTogTWFrZSBT
Ui1JT1YgY2FwYWJsZSBHUFUgd29ya2luZyBvbiB0aGUNCj4+Pj4gPiBTUi1JT1YgaW5jYXBhYmxl
IHBsYXRmb3JtDQo+Pj4+ID4NCj4+Pj4gPiBPbiBGcmksIDEyIE1heSAyMDE3IDAzOjQyOjQ2ICsw
MDAwICJDaGVuZywgQ29sbGlucyINCj4+Pj4gPiA8Q29sbGlucy5DaGVuZ0BhbWQuY29tPiB3cm90
ZToNCj4+Pj4gPg0KPj4+PiA+ID4gSGkgV2lsbGlhbXNvbiwNCj4+Pj4gPiA+DQo+Pj4+ID4gPiBH
UFUgY2FyZCBuZWVkcyBtb3JlIEJBUiBhcGVydHVyZSByZXNvdXJjZSB0aGFuIG90aGVyIFBDSQ0K
Pj4+PiA+ID4gZGV2aWNlcy4gRm9yDQo+PmV4YW1wbGUsIEludGVsIFNSLUlPViBuZXR3b3JrIGNh
cmQgb25seSByZXF1aXJlIDUxMktCIG1lbW9yeSByZXNvdXJjZQ0KPj5mb3IgYWxsIFZGcy4gQU1E
IFNSLUlPViBHUFUgY2FyZCBuZWVkcyAyNTZNQiB4MTYgVkYgPSA0R0IgbWVtb3J5DQo+PnJlc291
cmNlIGZvciBmcmFtZSBidWZmZXIgQkFSIGFwZXJ0dXJlLg0KPj4+PiA+ID4NCj4+Pj4gPiA+IElm
IHRoZSBzeXN0ZW0gQklPUyBzdXBwb3J0cyBTUi1JT1YsIGl0IHdpbGwgcmVzZXJ2ZSBlbm91Z2gN
Cj4+Pj4gPiA+IHJlc291cmNlIGZvciBhbGwNCj4+VkYgQkFScy4gSWYgdGhlIHN5c3RlbSBCSU9T
IGRvZXNuJ3Qgc3VwcG9ydCBTUi1JT1Ygb3IgY2Fubm90IGFsbG9jYXRlDQo+PnRoZSBlbm91Z2gg
cmVzb3VyY2UgZm9yIFZGIEJBUnMsIG9ubHkgUEYgQkFSIHdpbGwgYmUgYXNzaWduZWQgYW5kIFZG
DQo+PkJBUnMgYXJlIGVtcHR5LiBUaGVuIHN5c3RlbSBib290cyB0byBMaW51eCBrZXJuZWwgYW5k
IGtlcm5lbCBkb2Vzbid0DQo+PmNoZWNrIGlmIHRoZSBWRiBCQVJzIGFyZSBlbXB0eSBvciB2YWxp
ZC4gS2VybmVsIHdpbGwgcmUtYXNzaWduIHRoZSBCQVINCj4+cmVzb3VyY2VzIGZvciBQRiBhbmQg
YWxsIFZGcy4gVGhlIHByb2JsZW0gSSBzYXcgaXMgdGhhdCBrZXJuZWwgd2lsbA0KPj5mYWlsIHRv
IGFsbG9jYXRlIFBGIEJBUiByZXNvdXJjZSBiZWNhdXNlIHNvbWUgcmVzb3VyY2VzIGFyZSBhc3Np
Z25lZCB0bw0KPj5WRiwgdGhpcyBpcyBub3QgZXhwZWN0ZWQuIFNvIGtlcm5lbCBtaWdodCBuZWVk
IHRvIGRvIHNvbWUgY2hlY2sgYmVmb3JlDQo+PnJlLWFzc2lnbiB0aGUgUEYvVkYgcmVzb3VyY2Us
IHNvIHRoYXQgUEYgZGV2aWNlIHdpbGwgYmUgY29ycmVjdGx5IGFzc2lnbmVkIEJBUg0KPnJlc291
cmNlIGFuZCB1c2VyIGNhbiB1c2UgUEYgZGV2aWNlLg0KPj4+PiA+DQo+Pj4+ID4gU28gdGhlIHBy
b2JsZW0gaXMgdGhhdCBzb21ldGhpbmcgYmFkIGhhcHBlbnMgd2hlbiB0aGUga2VybmVsIGlzDQo+
Pj4+ID4gdHJ5aW5nIHRvIHJlYWxsb2NhdGUgcmVzb3VyY2VzIGluIG9yZGVyIHRvIGZ1bGZpbGwg
dGhlDQo+Pj4+ID4gcmVxdWlyZW1lbnRzIG9mIHRoZSBWRnMsIGxlYXZpbmcgdGhlIFBGIHJlc291
cmNlcyBpbmNvcnJlY3RseQ0KPj4+PiA+IHByb2dyYW1tZWQ/ICBXaHkgbm90IGp1c3QgZml4IHRo
YXQgYnVnIHJhdGhlciB0aGFuIGNyZWF0aW5nDQo+Pj4+ID4gc3BlY2lhbCBoYW5kbGluZyBmb3Ig
dGhpcyB2ZW5kb3IvY2xhc3Mgb2YgZGV2aWNlIHdoaWNoIGRpc2FibGVzDQo+Pj4+ID4gYW55IGF0
dGVtcHQgdG8gZml4dXAgcmVzb3VyY2VzIGZvciBTUi1JT1Y/ICBJT1csIHRoaXMgcGF0Y2gganVz
dA0KPj4+PiA+IGF2b2lkcyB0aGUgcHJvYmxlbSBmb3IgeW91ciBkZXZpY2VzIHJhdGhlciB0aGFu
IGZpeGluZyB0aGUgYnVnLg0KPj4+PiA+IEknZCBzdWdnZXN0IGZpeGluZyB0aGUgYnVnIHN1Y2gg
dGhhdCB0aGUgUEYgaXMgbGVmdCBpbiBhDQo+Pj4+ID4gZnVuY3Rpb25hbCBzdGF0ZSBpZiB0aGUg
a2VybmVsIGlzIHVuYWJsZSB0byBhbGxvY2F0ZSBzdWZmaWNpZW50DQo+Pj4+ID4gcmVzb3VyY2Vz
IGZvciB0aGUgVkZzLiAgVGhhbmtzLA0KPj4+PiA+DQo+Pj4+ID4gQWxleA0KPj4+PiA+DQo+Pj4+
ID4gPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPj4+PiA+ID4gRnJvbTogQWxleCBXaWxs
aWFtc29uIFttYWlsdG86YWxleC53aWxsaWFtc29uQHJlZGhhdC5jb21dDQo+Pj4+ID4gPiBTZW50
OiBGcmlkYXksIE1heSAxMiwgMjAxNyAxMToyMSBBTQ0KPj4+PiA+ID4gVG86IENoZW5nLCBDb2xs
aW5zIDxDb2xsaW5zLkNoZW5nQGFtZC5jb20+DQo+Pj4+ID4gPiBDYzogQmpvcm4gSGVsZ2FhcyA8
YmhlbGdhYXNAZ29vZ2xlLmNvbT47DQo+Pj4+ID4gPiBsaW51eC1wY2lAdmdlci5rZXJuZWwub3Jn
OyBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnOw0KPj4+PiA+ID4gRGV1Y2hlciwgQWxleGFu
ZGVyIDxBbGV4YW5kZXIuRGV1Y2hlckBhbWQuY29tPjsgWnl0YXJ1aywgS2VsbHkNCj4+Pj4gPiA+
IDxLZWxseS5aeXRhcnVrQGFtZC5jb20+DQo+Pj4+ID4gPiBTdWJqZWN0OiBSZTogW1BBVENIXSBQ
Q0k6IE1ha2UgU1ItSU9WIGNhcGFibGUgR1BVIHdvcmtpbmcgb24gdGhlDQo+Pj4+ID4gPiBTUi1J
T1YgaW5jYXBhYmxlIHBsYXRmb3JtDQo+Pj4+ID4gPg0KPj4+PiA+ID4gT24gRnJpLCAxMiBNYXkg
MjAxNyAwMjo1MDozMiArMDAwMCAiQ2hlbmcsIENvbGxpbnMiDQo+Pj4+ID4gPiA8Q29sbGlucy5D
aGVuZ0BhbWQuY29tPiB3cm90ZToNCj4+Pj4gPiA+DQo+Pj4+ID4gPiA+IEhpIEhlbGdhYXMsDQo+
Pj4+ID4gPiA+DQo+Pj4+ID4gPiA+IFNvbWUgQU1EIEdQVXMgaGF2ZSBoYXJkd2FyZSBzdXBwb3J0
IGZvciBncmFwaGljcyBTUi1JT1YuDQo+Pj4+ID4gPiA+IElmIHRoZSBTUi1JT1YgY2FwYWJsZSBH
UFUgaXMgcGx1Z2dlZCBpbnRvIHRoZSBTUi1JT1YgaW5jYXBhYmxlDQo+Pj4+ID4gPiA+IHBsYXRm
b3JtLiBJdCB3b3VsZCBjYXVzZSBhIHByb2JsZW0gb24gUENJIHJlc291cmNlIGFsbG9jYXRpb24N
Cj4+Pj4gPiA+ID4gaW4gY3VycmVudCBMaW51eCBrZXJuZWwuDQo+Pj4+ID4gPiA+DQo+Pj4+ID4g
PiA+IFRoZXJlZm9yZSBpbiBvcmRlciB0byBhbGxvdyB0aGUgUEYgKFBoeXNpY2FsIEZ1bmN0aW9u
KSBkZXZpY2UNCj4+Pj4gPiA+ID4gb2YgU1ItSU9WIGNhcGFibGUgR1BVIHRvIHdvcmsgb24gdGhl
IFNSLUlPViBpbmNhcGFibGUNCj4+Pj4gPiA+ID4gcGxhdGZvcm0sIGl0IGlzIHJlcXVpcmVkIHRv
IHZlcmlmeSBjb25kaXRpb25zIGZvciBpbml0aWFsaXppbmcNCj4+Pj4gPiA+ID4gQkFSIHJlc291
cmNlcyBvbiBBTUQgU1ItSU9WIGNhcGFibGUgR1BVcy4NCj4+Pj4gPiA+ID4NCj4+Pj4gPiA+ID4g
SWYgdGhlIGRldmljZSBpcyBhbiBBTUQgZ3JhcGhpY3MgZGV2aWNlIGFuZCBpdCBzdXBwb3J0cyBT
Ui1JT1YNCj4+Pj4gPiA+ID4gaXQgd2lsbCByZXF1aXJlIGEgbGFyZ2UgYW1vdW50IG9mIHJlc291
cmNlcy4NCj4+Pj4gPiA+ID4gQmVmb3JlIGNhbGxpbmcgc3Jpb3ZfaW5pdCgpIG11c3QgZW5zdXJl
IHRoYXQgdGhlIHN5c3RlbSBCSU9TDQo+Pj4+ID4gPiA+IGFsc28gc3VwcG9ydHMgU1ItSU9WIGFu
ZCB0aGF0IHN5c3RlbSBCSU9TIGhhcyBiZWVuIGFibGUgdG8NCj4+Pj4gPiA+ID4gYWxsb2NhdGUg
ZW5vdWdoIHJlc291cmNlcy4NCj4+Pj4gPiA+ID4gSWYgdGhlIFZGIEJBUnMgYXJlIHplcm8gdGhl
biB0aGUgc3lzdGVtIEJJT1MgZG9lcyBub3Qgc3VwcG9ydA0KPj4+PiA+ID4gPiBTUi1JT1Ygb3Ig
aXQgY291bGQgbm90IGFsbG9jYXRlIHRoZSByZXNvdXJjZXMgYW5kIHRoaXMNCj4+Pj4gPiA+ID4g
cGxhdGZvcm0gd2lsbCBub3Qgc3VwcG9ydCBBTUQgZ3JhcGhpY3MgU1ItSU9WLg0KPj4+PiA+ID4g
PiBUaGVyZWZvcmUgZG8gbm90IGNhbGwgc3Jpb3ZfaW5pdCgpLg0KPj4+PiA+ID4gPiBJZiB0aGUg
c3lzdGVtIEJJT1MgZG9lcyBzdXBwb3J0IFNSLUlPViB0aGVuIHRoZSBWRiBCQVJzIHdpbGwNCj4+
Pj4gPiA+ID4gYmUgcHJvcGVybHkgaW5pdGlhbGl6ZWQgdG8gbm9uLXplcm8gdmFsdWVzLg0KPj4+
PiA+ID4gPg0KPj4+PiA+ID4gPiBCZWxvdyBpcyB0aGUgcGF0Y2ggYWdhaW5zdCB0byBLZXJuZWwg
NC44ICYgNC45LiBQbGVhc2UgcmV2aWV3Lg0KPj4+PiA+ID4gPg0KPj4+PiA+ID4gPiBJIGNoZWNr
ZWQgdGhlIGRyaXZlcnMvcGNpL3F1aXJrcy5jLCBpdCBsb29rcyB0aGUNCj4+Pj4gPiA+ID4gd29y
a2Fyb3VuZHMvZml4ZXMgaW4gcXVpcmtzLmMgYXJlIGZvciBzcGVjaWZpYyBkZXZpY2VzIGFuZCBv
bmUNCj4+Pj4gPiA+ID4gb3IgbW9yZSBkZXZpY2UgSUQgYXJlIGRlZmluZWQgZm9yIHRoZSBzcGVj
aWZpYyBkZXZpY2VzLg0KPj4+PiA+ID4gPiBIb3dldmVyIG15IHBhdGNoIGlzIGZvciBhbGwgQU1E
IFNSLUlPViBjYXBhYmxlIEdQVXMsIHRoYXQNCj4+Pj4gPiA+ID4gaW5jbHVkZXMgYWxsIGV4aXN0
aW5nDQo+PmFuZCBmdXR1cmUgQU1EIHNlcnZlciBHUFVzLg0KPj4+PiA+ID4gPiBTbyBpdCBkb2Vz
bid0IHNlZW0gbGlrZSBhIGdvb2QgZml0IHRvIHB1dCB0aGUgZml4IGluIHF1aXJrcy5jLg0KPj4+
PiA+ID4NCj4+Pj4gPiA+DQo+Pj4+ID4gPiBXaHkgaXMgYW4gQU1EIGdyYXBoaWNzIGNhcmQgdW5p
cXVlIGhlcmU/ICBEb2Vzbid0IHNyaW92X2luaXQoKQ0KPj4+PiA+ID4gYWx3YXlzIG5lZWQgdG8g
YmUgYWJsZSB0byBkZWFsIHdpdGggZGV2aWNlcyBvZiBhbnkgdHlwZSB3aGVyZQ0KPj4+PiA+ID4g
dGhlIEJJT1MgaGFzbid0IGluaXRpYWxpemVkIHRoZSBTUi1JT1YgY2FwYWJpbGl0eT8gIFNvbWUg
U1ItSU9WDQo+Pj4+ID4gPiBkZXZpY2VzIGNhbiBmaXQgdGhlaXIgVkZzIHdpdGhpbiBhIG1pbmlt
dW0gYnJpZGdlIGFwZXJ0dXJlLCBtb3N0DQo+Pj4+ID4gPiBjYW5ub3QuICBJIGRvbid0IHVuZGVy
c3RhbmQgd2h5IHRoZSBWRiByZXNvdXJjZSByZXF1aXJlbWVudHMNCj4+Pj4gPiA+IGJlaW5nIGV4
Y2VwdGlvbmFsbHkgbGFyZ2UgZGljdGF0ZXMgdGhhdCB0aGV5IHJlY2VpdmUgc3BlY2lhbCBoYW5k
bGluZy4NCj4+Pj4gPiA+IFRoYW5rcywNCj4+Pj4gPiA+DQo+Pj4+ID4gPiBBbGV4DQo+Pj4+ID4g
Pg0KPj4+PiA+ID4gPiBTaWduZWQtb2ZmLWJ5OiBDb2xsaW5zIENoZW5nIDxjb2xsaW5zLmNoZW5n
QGFtZC5jb20+DQo+Pj4+ID4gPiA+IC0tLQ0KPj4+PiA+ID4gPiAgZHJpdmVycy9wY2kvaW92LmMg
fCA2Mw0KPj4+PiA+ID4gPiArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysr
KysrKysrKysrKysrLS0tDQo+Pj4+ID4gPiA+ICAxIGZpbGUgY2hhbmdlZCwgNjAgaW5zZXJ0aW9u
cygrKSwgMyBkZWxldGlvbnMoLSkNCj4+Pj4gPiA+ID4NCj4+Pj4gPiA+ID4gZGlmZiAtLWdpdCBh
L2RyaXZlcnMvcGNpL2lvdi5jIGIvZHJpdmVycy9wY2kvaW92LmMgaW5kZXgNCj4+Pj4gPiA+ID4g
ZTMwZjA1Yy4uZTRmMTQwNSAxMDA2NDQNCj4+Pj4gPiA+ID4gLS0tIGEvZHJpdmVycy9wY2kvaW92
LmMNCj4+Pj4gPiA+ID4gKysrIGIvZHJpdmVycy9wY2kvaW92LmMNCj4+Pj4gPiA+ID4gQEAgLTUy
Myw2ICs1MjMsNDUgQEAgc3RhdGljIHZvaWQgc3Jpb3ZfcmVzdG9yZV9zdGF0ZShzdHJ1Y3QNCj4+
Pj4gPiA+ID4gcGNpX2Rldg0KPj4qZGV2KQ0KPj4+PiA+ID4gPiAgICAgICAgICAgICAgICAgbXNs
ZWVwKDEwMCk7DQo+Pj4+ID4gPiA+ICB9DQo+Pj4+ID4gPiA+DQo+Pj4+ID4gPiA+ICsvKg0KPj4+
PiA+ID4gPiArICogcGNpX3ZmX2Jhcl92YWxpZCAtIGNoZWNrIGlmIFZGIEJBUnMgaGF2ZSByZXNv
dXJjZQ0KPj4+PiA+ID4gPiArYWxsb2NhdGVkDQo+Pj4+ID4gPiA+ICsgKiBAZGV2OiB0aGUgUENJ
IGRldmljZQ0KPj4+PiA+ID4gPiArICogQHBvczogcmVnaXN0ZXIgb2Zmc2V0IG9mIFNSLUlPViBj
YXBhYmlsaXR5IGluIFBDSSBjb25maWcNCj4+Pj4gPiA+ID4gK3NwYWNlDQo+Pj4+ID4gPiA+ICsg
KiBSZXR1cm5zIHRydWUgYW55IFZGIEJBUiBoYXMgcmVzb3VyY2UgYWxsb2NhdGVkLCBmYWxzZQ0K
Pj4+PiA+ID4gPiArICogaWYgYWxsIFZGIEJBUnMgYXJlIGVtcHR5Lg0KPj4+PiA+ID4gPiArICov
DQo+Pj4+ID4gPiA+ICtzdGF0aWMgYm9vbCBwY2lfdmZfYmFyX3ZhbGlkKHN0cnVjdCBwY2lfZGV2
ICpkZXYsIGludCBwb3MpIHsNCj4+Pj4gPiA+ID4gKyAgICAgICBpbnQgaTsNCj4+Pj4gPiA+ID4g
KyAgICAgICB1MzIgYmFyX3ZhbHVlOw0KPj4+PiA+ID4gPiArICAgICAgIHUzMiBiYXJfc2l6ZV9t
YXNrID0gfihQQ0lfQkFTRV9BRERSRVNTX1NQQUNFIHwNCj4+Pj4gPiA+ID4gKyAgICAgICAgICAg
ICAgICAgICAgICAgUENJX0JBU0VfQUREUkVTU19NRU1fVFlQRV82NCB8DQo+Pj4+ID4gPiA+ICsg
ICAgICAgICAgICAgICAgICAgICAgIFBDSV9CQVNFX0FERFJFU1NfTUVNX1BSRUZFVENIKTsNCj4+
Pj4gPiA+ID4gKw0KPj4+PiA+ID4gPiArICAgICAgIGZvciAoaSA9IDA7IGkgPCBQQ0lfU1JJT1Zf
TlVNX0JBUlM7IGkrKykgew0KPj4+PiA+ID4gPiArICAgICAgICAgICAgICAgcGNpX3JlYWRfY29u
ZmlnX2R3b3JkKGRldiwgcG9zICsgUENJX1NSSU9WX0JBUg0KPj4+PiA+ID4gPiArICsgaSAqIDQs
DQo+PiZiYXJfdmFsdWUpOw0KPj4+PiA+ID4gPiArICAgICAgICAgICAgICAgaWYgKGJhcl92YWx1
ZSAmIGJhcl9zaXplX21hc2spDQo+Pj4+ID4gPiA+ICsgICAgICAgICAgICAgICAgICAgICAgIHJl
dHVybiB0cnVlOw0KPj4+PiA+ID4gPiArICAgICAgIH0NCj4+Pj4gPiA+ID4gKw0KPj4+PiA+ID4g
PiArICAgICAgIHJldHVybiBmYWxzZTsNCj4+Pj4gPiA+ID4gK30NCj4+Pj4gPiA+ID4gKw0KPj4+
PiA+ID4gPiArLyoNCj4+Pj4gPiA+ID4gKyAqIGlzX2FtZF9kaXNwbGF5X2FkYXB0ZXIgLSBjaGVj
ayBpZiBpdCBpcyBhbiBBTUQvQVRJIEdQVQ0KPj4+PiA+ID4gPiArZGV2aWNlDQo+Pj4+ID4gPiA+
ICsgKiBAZGV2OiB0aGUgUENJIGRldmljZQ0KPj4+PiA+ID4gPiArICoNCj4+Pj4gPiA+ID4gKyAq
IFJldHVybnMgdHJ1ZSBpZiBkZXZpY2UgaXMgYW4gQU1EL0FUSSBkaXNwbGF5IGFkYXB0ZXIsDQo+
Pj4+ID4gPiA+ICsgKiBvdGhlcndpc2UgcmV0dXJuIGZhbHNlLg0KPj4+PiA+ID4gPiArICovDQo+
Pj4+ID4gPiA+ICsNCj4+Pj4gPiA+ID4gK3N0YXRpYyBib29sIGlzX2FtZF9kaXNwbGF5X2FkYXB0
ZXIoc3RydWN0IHBjaV9kZXYgKmRldikgew0KPj4+PiA+ID4gPiArICAgICAgIHJldHVybiAoKChk
ZXYtPmNsYXNzID4+IDE2KSA9PSBQQ0lfQkFTRV9DTEFTU19ESVNQTEFZKSAmJg0KPj4+PiA+ID4g
PiArICAgICAgICAgICAgICAgKGRldi0+dmVuZG9yID09IFBDSV9WRU5ET1JfSURfQVRJIHx8DQo+
Pj4+ID4gPiA+ICsgICAgICAgICAgICAgICBkZXYtPnZlbmRvciA9PSBQQ0lfVkVORE9SX0lEX0FN
RCkpOyB9DQo+Pj4+ID4gPiA+ICsNCj4+Pj4gPiA+ID4gIC8qKg0KPj4+PiA+ID4gPiAgICogcGNp
X2lvdl9pbml0IC0gaW5pdGlhbGl6ZSB0aGUgSU9WIGNhcGFiaWxpdHkNCj4+Pj4gPiA+ID4gICAq
IEBkZXY6IHRoZSBQQ0kgZGV2aWNlDQo+Pj4+ID4gPiA+IEBAIC01MzcsOSArNTc2LDI3IEBAIGlu
dCBwY2lfaW92X2luaXQoc3RydWN0IHBjaV9kZXYgKmRldikNCj4+Pj4gPiA+ID4gICAgICAgICAg
ICAgICAgIHJldHVybiAtRU5PREVWOw0KPj4+PiA+ID4gPg0KPj4+PiA+ID4gPiAgICAgICAgIHBv
cyA9IHBjaV9maW5kX2V4dF9jYXBhYmlsaXR5KGRldiwgUENJX0VYVF9DQVBfSURfU1JJT1YpOw0K
Pj4+PiA+ID4gPiAtICAgICAgIGlmIChwb3MpDQo+Pj4+ID4gPiA+IC0gICAgICAgICAgICAgICBy
ZXR1cm4gc3Jpb3ZfaW5pdChkZXYsIHBvcyk7DQo+Pj4+ID4gPiA+IC0NCj4+Pj4gPiA+ID4gKyAg
ICAgICBpZiAocG9zKSB7DQo+Pj4+ID4gPiA+ICsgICAgICAgLyoNCj4+Pj4gPiA+ID4gKyAgICAg
ICAgKiBJZiB0aGUgZGV2aWNlIGlzIGFuIEFNRCBncmFwaGljcyBkZXZpY2UgYW5kIGl0IHN1cHBv
cnRzDQo+Pj4+ID4gPiA+ICsgICAgICAgICogU1ItSU9WIGl0IHdpbGwgcmVxdWlyZSBhIGxhcmdl
IGFtb3VudCBvZiByZXNvdXJjZXMuDQo+Pj4+ID4gPiA+ICsgICAgICAgICogQmVmb3JlIGNhbGxp
bmcgc3Jpb3ZfaW5pdCgpIG11c3QgZW5zdXJlIHRoYXQgdGhlIHN5c3RlbQ0KPj4+PiA+ID4gPiAr
ICAgICAgICAqIEJJT1MgYWxzbyBzdXBwb3J0cyBTUi1JT1YgYW5kIHRoYXQgc3lzdGVtIEJJT1Mg
aGFzIGJlZW4NCj4+Pj4gPiA+ID4gKyAgICAgICAgKiBhYmxlIHRvIGFsbG9jYXRlIGVub3VnaCBy
ZXNvdXJjZXMuDQo+Pj4+ID4gPiA+ICsgICAgICAgICogSWYgdGhlIFZGIEJBUnMgYXJlIHplcm8g
dGhlbiB0aGUgc3lzdGVtIEJJT1MgZG9lcyBub3QNCj4+Pj4gPiA+ID4gKyAgICAgICAgKiBzdXBw
b3J0IFNSLUlPViBvciBpdCBjb3VsZCBub3QgYWxsb2NhdGUgdGhlIHJlc291cmNlcw0KPj4+PiA+
ID4gPiArICAgICAgICAqIGFuZCB0aGlzIHBsYXRmb3JtIHdpbGwgbm90IHN1cHBvcnQgQU1EIGdy
YXBoaWNzIFNSLUlPVi4NCj4+Pj4gPiA+ID4gKyAgICAgICAgKiBUaGVyZWZvcmUgZG8gbm90IGNh
bGwgc3Jpb3ZfaW5pdCgpLg0KPj4+PiA+ID4gPiArICAgICAgICAqIElmIHRoZSBzeXN0ZW0gQklP
UyBkb2VzIHN1cHBvcnQgU1ItSU9WIHRoZW4gdGhlIFZGIEJBUnMNCj4+Pj4gPiA+ID4gKyAgICAg
ICAgKiB3aWxsIGJlIHByb3Blcmx5IGluaXRpYWxpemVkIHRvIG5vbi16ZXJvIHZhbHVlcy4NCj4+
Pj4gPiA+ID4gKyAgICAgICAgKi8NCj4+Pj4gPiA+ID4gKyAgICAgICAgICAgICAgIGlmIChpc19h
bWRfZGlzcGxheV9hZGFwdGVyKGRldikpIHsNCj4+Pj4gPiA+ID4gKyAgICAgICAgICAgICAgICAg
ICAgICAgaWYgKHBjaV92Zl9iYXJfdmFsaWQoZGV2LCBwb3MpKQ0KPj4+PiA+ID4gPiArICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVybiBzcmlvdl9pbml0KGRldiwgcG9zKTsNCj4+
Pj4gPiA+ID4gKyAgICAgICAgICAgICAgIH0gZWxzZSB7DQo+Pj4+ID4gPiA+ICsgICAgICAgICAg
ICAgICAgICAgICAgIHJldHVybiBzcmlvdl9pbml0KGRldiwgcG9zKTsNCj4+Pj4gPiA+ID4gKyAg
ICAgICAgICAgICAgIH0NCj4+Pj4gPiA+ID4gKyAgICAgICB9DQo+Pj4+ID4gPiA+ICAgICAgICAg
cmV0dXJuIC1FTk9ERVY7DQo+Pj4+ID4gPiA+ICB9DQo+Pj4+ID4gPiA+DQo+Pj4+ID4gPg0KPj4+
PiA+DQo+Pj4+DQo+Pj4NCg==

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alexander Duyck]

I'd say the common solution is probably the parameter that allows the
user to disable SR-IOV in the kernel on boot.

The problem with trying to do this automatically is that there are too
many scenarios to know what it was that the BIOS was trying to do.

Another alternative would be to look at providing a means of changing
how the SR-IOV code tries to fix broken setups. Right now it defaults
to trying to allocate the data as it assumes it is going to enable
SR-IOV on every device that has SR-IOV support. An alternative might
be to make the kernel option support multiple options. You could have
it do nosriov as one option, and another option that only enables
SR-IOV on devices that are fully configured and disabled it otherwise,
and then our current default option which is to try enabling SR-IOV on
any device that could support it. Then you could probably also make
the default something you could have as a kernel configuration options
so you could build a kernel that defaults to the middle option that
leaves SR-IOV devices correctly configured enabled, and disables it
otherwise.

- Alex

On Sat, May 20, 2017 at 7:29 AM, Zytaruk, Kelly <Kelly.Zytaruk@amd.com> wrote:
> Collins,
>
> Okay, good to know.
> Is there a common solution that can handle all cases?
>
> Thanks,
> Kelly
>
>>-----Original Message-----
>>From: Cheng, Collins
>>Sent: Saturday, May 20, 2017 6:38 AM
>>To: Zytaruk, Kelly; Alexander Duyck; Alex Williamson
>>Cc: Bjorn Helgaas; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
>>Deucher, Alexander; Yinghai Lu
>>Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>>incapable platform
>>
>>Hi Kelly,
>>
>>This issue also happens in "not SR-IOV capable" SBIOS. It seems some "not SR-IOV
>>capable" SBIOS will directly report error in system BIOS boot stage and doesn't
>>boot to OS. But other "not SR-IOV capable" SBIOS would not report error and
>>boot to Linux.
>>
>>-Collins Cheng
>>
>>
>>-----Original Message-----
>>From: Zytaruk, Kelly
>>Sent: Saturday, May 20, 2017 6:28 PM
>>To: Cheng, Collins <Collins.Cheng@amd.com>; Alexander Duyck
>><alexander.duyck@gmail.com>; Alex Williamson <alex.williamson@redhat.com>
>>Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-
>>kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>;
>>Yinghai Lu <yinghai@kernel.org>
>>Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>>incapable platform
>>
>>
>>
>>>-----Original Message-----
>>>From: Cheng, Collins
>>>Sent: Saturday, May 20, 2017 12:53 AM
>>>To: Alexander Duyck; Alex Williamson
>>>Cc: Bjorn Helgaas; linux-pci@vger.kernel.org;
>>>linux-kernel@vger.kernel.org; Deucher, Alexander; Zytaruk, Kelly;
>>>Yinghai Lu
>>>Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>>>incapable platform
>>>
>>>Hi Alex,
>>>
>>>Yes, I hope kernel can disable SR-IOV and related VF resource
>>>allocation if the system BIOS is not SR-IOV capable.
>>>
>>>Adding the parameter "pci=nosriov" sounds a doable solution, but it
>>>would need user to add this parameter manually, right? I think an
>>>automatic detection would be better. My patch is trying to auto detect and
>>bypass VF resource allocation.
>>>
>>>
>>>-Collins Cheng
>>>
>>
>>Collins, be careful about this.  I don't think that this is what we want.  If you add
>>"pci=nosriov" then you are globally disabling SRIOV for all devices.  This is not the
>>solution that we are looking for.
>>Remember that there are 3 types of SBIOS; "not SR-IOV capable", "SR-IOV
>>capable but does not support large resources", "Complete SR-IOV support".
>>
>>The problem is that we are trying to find a fix for "broken" SBIOS that does
>>support SR-IOV but does not support the full SR-IOV capabilities that devices with
>>large resources require.
>>
>>Thanks,
>>Kelly
>>
>>>
>>>-----Original Message-----
>>>From: Alexander Duyck [mailto:alexander.duyck@gmail.com]
>>>Sent: Friday, May 19, 2017 11:44 PM
>>>To: Alex Williamson <alex.williamson@redhat.com>
>>>Cc: Cheng, Collins <Collins.Cheng@amd.com>; Bjorn Helgaas
>>><bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-
>>>kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>;
>>>Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
>>>Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
>>>incapable platform
>>>
>>>On Mon, May 15, 2017 at 10:53 AM, Alex Williamson
>>><alex.williamson@redhat.com> wrote:
>>>> On Mon, 15 May 2017 08:19:28 +0000
>>>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>>>
>>>>> Hi Williamson,
>>>>>
>>>>> We cannot assume BIOS supports SR-IOV, actually only newer server
>>>motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or
>>>older server motherboard BIOS doesn't support SR-IOV. This issue would
>>>happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop
>>motherboard.
>>>>
>>>> Servers should be supporting SR-IOV for a long time now.  What really
>>>> is there to a BIOS supporting SR-IOV anyway, it's simply reserving
>>>> sufficient bus number and MMIO resources such that we can enable the
>>>> VFs.  This process isn't exclusively reserved for the BIOS.  Some
>>>> platforms may choose to only initialize boot devices, leaving the
>>>> rest for the OS to program.  The initial proposal here to disable
>>>> SR-IOV if not programmed at OS hand-off disables even the possibility
>>>> of the OS reallocating resources for this device.
>>>
>>>There are differences between supporting SR-IOV and supporting SR-IOV
>>>on devices with massive resources. I know I have seen NICs that will
>>>keep a system from completing POST if SR-IOV is enabled, and MMIO
>>>beyond 4G is not. My guess would be that the issues being seen are
>>>probably that they disable SR-IOV in the BIOS in such a setup and end
>>>up running into issues when they try to boot into the Linux kernel as
>>>it goes through and tries to allocate resources for SR-IOV even though it was
>>disabled in the BIOS.
>>>
>>>It might make sense to add a kernel parameter something like a "pci=nosriov"
>>>that would allow for disabling SR-IOV and related resource allocation
>>>if that is what we are talking about. That way you could plug in these
>>>types of devices into a system with a legacy bios or that doesn't wan
>>>to allocate addresses above 32b for MMIO, and this parameter would be
>>>all that is needed to disable SR-IOV so you could plug in a NIC that has SR-IOV
>>associated with it.
>>>
>>>>> I agree that failure to allocate VF resources should leave the
>>>>> device in no
>>>worse condition than before it tried. I hope kernel could allocate PF
>>>device resource before allocating VF device resource, and keep PF
>>>device resource valid and functional if failed to allocate VF device resource.
>>>>>
>>>>> I will send out dmesg log lspci info tomorrow. Thanks.
>>>>
>>>> Thanks,
>>>> Alex
>>>>
>>>>> -----Original Message-----
>>>>> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>>>> Sent: Friday, May 12, 2017 10:43 PM
>>>>> To: Cheng, Collins <Collins.Cheng@amd.com>
>>>>> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>>>>> linux-kernel@vger.kernel.org; Deucher, Alexander
>>>>> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>;
>>>>> Yinghai Lu <yinghai@kernel.org>
>>>>> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>>>>> SR-IOV incapable platform
>>>>>
>>>>> On Fri, 12 May 2017 04:51:43 +0000
>>>>> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>>>>>
>>>>> > Hi Williamson,
>>>>> >
>>>>> > I verified the patch is working for both AMD SR-IOV GPU and Intel
>>>>> > SR-IOV
>>>NIC. I don't think it is redundant to check the VF BAR valid before
>>>call sriov_init(), it is safe and saving boot time, also there is no a
>>>better method to know if system BIOS has correctly initialized the SR-IOV
>>capability or not.
>>>>>
>>>>> It also masks an underlying bug and creates a maintenance issue that
>>>>> we won't
>>>know when it's safe to remove this workaround.  I don't think faster
>>>boot is valid rationale, in one case SR-IOV is completely disabled, the
>>>other we attempt to allocate the resources the BIOS failed to provide.
>>>I expect this is also a corner case, the BIOS should typically support
>>>SR-IOV, therefore this situation should be an exception.
>>>>>
>>>>> > I did not try to fix the issue from the kernel resource allocation
>>>>> > perspective,
>>>it is because:
>>>>> >     1. I am not very familiar with the PCI resource allocation scheme in
>>kernel.
>>>For example, in sriov_init(), kernel will re-assign the PCI resource
>>>for both VF and PF. I don't understand why kernel allocates resource
>>>for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
>>>>> >     2. I am not sure if kernel has error handler if PCI resource allocation
>>failed.
>>>In this case, kernel cannot allocate enough resource to PF. It should
>>>trigger some error handler to either just keep original BAR values set
>>>by system BIOS, or disable this device and log errors.
>>>>>
>>>>> I think these are the issues we should be trying to solve and I'm
>>>>> sure folks on the linux-pci list can help us identify the bug.
>>>>> Minimally, failure to allocate VF resources should leave the device
>>>>> in no worse condition than before it tried.  Perhaps you could post
>>>>> more details about the issue, boot with pci=earlydump, post dmesg of
>>>>> a boot where the PF resources are incorrectly re-allocated, and
>>>>> include lspci -vvv for the SR-IOV device.  Also, please test with
>>>>> the latest upstream kernel, upstream only patches old kernels
>>>>> through stable backports of commits to the latest kernel.  Adding
>>>>> Yinghai as a resource allocation expert. Thanks,
>>>>>
>>>>> Alex
>>>>>
>>>>> > -----Original Message-----
>>>>> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>>>> > Sent: Friday, May 12, 2017 12:01 PM
>>>>> > To: Cheng, Collins <Collins.Cheng@amd.com>
>>>>> > Cc: Bjorn Helgaas <bhelgaas@google.com>;
>>>>> > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher,
>>>>> > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>>>>> > <Kelly.Zytaruk@amd.com>
>>>>> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>>>>> > SR-IOV incapable platform
>>>>> >
>>>>> > On Fri, 12 May 2017 03:42:46 +0000 "Cheng, Collins"
>>>>> > <Collins.Cheng@amd.com> wrote:
>>>>> >
>>>>> > > Hi Williamson,
>>>>> > >
>>>>> > > GPU card needs more BAR aperture resource than other PCI
>>>>> > > devices. For
>>>example, Intel SR-IOV network card only require 512KB memory resource
>>>for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory
>>>resource for frame buffer BAR aperture.
>>>>> > >
>>>>> > > If the system BIOS supports SR-IOV, it will reserve enough
>>>>> > > resource for all
>>>VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate
>>>the enough resource for VF BARs, only PF BAR will be assigned and VF
>>>BARs are empty. Then system boots to Linux kernel and kernel doesn't
>>>check if the VF BARs are empty or valid. Kernel will re-assign the BAR
>>>resources for PF and all VFs. The problem I saw is that kernel will
>>>fail to allocate PF BAR resource because some resources are assigned to
>>>VF, this is not expected. So kernel might need to do some check before
>>>re-assign the PF/VF resource, so that PF device will be correctly assigned BAR
>>resource and user can use PF device.
>>>>> >
>>>>> > So the problem is that something bad happens when the kernel is
>>>>> > trying to reallocate resources in order to fulfill the
>>>>> > requirements of the VFs, leaving the PF resources incorrectly
>>>>> > programmed?  Why not just fix that bug rather than creating
>>>>> > special handling for this vendor/class of device which disables
>>>>> > any attempt to fixup resources for SR-IOV?  IOW, this patch just
>>>>> > avoids the problem for your devices rather than fixing the bug.
>>>>> > I'd suggest fixing the bug such that the PF is left in a
>>>>> > functional state if the kernel is unable to allocate sufficient
>>>>> > resources for the VFs.  Thanks,
>>>>> >
>>>>> > Alex
>>>>> >
>>>>> > > -----Original Message-----
>>>>> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>>>>> > > Sent: Friday, May 12, 2017 11:21 AM
>>>>> > > To: Cheng, Collins <Collins.Cheng@amd.com>
>>>>> > > Cc: Bjorn Helgaas <bhelgaas@google.com>;
>>>>> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
>>>>> > > Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>>>>> > > <Kelly.Zytaruk@amd.com>
>>>>> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>>>>> > > SR-IOV incapable platform
>>>>> > >
>>>>> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins"
>>>>> > > <Collins.Cheng@amd.com> wrote:
>>>>> > >
>>>>> > > > Hi Helgaas,
>>>>> > > >
>>>>> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
>>>>> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable
>>>>> > > > platform. It would cause a problem on PCI resource allocation
>>>>> > > > in current Linux kernel.
>>>>> > > >
>>>>> > > > Therefore in order to allow the PF (Physical Function) device
>>>>> > > > of SR-IOV capable GPU to work on the SR-IOV incapable
>>>>> > > > platform, it is required to verify conditions for initializing
>>>>> > > > BAR resources on AMD SR-IOV capable GPUs.
>>>>> > > >
>>>>> > > > If the device is an AMD graphics device and it supports SR-IOV
>>>>> > > > it will require a large amount of resources.
>>>>> > > > Before calling sriov_init() must ensure that the system BIOS
>>>>> > > > also supports SR-IOV and that system BIOS has been able to
>>>>> > > > allocate enough resources.
>>>>> > > > If the VF BARs are zero then the system BIOS does not support
>>>>> > > > SR-IOV or it could not allocate the resources and this
>>>>> > > > platform will not support AMD graphics SR-IOV.
>>>>> > > > Therefore do not call sriov_init().
>>>>> > > > If the system BIOS does support SR-IOV then the VF BARs will
>>>>> > > > be properly initialized to non-zero values.
>>>>> > > >
>>>>> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
>>>>> > > >
>>>>> > > > I checked the drivers/pci/quirks.c, it looks the
>>>>> > > > workarounds/fixes in quirks.c are for specific devices and one
>>>>> > > > or more device ID are defined for the specific devices.
>>>>> > > > However my patch is for all AMD SR-IOV capable GPUs, that
>>>>> > > > includes all existing
>>>and future AMD server GPUs.
>>>>> > > > So it doesn't seem like a good fit to put the fix in quirks.c.
>>>>> > >
>>>>> > >
>>>>> > > Why is an AMD graphics card unique here?  Doesn't sriov_init()
>>>>> > > always need to be able to deal with devices of any type where
>>>>> > > the BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV
>>>>> > > devices can fit their VFs within a minimum bridge aperture, most
>>>>> > > cannot.  I don't understand why the VF resource requirements
>>>>> > > being exceptionally large dictates that they receive special handling.
>>>>> > > Thanks,
>>>>> > >
>>>>> > > Alex
>>>>> > >
>>>>> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>>>>> > > > ---
>>>>> > > >  drivers/pci/iov.c | 63
>>>>> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>>>>> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
>>>>> > > >
>>>>> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>>>>> > > > e30f05c..e4f1405 100644
>>>>> > > > --- a/drivers/pci/iov.c
>>>>> > > > +++ b/drivers/pci/iov.c
>>>>> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct
>>>>> > > > pci_dev
>>>*dev)
>>>>> > > >                 msleep(100);
>>>>> > > >  }
>>>>> > > >
>>>>> > > > +/*
>>>>> > > > + * pci_vf_bar_valid - check if VF BARs have resource
>>>>> > > > +allocated
>>>>> > > > + * @dev: the PCI device
>>>>> > > > + * @pos: register offset of SR-IOV capability in PCI config
>>>>> > > > +space
>>>>> > > > + * Returns true any VF BAR has resource allocated, false
>>>>> > > > + * if all VF BARs are empty.
>>>>> > > > + */
>>>>> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>>>>> > > > +       int i;
>>>>> > > > +       u32 bar_value;
>>>>> > > > +       u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
>>>>> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
>>>>> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
>>>>> > > > +
>>>>> > > > +       for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>>>>> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR
>>>>> > > > + + i * 4,
>>>&bar_value);
>>>>> > > > +               if (bar_value & bar_size_mask)
>>>>> > > > +                       return true;
>>>>> > > > +       }
>>>>> > > > +
>>>>> > > > +       return false;
>>>>> > > > +}
>>>>> > > > +
>>>>> > > > +/*
>>>>> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU
>>>>> > > > +device
>>>>> > > > + * @dev: the PCI device
>>>>> > > > + *
>>>>> > > > + * Returns true if device is an AMD/ATI display adapter,
>>>>> > > > + * otherwise return false.
>>>>> > > > + */
>>>>> > > > +
>>>>> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
>>>>> > > > +       return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
>>>>> > > > +               (dev->vendor == PCI_VENDOR_ID_ATI ||
>>>>> > > > +               dev->vendor == PCI_VENDOR_ID_AMD)); }
>>>>> > > > +
>>>>> > > >  /**
>>>>> > > >   * pci_iov_init - initialize the IOV capability
>>>>> > > >   * @dev: the PCI device
>>>>> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>>>>> > > >                 return -ENODEV;
>>>>> > > >
>>>>> > > >         pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
>>>>> > > > -       if (pos)
>>>>> > > > -               return sriov_init(dev, pos);
>>>>> > > > -
>>>>> > > > +       if (pos) {
>>>>> > > > +       /*
>>>>> > > > +        * If the device is an AMD graphics device and it supports
>>>>> > > > +        * SR-IOV it will require a large amount of resources.
>>>>> > > > +        * Before calling sriov_init() must ensure that the system
>>>>> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
>>>>> > > > +        * able to allocate enough resources.
>>>>> > > > +        * If the VF BARs are zero then the system BIOS does not
>>>>> > > > +        * support SR-IOV or it could not allocate the resources
>>>>> > > > +        * and this platform will not support AMD graphics SR-IOV.
>>>>> > > > +        * Therefore do not call sriov_init().
>>>>> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
>>>>> > > > +        * will be properly initialized to non-zero values.
>>>>> > > > +        */
>>>>> > > > +               if (is_amd_display_adapter(dev)) {
>>>>> > > > +                       if (pci_vf_bar_valid(dev, pos))
>>>>> > > > +                               return sriov_init(dev, pos);
>>>>> > > > +               } else {
>>>>> > > > +                       return sriov_init(dev, pos);
>>>>> > > > +               }
>>>>> > > > +       }
>>>>> > > >         return -ENODEV;
>>>>> > > >  }
>>>>> > > >
>>>>> > >
>>>>> >
>>>>>
>>>>

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alex Williamson]

On Fri, 19 May 2017 08:43:38 -0700
Alexander Duyck <alexander.duyck@gmail.com> wrote:

> On Mon, May 15, 2017 at 10:53 AM, Alex Williamson
> <alex.williamson@redhat.com> wrote:
> > On Mon, 15 May 2017 08:19:28 +0000
> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >  
> >> Hi Williamson,
> >>
> >> We cannot assume BIOS supports SR-IOV, actually only newer server motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server motherboard BIOS doesn't support SR-IOV. This issue would happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.  
> >
> > Servers should be supporting SR-IOV for a long time now.  What really
> > is there to a BIOS supporting SR-IOV anyway, it's simply reserving
> > sufficient bus number and MMIO resources such that we can enable the
> > VFs.  This process isn't exclusively reserved for the BIOS.  Some
> > platforms may choose to only initialize boot devices, leaving the rest
> > for the OS to program.  The initial proposal here to disable SR-IOV if
> > not programmed at OS hand-off disables even the possibility of the OS
> > reallocating resources for this device.  
> 
> There are differences between supporting SR-IOV and supporting SR-IOV
> on devices with massive resources. I know I have seen NICs that will
> keep a system from completing POST if SR-IOV is enabled, and MMIO
> beyond 4G is not. My guess would be that the issues being seen are
> probably that they disable SR-IOV in the BIOS in such a setup and end
> up running into issues when they try to boot into the Linux kernel as
> it goes through and tries to allocate resources for SR-IOV even though
> it was disabled in the BIOS.
> 
> It might make sense to add a kernel parameter something like a
> "pci=nosriov" that would allow for disabling SR-IOV and related
> resource allocation if that is what we are talking about. That way you
> could plug in these types of devices into a system with a legacy bios
> or that doesn't wan to allocate addresses above 32b for MMIO, and this
> parameter would be all that is needed to disable SR-IOV so you could
> plug in a NIC that has SR-IOV associated with it.

Hi,

a) I think we're still ignoring the real bug that is something goes
wrong with the reallocation leaving the PF without resources.

b) Why does an option to avoid re-allocation need to be sr-iov
specific?  Shouldn't pci=realloc=off cover this?

Thanks,
Alex

> >> I agree that failure to allocate VF resources should leave the device in no worse condition than before it tried. I hope kernel could allocate PF device resource before allocating VF device resource, and keep PF device resource valid and functional if failed to allocate VF device resource.
> >>
> >> I will send out dmesg log lspci info tomorrow. Thanks.  
> >
> > Thanks,
> > Alex
> >  
> >> -----Original Message-----
> >> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> >> Sent: Friday, May 12, 2017 10:43 PM
> >> To: Cheng, Collins <Collins.Cheng@amd.com>
> >> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
> >> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform
> >>
> >> On Fri, 12 May 2017 04:51:43 +0000
> >> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >>  
> >> > Hi Williamson,
> >> >
> >> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.  
> >>
> >> It also masks an underlying bug and creates a maintenance issue that we won't know when it's safe to remove this workaround.  I don't think faster boot is valid rationale, in one case SR-IOV is completely disabled, the other we attempt to allocate the resources the BIOS failed to provide.  I expect this is also a corner case, the BIOS should typically support SR-IOV, therefore this situation should be an exception.
> >>  
> >> > I did not try to fix the issue from the kernel resource allocation perspective, it is because:
> >> >     1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
> >> >     2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.  
> >>
> >> I think these are the issues we should be trying to solve and I'm sure folks on the linux-pci list can help us identify the bug.  Minimally, failure to allocate VF resources should leave the device in no worse condition than before it tried.  Perhaps you could post more details about the issue, boot with pci=earlydump, post dmesg of a boot where the PF resources are incorrectly re-allocated, and include lspci -vvv for the SR-IOV device.  Also, please test with the latest upstream kernel, upstream only patches old kernels through stable backports of commits to the latest kernel.  Adding Yinghai as a resource allocation expert. Thanks,
> >>
> >> Alex
> >>  
> >> > -----Original Message-----
> >> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> >> > Sent: Friday, May 12, 2017 12:01 PM
> >> > To: Cheng, Collins <Collins.Cheng@amd.com>
> >> > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
> >> > linux-kernel@vger.kernel.org; Deucher, Alexander
> >> > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> >> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
> >> > SR-IOV incapable platform
> >> >
> >> > On Fri, 12 May 2017 03:42:46 +0000
> >> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >> >  
> >> > > Hi Williamson,
> >> > >
> >> > > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
> >> > >
> >> > > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.  
> >> >
> >> > So the problem is that something bad happens when the kernel is trying
> >> > to reallocate resources in order to fulfill the requirements of the
> >> > VFs, leaving the PF resources incorrectly programmed?  Why not just
> >> > fix that bug rather than creating special handling for this
> >> > vendor/class of device which disables any attempt to fixup resources
> >> > for SR-IOV?  IOW, this patch just avoids the problem for your devices
> >> > rather than fixing the bug.  I'd suggest fixing the bug such that the
> >> > PF is left in a functional state if the kernel is unable to allocate
> >> > sufficient resources for the VFs.  Thanks,
> >> >
> >> > Alex
> >> >  
> >> > > -----Original Message-----
> >> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> >> > > Sent: Friday, May 12, 2017 11:21 AM
> >> > > To: Cheng, Collins <Collins.Cheng@amd.com>
> >> > > Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
> >> > > linux-kernel@vger.kernel.org; Deucher, Alexander
> >> > > <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>
> >> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
> >> > > SR-IOV incapable platform
> >> > >
> >> > > On Fri, 12 May 2017 02:50:32 +0000
> >> > > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >> > >  
> >> > > > Hi Helgaas,
> >> > > >
> >> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
> >> > > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable
> >> > > > platform. It would cause a problem on PCI resource allocation in
> >> > > > current Linux kernel.
> >> > > >
> >> > > > Therefore in order to allow the PF (Physical Function) device of
> >> > > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is
> >> > > > required to verify conditions for initializing BAR resources on
> >> > > > AMD SR-IOV capable GPUs.
> >> > > >
> >> > > > If the device is an AMD graphics device and it supports SR-IOV it
> >> > > > will require a large amount of resources.
> >> > > > Before calling sriov_init() must ensure that the system BIOS also
> >> > > > supports SR-IOV and that system BIOS has been able to allocate
> >> > > > enough resources.
> >> > > > If the VF BARs are zero then the system BIOS does not support
> >> > > > SR-IOV or it could not allocate the resources and this platform
> >> > > > will not support AMD graphics SR-IOV.
> >> > > > Therefore do not call sriov_init().
> >> > > > If the system BIOS does support SR-IOV then the VF BARs will be
> >> > > > properly initialized to non-zero values.
> >> > > >
> >> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> >> > > >
> >> > > > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes
> >> > > > in quirks.c are for specific devices and one or more device ID are
> >> > > > defined for the specific devices. However my patch is for all AMD
> >> > > > SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> >> > > > So it doesn't seem like a good fit to put the fix in quirks.c.  
> >> > >
> >> > >
> >> > > Why is an AMD graphics card unique here?  Doesn't sriov_init()
> >> > > always need to be able to deal with devices of any type where the
> >> > > BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV devices
> >> > > can fit their VFs within a minimum bridge aperture, most cannot.  I
> >> > > don't understand why the VF resource requirements being
> >> > > exceptionally large dictates that they receive special handling.
> >> > > Thanks,
> >> > >
> >> > > Alex
> >> > >  
> >> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> >> > > > ---
> >> > > >  drivers/pci/iov.c | 63
> >> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> >> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
> >> > > >
> >> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> >> > > > e30f05c..e4f1405 100644
> >> > > > --- a/drivers/pci/iov.c
> >> > > > +++ b/drivers/pci/iov.c
> >> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
> >> > > >                 msleep(100);
> >> > > >  }
> >> > > >
> >> > > > +/*
> >> > > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> >> > > > + * @dev: the PCI device
> >> > > > + * @pos: register offset of SR-IOV capability in PCI config space
> >> > > > + * Returns true any VF BAR has resource allocated, false
> >> > > > + * if all VF BARs are empty.
> >> > > > + */
> >> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> >> > > > +       int i;
> >> > > > +       u32 bar_value;
> >> > > > +       u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> >> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
> >> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
> >> > > > +
> >> > > > +       for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> >> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> >> > > > +               if (bar_value & bar_size_mask)
> >> > > > +                       return true;
> >> > > > +       }
> >> > > > +
> >> > > > +       return false;
> >> > > > +}
> >> > > > +
> >> > > > +/*
> >> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> >> > > > + * @dev: the PCI device
> >> > > > + *
> >> > > > + * Returns true if device is an AMD/ATI display adapter,
> >> > > > + * otherwise return false.
> >> > > > + */
> >> > > > +
> >> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> >> > > > +       return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> >> > > > +               (dev->vendor == PCI_VENDOR_ID_ATI ||
> >> > > > +               dev->vendor == PCI_VENDOR_ID_AMD)); }
> >> > > > +
> >> > > >  /**
> >> > > >   * pci_iov_init - initialize the IOV capability
> >> > > >   * @dev: the PCI device
> >> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> >> > > >                 return -ENODEV;
> >> > > >
> >> > > >         pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> >> > > > -       if (pos)
> >> > > > -               return sriov_init(dev, pos);
> >> > > > -
> >> > > > +       if (pos) {
> >> > > > +       /*
> >> > > > +        * If the device is an AMD graphics device and it supports
> >> > > > +        * SR-IOV it will require a large amount of resources.
> >> > > > +        * Before calling sriov_init() must ensure that the system
> >> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
> >> > > > +        * able to allocate enough resources.
> >> > > > +        * If the VF BARs are zero then the system BIOS does not
> >> > > > +        * support SR-IOV or it could not allocate the resources
> >> > > > +        * and this platform will not support AMD graphics SR-IOV.
> >> > > > +        * Therefore do not call sriov_init().
> >> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
> >> > > > +        * will be properly initialized to non-zero values.
> >> > > > +        */
> >> > > > +               if (is_amd_display_adapter(dev)) {
> >> > > > +                       if (pci_vf_bar_valid(dev, pos))
> >> > > > +                               return sriov_init(dev, pos);
> >> > > > +               } else {
> >> > > > +                       return sriov_init(dev, pos);
> >> > > > +               }
> >> > > > +       }
> >> > > >         return -ENODEV;
> >> > > >  }
> >> > > >  
> >> > >  
> >> >  
> >>  
> >  

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

[-- Attachment #1: Type: text/plain, Size: 17323 bytes --]

Hi Alex,

I owe you a dmesg log. Attachment are two log files. 1.txt is without "pci=earlydump", 2.txt is with "pci=earlydump". The platform is an ASUS Z170-A motherboard that doesn't support SR-IOV. The graphics card is AMD FirePro S7150 card which enabled SR-IOV. 

You could find the error info like below in both logs. From the log, kernel failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (256MB needed), but kernel reallocated resource for BAR9 which is for VF. You are right, the real bug that is something goes wrong with the reallocation leaving the PF without resources.

[    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
[    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
[    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]

-Collins Cheng

-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com] 
Sent: Monday, May 22, 2017 11:44 PM
To: Alexander Duyck <alexander.duyck@gmail.com>
Cc: Cheng, Collins <Collins.Cheng@amd.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Fri, 19 May 2017 08:43:38 -0700
Alexander Duyck <alexander.duyck@gmail.com> wrote:

> On Mon, May 15, 2017 at 10:53 AM, Alex Williamson 
> <alex.williamson@redhat.com> wrote:
> > On Mon, 15 May 2017 08:19:28 +0000
> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >  
> >> Hi Williamson,
> >>
> >> We cannot assume BIOS supports SR-IOV, actually only newer server motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server motherboard BIOS doesn't support SR-IOV. This issue would happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.  
> >
> > Servers should be supporting SR-IOV for a long time now.  What 
> > really is there to a BIOS supporting SR-IOV anyway, it's simply 
> > reserving sufficient bus number and MMIO resources such that we can 
> > enable the VFs.  This process isn't exclusively reserved for the 
> > BIOS.  Some platforms may choose to only initialize boot devices, 
> > leaving the rest for the OS to program.  The initial proposal here 
> > to disable SR-IOV if not programmed at OS hand-off disables even the 
> > possibility of the OS reallocating resources for this device.
> 
> There are differences between supporting SR-IOV and supporting SR-IOV 
> on devices with massive resources. I know I have seen NICs that will 
> keep a system from completing POST if SR-IOV is enabled, and MMIO 
> beyond 4G is not. My guess would be that the issues being seen are 
> probably that they disable SR-IOV in the BIOS in such a setup and end 
> up running into issues when they try to boot into the Linux kernel as 
> it goes through and tries to allocate resources for SR-IOV even though 
> it was disabled in the BIOS.
> 
> It might make sense to add a kernel parameter something like a 
> "pci=nosriov" that would allow for disabling SR-IOV and related 
> resource allocation if that is what we are talking about. That way you 
> could plug in these types of devices into a system with a legacy bios 
> or that doesn't wan to allocate addresses above 32b for MMIO, and this 
> parameter would be all that is needed to disable SR-IOV so you could 
> plug in a NIC that has SR-IOV associated with it.

Hi,

a) I think we're still ignoring the real bug that is something goes wrong with the reallocation leaving the PF without resources.

b) Why does an option to avoid re-allocation need to be sr-iov specific?  Shouldn't pci=realloc=off cover this?

Thanks,
Alex

> >> I agree that failure to allocate VF resources should leave the device in no worse condition than before it tried. I hope kernel could allocate PF device resource before allocating VF device resource, and keep PF device resource valid and functional if failed to allocate VF device resource.
> >>
> >> I will send out dmesg log lspci info tomorrow. Thanks.  
> >
> > Thanks,
> > Alex
> >  
> >> -----Original Message-----
> >> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> >> Sent: Friday, May 12, 2017 10:43 PM
> >> To: Cheng, Collins <Collins.Cheng@amd.com>
> >> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> >> linux-kernel@vger.kernel.org; Deucher, Alexander 
> >> <Alexander.Deucher@amd.com>; Zytaruk, Kelly 
> >> <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
> >> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> >> SR-IOV incapable platform
> >>
> >> On Fri, 12 May 2017 04:51:43 +0000
> >> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> >>  
> >> > Hi Williamson,
> >> >
> >> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.  
> >>
> >> It also masks an underlying bug and creates a maintenance issue that we won't know when it's safe to remove this workaround.  I don't think faster boot is valid rationale, in one case SR-IOV is completely disabled, the other we attempt to allocate the resources the BIOS failed to provide.  I expect this is also a corner case, the BIOS should typically support SR-IOV, therefore this situation should be an exception.
> >>  
> >> > I did not try to fix the issue from the kernel resource allocation perspective, it is because:
> >> >     1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
> >> >     2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.  
> >>
> >> I think these are the issues we should be trying to solve and I'm 
> >> sure folks on the linux-pci list can help us identify the bug.  
> >> Minimally, failure to allocate VF resources should leave the device 
> >> in no worse condition than before it tried.  Perhaps you could post 
> >> more details about the issue, boot with pci=earlydump, post dmesg 
> >> of a boot where the PF resources are incorrectly re-allocated, and 
> >> include lspci -vvv for the SR-IOV device.  Also, please test with 
> >> the latest upstream kernel, upstream only patches old kernels 
> >> through stable backports of commits to the latest kernel.  Adding 
> >> Yinghai as a resource allocation expert. Thanks,
> >>
> >> Alex
> >>  
> >> > -----Original Message-----
> >> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> >> > Sent: Friday, May 12, 2017 12:01 PM
> >> > To: Cheng, Collins <Collins.Cheng@amd.com>
> >> > Cc: Bjorn Helgaas <bhelgaas@google.com>; 
> >> > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, 
> >> > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly 
> >> > <Kelly.Zytaruk@amd.com>
> >> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> >> > SR-IOV incapable platform
> >> >
> >> > On Fri, 12 May 2017 03:42:46 +0000 "Cheng, Collins" 
> >> > <Collins.Cheng@amd.com> wrote:
> >> >  
> >> > > Hi Williamson,
> >> > >
> >> > > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
> >> > >
> >> > > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.  
> >> >
> >> > So the problem is that something bad happens when the kernel is 
> >> > trying to reallocate resources in order to fulfill the 
> >> > requirements of the VFs, leaving the PF resources incorrectly 
> >> > programmed?  Why not just fix that bug rather than creating 
> >> > special handling for this vendor/class of device which disables 
> >> > any attempt to fixup resources for SR-IOV?  IOW, this patch just 
> >> > avoids the problem for your devices rather than fixing the bug.  
> >> > I'd suggest fixing the bug such that the PF is left in a 
> >> > functional state if the kernel is unable to allocate sufficient 
> >> > resources for the VFs.  Thanks,
> >> >
> >> > Alex
> >> >  
> >> > > -----Original Message-----
> >> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> >> > > Sent: Friday, May 12, 2017 11:21 AM
> >> > > To: Cheng, Collins <Collins.Cheng@amd.com>
> >> > > Cc: Bjorn Helgaas <bhelgaas@google.com>; 
> >> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; 
> >> > > Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly 
> >> > > <Kelly.Zytaruk@amd.com>
> >> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on 
> >> > > the SR-IOV incapable platform
> >> > >
> >> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins" 
> >> > > <Collins.Cheng@amd.com> wrote:
> >> > >  
> >> > > > Hi Helgaas,
> >> > > >
> >> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
> >> > > > If the SR-IOV capable GPU is plugged into the SR-IOV 
> >> > > > incapable platform. It would cause a problem on PCI resource 
> >> > > > allocation in current Linux kernel.
> >> > > >
> >> > > > Therefore in order to allow the PF (Physical Function) device 
> >> > > > of SR-IOV capable GPU to work on the SR-IOV incapable 
> >> > > > platform, it is required to verify conditions for 
> >> > > > initializing BAR resources on AMD SR-IOV capable GPUs.
> >> > > >
> >> > > > If the device is an AMD graphics device and it supports 
> >> > > > SR-IOV it will require a large amount of resources.
> >> > > > Before calling sriov_init() must ensure that the system BIOS 
> >> > > > also supports SR-IOV and that system BIOS has been able to 
> >> > > > allocate enough resources.
> >> > > > If the VF BARs are zero then the system BIOS does not support 
> >> > > > SR-IOV or it could not allocate the resources and this 
> >> > > > platform will not support AMD graphics SR-IOV.
> >> > > > Therefore do not call sriov_init().
> >> > > > If the system BIOS does support SR-IOV then the VF BARs will 
> >> > > > be properly initialized to non-zero values.
> >> > > >
> >> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> >> > > >
> >> > > > I checked the drivers/pci/quirks.c, it looks the 
> >> > > > workarounds/fixes in quirks.c are for specific devices and 
> >> > > > one or more device ID are defined for the specific devices. 
> >> > > > However my patch is for all AMD SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> >> > > > So it doesn't seem like a good fit to put the fix in quirks.c.  
> >> > >
> >> > >
> >> > > Why is an AMD graphics card unique here?  Doesn't sriov_init() 
> >> > > always need to be able to deal with devices of any type where 
> >> > > the BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV 
> >> > > devices can fit their VFs within a minimum bridge aperture, 
> >> > > most cannot.  I don't understand why the VF resource 
> >> > > requirements being exceptionally large dictates that they receive special handling.
> >> > > Thanks,
> >> > >
> >> > > Alex
> >> > >  
> >> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> >> > > > ---
> >> > > >  drivers/pci/iov.c | 63
> >> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> >> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
> >> > > >
> >> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> >> > > > e30f05c..e4f1405 100644
> >> > > > --- a/drivers/pci/iov.c
> >> > > > +++ b/drivers/pci/iov.c
> >> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
> >> > > >                 msleep(100);
> >> > > >  }
> >> > > >
> >> > > > +/*
> >> > > > + * pci_vf_bar_valid - check if VF BARs have resource 
> >> > > > +allocated
> >> > > > + * @dev: the PCI device
> >> > > > + * @pos: register offset of SR-IOV capability in PCI config 
> >> > > > +space
> >> > > > + * Returns true any VF BAR has resource allocated, false
> >> > > > + * if all VF BARs are empty.
> >> > > > + */
> >> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> >> > > > +       int i;
> >> > > > +       u32 bar_value;
> >> > > > +       u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> >> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
> >> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
> >> > > > +
> >> > > > +       for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> >> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> >> > > > +               if (bar_value & bar_size_mask)
> >> > > > +                       return true;
> >> > > > +       }
> >> > > > +
> >> > > > +       return false;
> >> > > > +}
> >> > > > +
> >> > > > +/*
> >> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU 
> >> > > > +device
> >> > > > + * @dev: the PCI device
> >> > > > + *
> >> > > > + * Returns true if device is an AMD/ATI display adapter,
> >> > > > + * otherwise return false.
> >> > > > + */
> >> > > > +
> >> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> >> > > > +       return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> >> > > > +               (dev->vendor == PCI_VENDOR_ID_ATI ||
> >> > > > +               dev->vendor == PCI_VENDOR_ID_AMD)); }
> >> > > > +
> >> > > >  /**
> >> > > >   * pci_iov_init - initialize the IOV capability
> >> > > >   * @dev: the PCI device
> >> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> >> > > >                 return -ENODEV;
> >> > > >
> >> > > >         pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> >> > > > -       if (pos)
> >> > > > -               return sriov_init(dev, pos);
> >> > > > -
> >> > > > +       if (pos) {
> >> > > > +       /*
> >> > > > +        * If the device is an AMD graphics device and it supports
> >> > > > +        * SR-IOV it will require a large amount of resources.
> >> > > > +        * Before calling sriov_init() must ensure that the system
> >> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
> >> > > > +        * able to allocate enough resources.
> >> > > > +        * If the VF BARs are zero then the system BIOS does not
> >> > > > +        * support SR-IOV or it could not allocate the resources
> >> > > > +        * and this platform will not support AMD graphics SR-IOV.
> >> > > > +        * Therefore do not call sriov_init().
> >> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
> >> > > > +        * will be properly initialized to non-zero values.
> >> > > > +        */
> >> > > > +               if (is_amd_display_adapter(dev)) {
> >> > > > +                       if (pci_vf_bar_valid(dev, pos))
> >> > > > +                               return sriov_init(dev, pos);
> >> > > > +               } else {
> >> > > > +                       return sriov_init(dev, pos);
> >> > > > +               }
> >> > > > +       }
> >> > > >         return -ENODEV;
> >> > > >  }
> >> > > >  
> >> > >  
> >> >  
> >>  
> >  


[-- Attachment #2: 1.txt --]
[-- Type: text/plain, Size: 83636 bytes --]

[    0.000000] Linux version 4.9.0-custom (root@rexzhu) (gcc version 4.8.2 (GCC) ) #2 SMP Fri Mar 31 12:52:40 CST 2017
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-custom root=UUID=900c4a92-9f8e-44be-86bc-9673dd3b0d78 ro console=ttyS0,115200,8N1 console=tty0 no_console_suspend quiet splash vt.handoff=7
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009c7ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009c800-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007d438fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007d439000-0x000000007d439fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000007d43a000-0x000000007d483fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007d484000-0x000000007d4c4fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007d4c5000-0x000000007db49fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007db4a000-0x00000000858a7fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000858a8000-0x0000000086db6fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000086db7000-0x0000000086deffff] ACPI data
[    0.000000] BIOS-e820: [mem 0x0000000086df0000-0x000000008774bfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000008774c000-0x0000000087ffefff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000087fff000-0x0000000087ffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000880fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000004767fffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 3.0 present.
[    0.000000] DMI: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x476800 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 00C0000000 mask 7FC0000000 uncachable
[    0.000000]   1 base 00A0000000 mask 7FE0000000 uncachable
[    0.000000]   2 base 0090000000 mask 7FF0000000 uncachable
[    0.000000]   3 base 008C000000 mask 7FFC000000 uncachable
[    0.000000]   4 base 008A000000 mask 7FFE000000 uncachable
[    0.000000]   5 base 0089000000 mask 7FFF000000 uncachable
[    0.000000]   6 base 0088800000 mask 7FFF800000 uncachable
[    0.000000]   7 disabled
[    0.000000]   8 disabled
[    0.000000]   9 disabled
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT  
[    0.000000] e820: last_pfn = 0x88000 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000fcc50-0x000fcc5f] mapped at [ffff8800000fcc50]
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff880000096000] 96000 size 24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] BRK [0x020c7000, 0x020c7fff] PGTABLE
[    0.000000] BRK [0x020c8000, 0x020c8fff] PGTABLE
[    0.000000] BRK [0x020c9000, 0x020c9fff] PGTABLE
[    0.000000] BRK [0x020ca000, 0x020cafff] PGTABLE
[    0.000000] BRK [0x020cb000, 0x020cbfff] PGTABLE
[    0.000000] BRK [0x020cc000, 0x020ccfff] PGTABLE
[    0.000000] RAMDISK: [mem 0x2f6de000-0x33b66fff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000F0580 000024 (v02 ALASKA)
[    0.000000] ACPI: XSDT 0x0000000086DC2098 0000AC (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FACP 0x0000000086DE5038 00010C (v05 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x0000000086DC21E0 022E52 (v02 ALASKA A M I    01072009 INTL 20120913)
[    0.000000] ACPI: FACS 0x000000008774BF80 000040
[    0.000000] ACPI: APIC 0x0000000086DE5148 0000BC (v03 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FPDT 0x0000000086DE5208 000044 (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FIDT 0x0000000086DE5250 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: MCFG 0x0000000086DE52F0 00003C (v01 ALASKA A M I    01072009 MSFT 00000097)
[    0.000000] ACPI: HPET 0x0000000086DE5330 000038 (v01 ALASKA A M I    01072009 AMI. 0005000B)
[    0.000000] ACPI: SSDT 0x0000000086DE5368 00036D (v01 SataRe SataTabl 00001000 INTL 20120913)
[    0.000000] ACPI: LPIT 0x0000000086DE56D8 000094 (v01 INTEL  SKL      00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000086DE5770 000248 (v02 INTEL  sensrhub 00000000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE59B8 002BAE (v02 INTEL  PtidDevc 00001000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE8568 000C45 (v02 INTEL  Ther_Rvp 00001000 INTL 20120913)
[    0.000000] ACPI: DBGP 0x0000000086DE91B0 000034 (v01 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: DBG2 0x0000000086DE91E8 000054 (v00 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000086DE9240 0006FD (v02 INTEL  xh_rvp08 00000000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE9940 005271 (v02 SaSsdt SaSsdt   00003000 INTL 20120913)
[    0.000000] ACPI: UEFI 0x0000000086DEEBB8 000042 (v01                 00000000      00000000)
[    0.000000] ACPI: SSDT 0x0000000086DEEC00 000E58 (v02 CpuRef CpuSsdt  00003000 INTL 20120913)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x00000004767fffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x4767f8000-0x4767fbfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000004767fffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009bfff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x000000007d438fff]
[    0.000000]   node   0: [mem 0x000000007d484000-0x000000007d4c4fff]
[    0.000000]   node   0: [mem 0x000000007db4a000-0x00000000858a7fff]
[    0.000000]   node   0: [mem 0x0000000087fff000-0x0000000087ffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x00000004767fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000004767fffff]
[    0.000000] On node 0 totalpages: 4176244
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3995 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 8456 pages used for memmap
[    0.000000]   DMA32 zone: 541145 pages, LIFO batch:31
[    0.000000]   Normal zone: 56736 pages used for memmap
[    0.000000]   Normal zone: 3631104 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x1808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009c000-0x0009cfff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d439000-0x7d439fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d43a000-0x7d483fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d4c5000-0x7db49fff]
[    0.000000] PM: Registered nosave memory: [mem 0x858a8000-0x86db6fff]
[    0.000000] PM: Registered nosave memory: [mem 0x86db7000-0x86deffff]
[    0.000000] PM: Registered nosave memory: [mem 0x86df0000-0x8774bfff]
[    0.000000] PM: Registered nosave memory: [mem 0x8774c000-0x87ffefff]
[    0.000000] PM: Registered nosave memory: [mem 0x88000000-0x880fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x88100000-0xdfffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xe0000000-0xefffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfdffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe000000-0xfe010fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe011000-0xfebfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfedfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfeffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff]
[    0.000000] e820: [mem 0x88100000-0xdfffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:8 nr_node_ids:1
[    0.000000] percpu: Embedded 36 pages/cpu @ffff880476400000 s108056 r8192 d31208 u262144
[    0.000000] pcpu-alloc: s108056 r8192 d31208 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 4110967
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-custom root=UUID=900c4a92-9f8e-44be-86bc-9673dd3b0d78 ro console=ttyS0,115200,8N1 console=tty0 no_console_suspend quiet splash vt.handoff=7
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 16288164K/16704976K available (8091K kernel code, 1445K rwdata, 3668K rodata, 2220K init, 1180K bss, 416812K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=8.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=8
[    0.000000] NR_IRQS:16640 nr_irqs:2048 16
[    0.000000] 	Offload RCU callbacks from all CPUs
[    0.000000] 	Offload RCU callbacks from CPUs: 0-7.
[    0.000000] spurious 8259A interrupt: IRQ7.
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [ttyS0] enabled
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Detected 3400.000 MHz processor
[    0.000015] Calibrating delay loop (skipped), value calculated using timer frequency.. 6816.00 BogoMIPS (lpj=13632000)
[    0.000016] pid_max: default: 32768 minimum: 301
[    0.000035] ACPI: Core revision 20160831
[    0.049168] ACPI Error: [\_SB_.PCI0.XHC_.RHUB.HS11] Namespace lookup failure, AE_NOT_FOUND (20160831/dswload-210)
[    0.059605] ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20160831/psobject-227)
[    0.068365] ACPI Exception: AE_NOT_FOUND, (SSDT:xh_rvp08) while loading table (20160831/tbxfload-228)
[    0.089753] ACPI Error: 1 table load failures, 7 successful (20160831/tbxfload-246)
[    0.097652] Security Framework initialized
[    0.097653] Yama: becoming mindful.
[    0.097663] AppArmor: AppArmor initialized
[    0.098243] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.100214] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.101064] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.101074] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.101443] CPU: Physical Processor ID: 0
[    0.101443] CPU: Processor Core ID: 0
[    0.101446] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.101446] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.101449] mce: CPU supports 10 MCE banks
[    0.101457] CPU0: Thermal monitoring enabled (TM1)
[    0.101471] process: using mwait in idle threads
[    0.101472] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.101473] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.101726] Freeing SMP alternatives memory: 32K (ffffffff81f96000 - ffffffff81f9e000)
[    0.132575] ftrace: allocating 32207 entries in 126 pages
[    0.141555] smpboot: APIC(0) Converting physical 0 to logical package 0
[    0.141555] smpboot: Max logical packages: 2
[    0.142755] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.145898] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[    0.185586] TSC deadline timer enabled
[    0.185591] smpboot: CPU0: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz (family: 0x6, model: 0x5e, stepping: 0x3)
[    0.185593] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver.
[    0.185616] ... version:                4
[    0.185616] ... bit width:              48
[    0.185616] ... generic registers:      4
[    0.185617] ... value mask:             0000ffffffffffff
[    0.185617] ... max period:             00007fffffffffff
[    0.185617] ... fixed-purpose events:   3
[    0.185617] ... event mask:             000000070000000f
[    0.186321] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    0.186431] x86: Booting SMP configuration:
[    0.186432] .... node  #0, CPUs:      #1 #2 #3 #4 #5 #6 #7
[    0.747742] x86: Booted up 1 node, 8 CPUs
[    0.747744] smpboot: Total of 8 processors activated (54535.48 BogoMIPS)
[    0.754638] devtmpfs: initialized
[    0.754751] x86/mm: Memory block size: 128MB
[    0.759295] evm: security.selinux
[    0.759296] evm: security.SMACK64
[    0.759296] evm: security.ima
[    0.759296] evm: security.capability
[    0.759348] PM: Registering ACPI NVS region [mem 0x7d439000-0x7d439fff] (4096 bytes)
[    0.759349] PM: Registering ACPI NVS region [mem 0x86df0000-0x8774bfff] (9814016 bytes)
[    0.760138] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.760183] pinctrl core: initialized pinctrl subsystem
[    0.760416] RTC time:  3:02:30, date: 05/23/17
[    0.760526] NET: Registered protocol family 16
[    0.778351] cpuidle: using governor ladder
[    0.786145] cpuidle: using governor menu
[    0.786234] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.786235] ACPI: bus type PCI registered
[    0.786236] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.786356] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.786357] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[    0.786367] PCI: Using configuration type 1 for base access
[    0.810489] HugeTLB registered 1 GB page size, pre-allocated 0 pages
[    0.810489] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.810791] ACPI: Added _OSI(Module Device)
[    0.810791] ACPI: Added _OSI(Processor Device)
[    0.810791] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.810792] ACPI: Added _OSI(Processor Aggregator Device)
[    0.813087] ACPI: Executed 21 blocks of module-level executable AML code
[    0.832491] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    0.836090] ACPI: Dynamic OEM Table Load:
[    0.836094] ACPI: SSDT 0xFFFF8804625B8800 0006E4 (v02 PmRef  Cpu0Ist  00003000 INTL 20120913)
[    0.836605] ACPI: \_PR_.CPU0: _OSC native thermal LVT Acked
[    0.839290] ACPI: Dynamic OEM Table Load:
[    0.839293] ACPI: SSDT 0xFFFF880464BD0800 00037F (v02 PmRef  Cpu0Cst  00003001 INTL 20120913)
[    0.840448] ACPI: Dynamic OEM Table Load:
[    0.840452] ACPI: SSDT 0xFFFF880464042800 0005AA (v02 PmRef  ApIst    00003000 INTL 20120913)
[    0.841428] ACPI: Dynamic OEM Table Load:
[    0.841431] ACPI: SSDT 0xFFFF880464037600 000119 (v02 PmRef  ApCst    00003000 INTL 20120913)
[    0.849879] ACPI: Interpreter enabled
[    0.849905] ACPI: (supports S0 S3 S4 S5)
[    0.849905] ACPI: Using IOAPIC for interrupt routing
[    0.849949] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.853219] ACPI: Power Resource [PG00] (on)
[    0.853977] ACPI: Power Resource [PG01] (on)
[    0.854692] ACPI: Power Resource [PG02] (on)
[    0.858945] ACPI: Power Resource [WRST] (off)
[    0.859672] ACPI: Power Resource [WRST] (off)
[    0.860398] ACPI: Power Resource [WRST] (off)
[    0.861120] ACPI: Power Resource [WRST] (off)
[    0.861840] ACPI: Power Resource [WRST] (off)
[    0.862575] ACPI: Power Resource [WRST] (off)
[    0.863297] ACPI: Power Resource [WRST] (off)
[    0.864037] ACPI: Power Resource [WRST] (off)
[    0.864757] ACPI: Power Resource [WRST] (off)
[    0.865477] ACPI: Power Resource [WRST] (off)
[    0.866197] ACPI: Power Resource [WRST] (off)
[    0.866925] ACPI: Power Resource [WRST] (off)
[    0.867652] ACPI: Power Resource [WRST] (off)
[    0.868385] ACPI: Power Resource [WRST] (off)
[    0.869106] ACPI: Power Resource [WRST] (off)
[    0.869824] ACPI: Power Resource [WRST] (off)
[    0.870545] ACPI: Power Resource [WRST] (off)
[    0.871266] ACPI: Power Resource [WRST] (off)
[    0.871986] ACPI: Power Resource [WRST] (off)
[    0.872705] ACPI: Power Resource [WRST] (off)
[    0.887280] ACPI: Power Resource [FN00] (off)
[    0.887440] ACPI: Power Resource [FN01] (off)
[    0.887601] ACPI: Power Resource [FN02] (off)
[    0.887759] ACPI: Power Resource [FN03] (off)
[    0.887917] ACPI: Power Resource [FN04] (off)
[    0.889869] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe])
[    0.889874] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    0.889951] acpi PNP0A08:00: _OSC failed (AE_ERROR); disabling ASPM
[    0.891317] PCI host bridge to bus 0000:00
[    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
[    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
[    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
[    0.891331] pci 0000:00:00.0: [8086:191f] type 00 class 0x060000
[    0.891877] pci 0000:00:14.0: [8086:a12f] type 00 class 0x0c0330
[    0.891892] pci 0000:00:14.0: reg 0x10: [mem 0xdff30000-0xdff3ffff 64bit]
[    0.891948] pci 0000:00:14.0: PME# supported from D3hot D3cold
[    0.892169] pci 0000:00:14.0: System wakeup disabled by ACPI
[    0.892253] pci 0000:00:16.0: [8086:a13a] type 00 class 0x078000
[    0.892266] pci 0000:00:16.0: reg 0x10: [mem 0xdff4d000-0xdff4dfff 64bit]
[    0.892311] pci 0000:00:16.0: PME# supported from D3hot
[    0.892576] pci 0000:00:17.0: [8086:a102] type 00 class 0x010601
[    0.892587] pci 0000:00:17.0: reg 0x10: [mem 0xdff48000-0xdff49fff]
[    0.892592] pci 0000:00:17.0: reg 0x14: [mem 0xdff4c000-0xdff4c0ff]
[    0.892598] pci 0000:00:17.0: reg 0x18: [io  0xf050-0xf057]
[    0.892603] pci 0000:00:17.0: reg 0x1c: [io  0xf040-0xf043]
[    0.892609] pci 0000:00:17.0: reg 0x20: [io  0xf020-0xf03f]
[    0.892614] pci 0000:00:17.0: reg 0x24: [mem 0xdff4b000-0xdff4b7ff]
[    0.892647] pci 0000:00:17.0: PME# supported from D3hot
[    0.892910] pci 0000:00:1b.0: [8086:a167] type 01 class 0x060400
[    0.892959] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
[    0.893228] pci 0000:00:1b.0: System wakeup disabled by ACPI
[    0.893312] pci 0000:00:1c.0: [8086:a110] type 01 class 0x060400
[    0.893362] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    0.893630] pci 0000:00:1c.0: System wakeup disabled by ACPI
[    0.893694] pci 0000:00:1c.2: [8086:a112] type 01 class 0x060400
[    0.893743] pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold
[    0.894012] pci 0000:00:1c.2: System wakeup disabled by ACPI
[    0.894090] pci 0000:00:1d.0: [8086:a118] type 01 class 0x060400
[    0.894141] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[    0.894406] pci 0000:00:1d.0: System wakeup disabled by ACPI
[    0.894496] pci 0000:00:1f.0: [8086:a145] type 00 class 0x060100
[    0.894828] pci 0000:00:1f.2: [8086:a121] type 00 class 0x058000
[    0.894838] pci 0000:00:1f.2: reg 0x10: [mem 0xdff44000-0xdff47fff]
[    0.895124] pci 0000:00:1f.3: [8086:a170] type 00 class 0x040300
[    0.895143] pci 0000:00:1f.3: reg 0x10: [mem 0xdff40000-0xdff43fff 64bit]
[    0.895171] pci 0000:00:1f.3: reg 0x20: [mem 0xdff20000-0xdff2ffff 64bit]
[    0.895215] pci 0000:00:1f.3: PME# supported from D3hot D3cold
[    0.895487] pci 0000:00:1f.3: System wakeup disabled by ACPI
[    0.895555] pci 0000:00:1f.4: [8086:a123] type 00 class 0x0c0500
[    0.895601] pci 0000:00:1f.4: reg 0x10: [mem 0xdff4a000-0xdff4a0ff 64bit]
[    0.895670] pci 0000:00:1f.4: reg 0x20: [io  0xf000-0xf01f]
[    0.895986] pci 0000:00:1f.6: [8086:15b8] type 00 class 0x020000
[    0.896005] pci 0000:00:1f.6: reg 0x10: [mem 0xdff00000-0xdff1ffff]
[    0.896105] pci 0000:00:1f.6: PME# supported from D0 D3hot D3cold
[    0.896313] pci 0000:00:1f.6: System wakeup disabled by ACPI
[    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.896590] pci 0000:01:00.0: supports D1 D2
[    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
[    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
[    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
[    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
[    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
[    0.906695] pci 0000:02:00.0: [1b21:1242] type 00 class 0x0c0330
[    0.906715] pci 0000:02:00.0: reg 0x10: [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.906823] pci 0000:02:00.0: PME# supported from D3hot D3cold
[    0.906880] pci 0000:02:00.0: System wakeup disabled by ACPI
[    0.922505] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.922513] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.922672] pci 0000:03:00.0: [1b21:1080] type 01 class 0x060400
[    0.922788] pci 0000:03:00.0: supports D1 D2
[    0.922789] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.922830] pci 0000:03:00.0: System wakeup disabled by ACPI
[    0.938523] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.938689] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.938842] acpiphp: Slot [1] registered
[    0.938845] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.941676] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.941790] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.941903] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 10 11 12 14 *15)
[    0.942016] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.942128] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.942240] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.942352] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.942462] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
[    0.943304] ACPI: Enabled 6 GPEs in block 00 to 7F
[    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.943586] vgaarb: loaded
[    0.943586] vgaarb: bridge control possible 0000:01:00.0
[    0.943858] SCSI subsystem initialized
[    0.943913] libata version 3.00 loaded.
[    0.943946] ACPI: bus type USB registered
[    0.943974] usbcore: registered new interface driver usbfs
[    0.943989] usbcore: registered new interface driver hub
[    0.944011] usbcore: registered new device driver usb
[    0.944149] PCI: Using ACPI for IRQ routing
[    0.971976] PCI: pci_cache_line_size set to 64 bytes
[    0.972016] e820: reserve RAM buffer [mem 0x0009c800-0x0009ffff]
[    0.972018] e820: reserve RAM buffer [mem 0x7d439000-0x7fffffff]
[    0.972019] e820: reserve RAM buffer [mem 0x7d4c5000-0x7fffffff]
[    0.972020] e820: reserve RAM buffer [mem 0x858a8000-0x87ffffff]
[    0.972021] e820: reserve RAM buffer [mem 0x476800000-0x477ffffff]
[    0.972180] NetLabel: Initializing
[    0.972180] NetLabel:  domain hash size = 128
[    0.972181] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.972199] NetLabel:  unlabeled traffic allowed by default
[    0.972329] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    0.972332] hpet0: 8 comparators, 64-bit 24.000000 MHz counter
[    0.974373] clocksource: Switched to clocksource hpet
[    0.984338] VFS: Disk quotas dquot_6.6.0
[    0.984373] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.984505] AppArmor: AppArmor Filesystem Enabled
[    0.984561] pnp: PnP ACPI init
[    0.984848] system 00:00: [io  0x0290-0x029f] has been reserved
[    0.984850] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.985322] pnp 00:01: [dma 0 disabled]
[    0.985388] pnp 00:01: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.985626] system 00:02: [io  0x0680-0x069f] has been reserved
[    0.985627] system 00:02: [io  0xffff] has been reserved
[    0.985629] system 00:02: [io  0xffff] has been reserved
[    0.985630] system 00:02: [io  0xffff] has been reserved
[    0.985632] system 00:02: [io  0x1800-0x18fe] has been reserved
[    0.985633] system 00:02: [io  0x164e-0x164f] has been reserved
[    0.985635] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.985744] system 00:03: [io  0x0800-0x087f] has been reserved
[    0.985745] system 00:03: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.985785] pnp 00:04: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.985855] system 00:05: [io  0x1854-0x1857] has been reserved
[    0.985856] system 00:05: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
[    0.986242] system 00:06: [mem 0xfed10000-0xfed17fff] has been reserved
[    0.986244] system 00:06: [mem 0xfed18000-0xfed18fff] has been reserved
[    0.986245] system 00:06: [mem 0xfed19000-0xfed19fff] has been reserved
[    0.986247] system 00:06: [mem 0xe0000000-0xefffffff] has been reserved
[    0.986248] system 00:06: [mem 0xfed20000-0xfed3ffff] has been reserved
[    0.986250] system 00:06: [mem 0xfed90000-0xfed93fff] has been reserved
[    0.986251] system 00:06: [mem 0xfed45000-0xfed8ffff] has been reserved
[    0.986253] system 00:06: [mem 0xff000000-0xffffffff] has been reserved
[    0.986255] system 00:06: [mem 0xfee00000-0xfeefffff] could not be reserved
[    0.986256] system 00:06: [mem 0xdffc0000-0xdffdffff] has been reserved
[    0.986257] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.986332] system 00:07: [mem 0xfd000000-0xfdabffff] has been reserved
[    0.986334] system 00:07: [mem 0xfdad0000-0xfdadffff] has been reserved
[    0.986335] system 00:07: [mem 0xfdb00000-0xfdffffff] has been reserved
[    0.986337] system 00:07: [mem 0xfe000000-0xfe01ffff] could not be reserved
[    0.986338] system 00:07: [mem 0xfe036000-0xfe03bfff] has been reserved
[    0.986340] system 00:07: [mem 0xfe03d000-0xfe3fffff] has been reserved
[    0.986341] system 00:07: [mem 0xfe410000-0xfe7fffff] has been reserved
[    0.986342] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.988120] system 00:08: [mem 0xfdaf0000-0xfdafffff] has been reserved
[    0.988122] system 00:08: [mem 0xfdae0000-0xfdaeffff] has been reserved
[    0.988123] system 00:08: [mem 0xfdac0000-0xfdacffff] has been reserved
[    0.988125] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.989495] pnp: PnP ACPI: found 9 devices
[    0.997451] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.997454] pci_bus 0000:00: max bus depth: 2 pci_try_num: 3
[    0.997482] pci 0000:00:1d.0: BAR 14: assigned [mem 0x88800000-0x889fffff]
[    0.997485] pci 0000:00:1d.0: BAR 15: assigned [mem 0x88a00000-0x88bfffff 64bit pref]
[    0.997486] pci 0000:00:1d.0: BAR 13: assigned [io  0x2000-0x2fff]
[    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
[    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.997505] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997507] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.997512] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.997525] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.997531] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997532] pci 0000:00:1d.0:   bridge window [io  0x2000-0x2fff]
[    0.997534] pci 0000:00:1d.0:   bridge window [mem 0x88800000-0x889fffff]
[    0.997536] pci 0000:00:1d.0:   bridge window [mem 0x88a00000-0x88bfffff 64bit pref]
[    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
[    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
[    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] released
[    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64bit pref] released
[    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adfffff]
[    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
[    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
[    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
[    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff]
[    0.997613] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997616] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.997620] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.997633] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.997638] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997640] pci 0000:00:1d.0:   bridge window [io  0x2000-0x2fff]
[    0.997642] pci 0000:00:1d.0:   bridge window [mem 0x88800000-0x889fffff]
[    0.997644] pci 0000:00:1d.0:   bridge window [mem 0x88a00000-0x88bfffff 64bit pref]
[    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
[    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] released
[    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] released
[    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64bit pref] released
[    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 64bit pref] to [bus 01] add_size 48000000 add_align 8000000
[    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] to [bus 01] add_size 2200000 add_align 400000
[    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
[    0.997692] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0x5fffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0xa7ffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x022fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997694] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x044fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997695] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997696] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x004fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 64bit pref]
[    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000000 64bit pref]
[    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cffffff]
[    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88afffff]
[    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3fffff 64bit pref]
[    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abfffff]
[    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acfffff]
[    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aefffff 64bit pref]
[    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0fffff 64bit pref] (expanded by 0x200000)
[    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7fffff] (expanded by 0x2200000)
[    0.997719] pci 0000:01:00.0: res[7]=[mem size 0x00000000 64bit pref] res_to_dev_res add_size 40000000 min_align 0
[    0.997720] pci 0000:01:00.0: res[9]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 2000000 min_align 0
[    0.997721] pci 0000:01:00.0: res[12]=[mem size 0x00000000] res_to_dev_res add_size 2000000 min_align 0
[    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff 64bit pref]
[    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3fffff]
[    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff 64bit pref]
[    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff]
[    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff 64bit pref]
[    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff]
[    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7fffff]
[    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff]
[    0.997775] pci 0000:02:00.0: BAR 0: assigned [mem 0x8ac00000-0x8ac07fff 64bit]
[    0.997784] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997786] pci 0000:00:1c.0:   bridge window [mem 0x8ac00000-0x8acfffff]
[    0.997790] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.997803] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.997809] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997810] pci 0000:00:1d.0:   bridge window [io  0x2000-0x2fff]
[    0.997813] pci 0000:00:1d.0:   bridge window [mem 0x88800000-0x889fffff]
[    0.997815] pci 0000:00:1d.0:   bridge window [mem 0x8ad00000-0x8b0fffff 64bit pref]
[    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if you have problem, try booting with pci=realloc=off
[    0.997819] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.997820] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.997820] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.997821] pci_bus 0000:00: resource 7 [mem 0x88800000-0xdfffffff window]
[    0.997822] pci_bus 0000:00: resource 8 [mem 0xfd000000-0xfe7fffff window]
[    0.997822] pci_bus 0000:01: resource 0 [io  0xe000-0xefff]
[    0.997823] pci_bus 0000:01: resource 1 [mem 0x8b400000-0x8f7fffff]
[    0.997824] pci_bus 0000:02: resource 1 [mem 0x8ac00000-0x8acfffff]
[    0.997824] pci_bus 0000:05: resource 0 [io  0x2000-0x2fff]
[    0.997825] pci_bus 0000:05: resource 1 [mem 0x88800000-0x889fffff]
[    0.997825] pci_bus 0000:05: resource 2 [mem 0x8ad00000-0x8b0fffff 64bit pref]
[    0.998017] NET: Registered protocol family 2
[    0.998224] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.998344] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.998439] TCP: Hash tables configured (established 131072 bind 65536)
[    0.998460] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    0.998489] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    0.998579] NET: Registered protocol family 1
[    0.999248] pci 0000:01:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.999926] PCI: CLS 0 bytes, default 64
[    0.999972] Trying to unpack rootfs image as initramfs...
[    1.727335] Freeing initrd memory: 70180K (ffff88002f6de000 - ffff880033b67000)
[    1.727337] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.727339] software IO TLB [mem 0x818a8000-0x858a8000] (64MB) mapped at [ffff8800818a8000-ffff8800858a7fff]
[    1.727553] RAPL PMU: API unit is 2^-32 Joules, 5 fixed counters, 655360 ms ovfl timer
[    1.727553] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
[    1.727554] RAPL PMU: hw unit of domain package 2^-14 Joules
[    1.727554] RAPL PMU: hw unit of domain dram 2^-14 Joules
[    1.727554] RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules
[    1.727555] RAPL PMU: hw unit of domain psys 2^-14 Joules
[    1.728135] Scanning for low memory corruption every 60 seconds
[    1.728726] futex hash table entries: 2048 (order: 5, 131072 bytes)
[    1.728752] audit: initializing netlink subsys (disabled)
[    1.728769] audit: type=2000 audit(1495508550.692:1): initialized
[    1.729532] Initialise system trusted keyrings
[    1.729674] workingset: timestamp_bits=40 max_order=22 bucket_order=0
[    1.734226] fuse init (API version 7.26)
[    1.735699] Key type asymmetric registered
[    1.735699] Asymmetric key parser 'x509' registered
[    1.735764] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    1.735822] io scheduler noop registered
[    1.735823] io scheduler deadline registered (default)
[    1.735830] io scheduler cfq registered
[    1.736320] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    1.736328] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    1.736368] efifb: probing for efifb
[    1.736371] efifb: framebuffer at 0xa0000, using 64k, total 64k
[    1.736372] efifb: mode is 640x480x1, linelength=80, pages=1
[    1.736372] efifb: scrolling: redraw
[    1.736373] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.737120] Console: switching to colour frame buffer device 80x30
[    1.737742] fb0: EFI VGA frame buffer device
[    1.737750] intel_idle: MWAIT substates: 0x142120
[    1.737750] intel_idle: v0.4.1 model 0x5E
[    1.738495] intel_idle: lapic_timer_reliable_states 0xffffffff
[    1.738505] ipmi message handler version 39.2
[    1.738612] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input0
[    1.738615] ACPI: Sleep Button [SLPB]
[    1.738678] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[    1.738680] ACPI: Power Button [PWRB]
[    1.738741] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[    1.738743] ACPI: Power Button [PWRF]
[    1.740527] thermal LNXTHERM:00: registered as thermal_zone0
[    1.740528] ACPI: Thermal Zone [TZ00] (28 C)
[    1.740874] thermal LNXTHERM:01: registered as thermal_zone1
[    1.740875] ACPI: Thermal Zone [TZ01] (30 C)
[    1.740903] GHES: HEST is not enabled!
[    1.741044] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    1.761587] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.764644] Linux agpgart interface v0.103
[    1.770629] brd: module loaded
[    1.773654] loop: module loaded
[    1.774080] libphy: Fixed MDIO Bus: probed
[    1.774490] tun: Universal TUN/TAP device driver, 1.6
[    1.774490] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    1.774586] PPP generic driver version 2.4.2
[    1.774670] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.774673] ehci-pci: EHCI PCI platform driver
[    1.774685] ehci-platform: EHCI generic platform driver
[    1.774696] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.774697] ohci-pci: OHCI PCI platform driver
[    1.774708] ohci-platform: OHCI generic platform driver
[    1.774716] uhci_hcd: USB Universal Host Controller Interface driver
[    1.774901] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.774907] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[    1.775990] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x00109810
[    1.775993] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
[    1.776103] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    1.776104] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.776105] usb usb1: Product: xHCI Host Controller
[    1.776105] usb usb1: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.776106] usb usb1: SerialNumber: 0000:00:14.0
[    1.776299] hub 1-0:1.0: USB hub found
[    1.776316] hub 1-0:1.0: 16 ports detected
[    1.791067] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.791071] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[    1.791114] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[    1.791114] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.791115] usb usb2: Product: xHCI Host Controller
[    1.791116] usb usb2: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.791116] usb usb2: SerialNumber: 0000:00:14.0
[    1.791311] hub 2-0:1.0: USB hub found
[    1.791324] hub 2-0:1.0: 10 ports detected
[    1.800679] xhci_hcd 0000:02:00.0: xHCI Host Controller
[    1.800684] xhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 3
[    1.859445] xhci_hcd 0000:02:00.0: hcc params 0x0200eec1 hci version 0x110 quirks 0x00000010
[    1.859790] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
[    1.859791] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.859792] usb usb3: Product: xHCI Host Controller
[    1.859792] usb usb3: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.859793] usb usb3: SerialNumber: 0000:02:00.0
[    1.859984] hub 3-0:1.0: USB hub found
[    1.859992] hub 3-0:1.0: 2 ports detected
[    1.860124] xhci_hcd 0000:02:00.0: xHCI Host Controller
[    1.860127] xhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 4
[    1.860157] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    1.860183] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003
[    1.860183] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.860184] usb usb4: Product: xHCI Host Controller
[    1.860184] usb usb4: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.860185] usb usb4: SerialNumber: 0000:02:00.0
[    1.860373] hub 4-0:1.0: USB hub found
[    1.860381] hub 4-0:1.0: 2 ports detected
[    1.860546] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.863868] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.863871] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.864131] mousedev: PS/2 mouse device common for all mice
[    1.864487] rtc_cmos 00:04: RTC can wake from S4
[    1.864922] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
[    1.865003] rtc_cmos 00:04: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
[    1.865077] device-mapper: uevent: version 1.0.3
[    1.865220] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
[    1.865222] intel_pstate: Intel P-state driver initializing
[    1.866102] intel_pstate: HWP enabled
[    1.866565] ledtrig-cpu: registered to indicate activity on CPUs
[    1.866850] NET: Registered protocol family 10
[    1.867238] NET: Registered protocol family 17
[    1.867244] Key type dns_resolver registered
[    1.867739] microcode: sig=0x506e3, pf=0x2, revision=0x33
[    1.868149] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    1.868548] registered taskstats version 1
[    1.868549] Loading compiled-in X.509 certificates
[    1.870972] Loaded X.509 cert 'Build time autogenerated kernel key: 425e568c95731e69b073bb62214af700479a67a7'
[    1.871139] kmemleak: Kernel memory leak detector initialized
[    1.871194] kmemleak: Automatic memory scanning thread started
[    1.873242] Key type trusted registered
[    1.876986] Key type encrypted registered
[    1.876988] AppArmor: AppArmor sha1 policy hashing enabled
[    1.876989] ima: No TPM chip found, activating TPM-bypass!
[    1.877006] evm: HMAC attrs: 0x1
[    1.878412]   Magic number: 1:0:8
[    1.878526] memory memory52: hash matches
[    1.878693] rtc_cmos 00:04: setting system clock to 2017-05-23 03:02:31 UTC (1495508551)
[    1.878887] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    1.878887] EDD information not available.
[    1.879062] PM: Hibernation image not present or could not be loaded.
[    1.879872] Freeing unused kernel memory: 2220K (ffffffff81d6b000 - ffffffff81f96000)
[    1.879872] Write protecting the kernel read-only data: 12288k
[    1.880062] Freeing unused kernel memory: 84K (ffff8800017eb000 - ffff880001800000)
[    1.880656] Freeing unused kernel memory: 428K (ffff880001b95000 - ffff880001c00000)
[    1.889484] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.889540] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.889547] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.890513] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.890535] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.891569] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.891608] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.892119] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.892156] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.892194] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.921124] hid: module verification failed: signature and/or required key missing - tainting kernel
[    1.921710] hidraw: raw HID events driver (C) Jiri Kosina
[    1.927701] wmi: Mapper loaded
[    1.942001] ahci 0000:00:17.0: version 3.0
[    1.942573] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[    1.942574] ahci 0000:00:17.0: flags: 64bit ncq sntf led clo only pio slum part ems deso sadm sds apst 
[    1.944060] pps_core: LinuxPPS API ver. 1 registered
[    1.944060] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.944471] PTP clock support registered
[    1.945229] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.945229] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.008270] scsi host0: ahci
[    2.008680] scsi host1: ahci
[    2.009081] scsi host2: ahci
[    2.009535] scsi host3: ahci
[    2.009917] scsi host4: ahci
[    2.010244] scsi host5: ahci
[    2.010364] ata1: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b100 irq 129
[    2.010368] ata2: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b180 irq 129
[    2.010369] ata3: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b200 irq 129
[    2.010370] ata4: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b280 irq 129
[    2.010371] ata5: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b300 irq 129
[    2.010385] ata6: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b380 irq 129
[    2.010800] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    2.118466] usb 1-10: new full-speed USB device number 2 using xhci_hcd
[    2.261613] usb 1-10: New USB device found, idVendor=046d, idProduct=c534
[    2.261614] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.261615] usb 1-10: Product: USB Receiver
[    2.261615] usb 1-10: Manufacturer: Logitech
[    2.266734] usbcore: registered new interface driver usbhid
[    2.266735] usbhid: USB HID core driver
[    2.267863] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C534.0001/input/input6
[    2.282262] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    2.326534] CPU: 7 PID: 212 Comm: scsi_eh_4 Tainted: G            E   4.9.0-custom #2
[    2.326535] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326535]  ffffc90002067ae8 ffffffff813c2d3f ffff880452acde80 ffff880452ace2b0
[    2.326537]  ffffc90002067b10 ffffffff815a7226 ffffc90002067b10 0000000000000286
[    2.326538]  ffffc90000029300 ffffc90002067b30 ffffffffa003f634 ffff880452acde80
[    2.326539] Call Trace:
[    2.326542]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.326544]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.326546]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.326547]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.326549]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326550]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326551]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326552]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326553]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326554]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326555]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.326556]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326557]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326558]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326559]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.326560]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.326562]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.326564]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.326565]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.326566]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.326567]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.326569]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326570]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.326571]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326572]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326573]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.326574]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326576]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.326582] ata5: SATA link down (SStatus 4 SControl 300)
[    2.326612] CPU: 2 PID: 204 Comm: scsi_eh_0 Tainted: G            E   4.9.0-custom #2
[    2.326613] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326613]  ffffc90001febae8 ffffffff813c2d3f ffff880452ab9e80 ffff880452aba2b0
[    2.326615]  ffffc90001febb10 ffffffff815a7226 ffffc90001febb10 0000000000000286
[    2.326616]  ffffc90000029100 ffffc90001febb30 ffffffffa003f634 ffff880452ab9e80
[    2.326617] Call Trace:
[    2.326619]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.326620]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.326622]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.326623]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.326624]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326625]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326626]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326627]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326628]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326629]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326630]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.326631]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326632]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326633]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326634]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.326635]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.326636]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.326638]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.326639]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.326640]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.326641]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.326642]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326643]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.326644]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326645]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326646]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.326647]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326648]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.326662] ata1: SATA link down (SStatus 4 SControl 300)
[    2.326902] CPU: 4 PID: 206 Comm: scsi_eh_1 Tainted: G            E   4.9.0-custom #2
[    2.326903] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326903]  ffffc9000202bae8 ffffffff813c2d3f ffff880452abde80 ffff880452abe2b0
[    2.326905]  ffffc9000202bb10 ffffffff815a7226 ffffc9000202bb10 0000000000000286
[    2.326907]  ffffc90000029180 ffffc9000202bb30 ffffffffa003f634 ffff880452abde80
[    2.326908] Call Trace:
[    2.326911]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.326912]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.326914]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.326916]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.326930]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326932]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326934]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326935]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326937]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326938]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326940]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.326941]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326943]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326944]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326945]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.326946]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.326948]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.326949]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.326951]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.326952]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.326953]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.326954]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326955]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.326956]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326957]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326958]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.326959]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326960]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.326973] ata2: SATA link down (SStatus 4 SControl 300)
[    2.326991] hid-generic 0003:046D:C534.0001: input,hidraw0: USB HID v1.11 Keyboard [Logitech USB Receiver] on usb-0000:00:14.0-10/input0
[    2.326993] CPU: 5 PID: 214 Comm: scsi_eh_5 Tainted: G            E   4.9.0-custom #2
[    2.326994] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326994]  ffffc900020efae8 ffffffff813c2d3f ffff880452ad1e80 ffff880452ad22b0
[    2.326996]  ffffc900020efb10 ffffffff815a7226 ffffc900020efb10 0000000000000286
[    2.326997]  ffffc90000029380 ffffc900020efb30 ffffffffa003f634 ffff880452ad1e80
[    2.326998] Call Trace:
[    2.327001]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.327002]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.327003]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.327004]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.327006]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.327008]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.327010]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327011]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327013]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327014]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.327015]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.327017]  [<ffffffff8146d8e8>] ? acpi_ds_exec_end_op+0x2b0/0x3ec
[    2.327018]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.327019]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327020]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.327021]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.327023]  [<ffffffff816013f7>] ? usb_acpi_find_companion+0x177/0x210
[    2.327025]  [<ffffffff8145e255>] ? acpi_platform_notify+0x3c/0xa7
[    2.327026]  [<ffffffff8152e448>] ? device_add+0x1a8/0x5f0
[    2.327027]  [<ffffffff8152e8aa>] ? device_register+0x1a/0x20
[    2.327028]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.327029]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.327031]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.327032]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.327033]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.327034]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.327035]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.327037]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.327038]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.327039]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.327040]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.327041]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.327042]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.327045] ata6: SATA link down (SStatus 4 SControl 300)
[    2.327080] CPU: 1 PID: 210 Comm: scsi_eh_3 Tainted: G            E   4.9.0-custom #2
[    2.327080] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.327081]  ffffc9000204bae8 ffffffff813c2d3f ffff880452ac5e80 ffff880452ac62b0
[    2.327082]  ffffc9000204bb10 ffffffff815a7226 ffffc9000204bb10 0000000000000286
[    2.327084]  ffffc90000029280 ffffc9000204bb30 ffffffffa003f634 ffff880452ac5e80
[    2.327085] Call Trace:
[    2.327086]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.327087]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.327088]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.327090]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.327091]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.327092]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.327093]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327094]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327095]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327096]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.327097]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.327098]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.327099]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327100]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.327100]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.327101]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.327103]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.327104]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.327105]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.327106]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.327107]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.327108]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.327109]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.327110]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.327111]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.327112]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.327113]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.327114]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.327123] ata4: SATA link down (SStatus 4 SControl 300)
[    2.327144] CPU: 2 PID: 208 Comm: scsi_eh_2 Tainted: G            E   4.9.0-custom #2
[    2.327145] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.327145]  ffffc9000203bae8 ffffffff813c2d3f ffff880452ac1e80 ffff880452ac22b0
[    2.327147]  ffffc9000203bb10 ffffffff815a7226 ffffc9000203bb10 0000000000000286
[    2.327148]  ffffc90000029200 ffffc9000203bb30 ffffffffa003f634 ffff880452ac1e80
[    2.327149] Call Trace:
[    2.327151]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.327152]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.327153]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.327154]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.327156]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.327157]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.327158]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327159]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327160]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327161]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.327162]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.327163]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.327164]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.327165]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.327166]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.327167]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.327183]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.327185]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.327186]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.327187]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.327188]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.327189]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.327190]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.327191]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.327192]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.327192] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.1/0003:046D:C534.0002/input/input7
[    2.327193]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.327194]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.327196]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.327206] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.328461] ata3.00: ATA-8: ST3500413AS, JC45, max UDMA/133
[    2.328462] ata3.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32)
[    2.329997] ata3.00: configured for UDMA/133
[    2.330447] scsi 2:0:0:0: Direct-Access     ATA      ST3500413AS      JC45 PQ: 0 ANSI: 5
[    2.370687] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 30:5a:3a:e0:7f:4d
[    2.370688] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    2.370774] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: FFFFFF-0FF
[    2.371429] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
[    2.387154] hid-generic 0003:046D:C534.0002: input,hiddev0,hidraw1: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-0000:00:14.0-10/input1
[    2.411000] sd 2:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/466 GiB)
[    2.411006] sd 2:0:0:0: Attached scsi generic sg0 type 0
[    2.411463] sd 2:0:0:0: [sda] Write Protect is off
[    2.411464] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.411601] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.443628]  sda: sda1 sda2 < sda5 >
[    2.445142] sd 2:0:0:0: [sda] Attached SCSI disk
[    2.750516] tsc: Refined TSC clocksource calibration: 3407.998 MHz
[    2.750522] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x311fd208cfc, max_idle_ns: 440795283699 ns
[    2.781745] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[    2.835964] random: fast init done
[    3.717395] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[    3.717527] systemd[1]: Detected architecture x86-64.
[    3.724157] systemd[1]: Set hostname to <jenkins-MS-7984>.
[    3.774725] clocksource: Switched to clocksource tsc
[    4.332780] systemd[1]: Created slice User and Session Slice.
[    4.332838] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    4.332855] systemd[1]: Listening on fsck to fsckd communication Socket.
[    4.332860] systemd[1]: Reached target User and Group Name Lookups.
[    4.332880] systemd[1]: Listening on Syslog Socket.
[    4.332895] systemd[1]: Listening on udev Kernel Socket.
[    4.332917] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
[    4.999616] random: crng init done
[    5.135776] lp: driver loaded but no devices found
[    5.137860] ppdev: user-space parallel port driver
[    5.261416] RPC: Registered named UNIX socket transport module.
[    5.261416] RPC: Registered udp transport module.
[    5.261416] RPC: Registered tcp transport module.
[    5.261417] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    5.494378] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    6.005572] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro
[    6.329563] systemd-journald[293]: Received request to flush runtime journal from PID 1
[    6.524603] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    7.177214] mei_me 0000:00:16.0: enabling device (0000 -> 0002)
[    7.247634] AVX2 version of gcm_enc/dec engaged.
[    7.247635] AES CTR mode by8 optimization enabled
[    7.512754] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC892: line_outs=3 (0x14/0x15/0x16/0x0/0x0) type:line
[    7.512755] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    7.512756] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    7.512756] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    7.512757] snd_hda_codec_realtek hdaudioC0D0:    dig-out=0x11/0x1e
[    7.512757] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    7.512758] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[    7.512759] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[    7.512759] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a
[    7.524838] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input8
[    7.524935] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9
[    7.525031] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
[    7.525125] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
[    7.525225] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1f.3/sound/card0/input12
[    7.525334] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13
[    7.525441] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input14
[    7.966471] intel_rapl: Found RAPL domain package
[    7.966473] intel_rapl: Found RAPL domain core
[    7.966475] intel_rapl: Found RAPL domain dram
[    8.742764] audit: type=1400 audit(1495508558.360:2): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine" pid=696 comm="apparmor_parser"
[    8.742771] audit: type=1400 audit(1495508558.360:3): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=696 comm="apparmor_parser"
[    8.746550] audit: type=1400 audit(1495508558.364:4): apparmor="STATUS" operation="profile_load" name="/usr/sbin/ippusbxd" pid=699 comm="apparmor_parser"
[    8.762212] audit: type=1400 audit(1495508558.376:5): apparmor="STATUS" operation="profile_load" name="/usr/sbin/cups-browsed" pid=697 comm="apparmor_parser"
[    8.762734] audit: type=1400 audit(1495508558.380:6): apparmor="STATUS" operation="profile_load" name="/sbin/dhclient" pid=692 comm="apparmor_parser"
[    8.762741] audit: type=1400 audit(1495508558.380:7): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=692 comm="apparmor_parser"
[    8.762746] audit: type=1400 audit(1495508558.380:8): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=692 comm="apparmor_parser"
[    8.762751] audit: type=1400 audit(1495508558.380:9): apparmor="STATUS" operation="profile_load" name="/usr/lib/connman/scripts/dhclient-script" pid=692 comm="apparmor_parser"
[    8.810797] audit: type=1400 audit(1495508558.428:10): apparmor="STATUS" operation="profile_load" name="/usr/sbin/tcpdump" pid=701 comm="apparmor_parser"
[    8.875653] audit: type=1400 audit(1495508558.492:11): apparmor="STATUS" operation="profile_load" name="/usr/lib/lightdm/lightdm-guest-session" pid=691 comm="apparmor_parser"
[    9.312053] asus_wmi: ASUS WMI generic driver loaded
[    9.316050] asus_wmi: Initialization: 0x0
[    9.316117] asus_wmi: BIOS WMI version: 0.9
[    9.316246] asus_wmi: SFUN value: 0x0
[    9.317314] input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input15
[    9.317597] asus_wmi: Number of fans: 1
[    9.889318] Adding 8348668k swap on /dev/sda5.  Priority:-1 extents:1 across:8348668k FS
[   14.210964] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[   14.237443] NFSD: starting 90-second grace period (net ffffffff81d06400)
[   15.024108] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   15.238703] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   18.062124] e1000e: enp0s31f6 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx
[   18.062126] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[   18.062170] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready

=========================================================================================
00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 07)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31)
00:1b.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #17 (rev f1)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #1 (rev f1)
00:1c.2 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #3 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V (rev 31)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
02:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)

=====================================================================================
root@jenkins-MS-7984:~# lspci -s 01:00.0 -vvv
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at <ignored> (64-bit, prefetchable)
	Region 2: Memory at 8b400000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at e000 [size=256]
	Region 5: Memory at 8b600000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [200 v1] #15
	Capabilities: [270 v1] #19
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] #13
	Capabilities: [2d0 v1] #1b
	Capabilities: [328 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [330 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 256, stride: 1, Device ID: 692f
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
		Region 2: Memory at 000000008d800000 (64-bit, prefetchable)
		Region 5: Memory at 8b800000 (32-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [400 v1] Vendor Specific Information: ID=0002 Rev=1 Len=070 <?>
	Kernel modules: amdgpu

==========================================================================================================


[-- Attachment #3: 2.txt --]
[-- Type: text/plain, Size: 175475 bytes --]

[    0.000000] Linux version 4.9.0-custom (root@rexzhu) (gcc version 4.8.2 (GCC) ) #2 SMP Fri Mar 31 12:52:40 CST 2017
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-custom root=UUID=900c4a92-9f8e-44be-86bc-9673dd3b0d78 ro console=ttyS0,115200,8N1 console=tty0 no_console_suspend pci=earlydump quiet splash vt.handoff=7
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009c7ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009c800-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007d438fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007d439000-0x000000007d439fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000007d43a000-0x000000007d483fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007d484000-0x000000007d4c4fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007d4c5000-0x000000007db49fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007db4a000-0x00000000858a7fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000858a8000-0x0000000086db6fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000086db7000-0x0000000086deffff] ACPI data
[    0.000000] BIOS-e820: [mem 0x0000000086df0000-0x000000008774bfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000008774c000-0x0000000087ffefff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000087fff000-0x0000000087ffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000880fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000004767fffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] pci 0000:00:00.0 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  1f
[    0.000000]  19
[    0.000000]  06
[    0.000000]  00
[    0.000000]  90
[    0.000000]  20
[    0.000000]  07
[    0.000000]  00
[    0.000000]  00
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  e0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  01
[    0.000000]  90
[    0.000000]  d1
[    0.000000]  fe
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  d1
[    0.000000]  fe
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  21
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  40
[    0.000000]  88
[    0.000000] 
                 60:
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  e0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  80
[    0.000000]  d1
[    0.000000]  fe
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  ff
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0c
[    0.000000]  00
[    0.000000]  ff
[    0.000000]  7f
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  10
[    0.000000]  11
[    0.000000]  11
[    0.000000]  11
[    0.000000]  11
[    0.000000]  11
[    0.000000]  11
[    0.000000]  00
[    0.000000]  1a
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  ff
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  70
[    0.000000]  76
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  80
[    0.000000]  76
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  01
[    0.000000]  00
[    0.000000]  80
[    0.000000]  88
[    0.000000]  01
[    0.000000]  00
[    0.000000]  80
[    0.000000]  88
[    0.000000]  01
[    0.000000]  00
[    0.000000]  40
[    0.000000]  88
[    0.000000]  01
[    0.000000]  00
[    0.000000]  80
[    0.000000]  88
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  09
[    0.000000]  00
[    0.000000]  10
[    0.000000]  01
[    0.000000]  7d
[    0.000000]  26
[    0.000000]  01
[    0.000000]  62
[    0.000000]  c8
[    0.000000]  00
[    0.000000]  04
[    0.000000]  96
[    0.000000]  00
[    0.000000]  00
[    0.000000]  07
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  c8
[    0.000000]  0f
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:00:14.0 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  2f
[    0.000000]  a1
[    0.000000]  06
[    0.000000]  00
[    0.000000]  90
[    0.000000]  02
[    0.000000]  31
[    0.000000]  30
[    0.000000]  03
[    0.000000]  0c
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  04
[    0.000000]  00
[    0.000000]  f3
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  70
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  fd
[    0.000000]  01
[    0.000000]  34
[    0.000000]  80
[    0.000000]  88
[    0.000000]  c6
[    0.000000]  0f
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  5f
[    0.000000]  6e
[    0.000000]  ce
[    0.000000]  0f
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  30
[    0.000000]  60
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  01
[    0.000000]  80
[    0.000000]  c2
[    0.000000]  c1
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  05
[    0.000000]  00
[    0.000000]  86
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  09
[    0.000000]  00
[    0.000000]  14
[    0.000000]  f0
[    0.000000]  10
[    0.000000]  00
[    0.000000]  40
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  c1
[    0.000000]  0a
[    0.000000]  08
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  00
[    0.000000]  08
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  18
[    0.000000]  00
[    0.000000]  00
[    0.000000]  8f
[    0.000000]  40
[    0.000000]  02
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  04
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  80
[    0.000000]  c0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  30
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  02
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  0c
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  02
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  28
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  18
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  c0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:00:16.0 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  3a
[    0.000000]  a1
[    0.000000]  00
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  31
[    0.000000]  00
[    0.000000]  80
[    0.000000]  07
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  04
[    0.000000]  d0
[    0.000000]  f4
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  50
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  45
[    0.000000]  02
[    0.000000]  00
[    0.000000]  90
[    0.000000]  10
[    0.000000]  00
[    0.000000]  01
[    0.000000]  80
[    0.000000]  06
[    0.000000]  05
[    0.000000]  f2
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  01
[    0.000000]  8c
[    0.000000]  03
[    0.000000]  40
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000] 
                 70:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  05
[    0.000000]  00
[    0.000000]  80
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:00:17.0 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  02
[    0.000000]  a1
[    0.000000]  07
[    0.000000]  00
[    0.000000]  b0
[    0.000000]  02
[    0.000000]  31
[    0.000000]  01
[    0.000000]  06
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  80
[    0.000000]  f4
[    0.000000]  df
[    0.000000]  00
[    0.000000]  c0
[    0.000000]  f4
[    0.000000]  df
[    0.000000]  51
[    0.000000]  f0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  41
[    0.000000]  f0
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  21
[    0.000000]  f0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  b0
[    0.000000]  f4
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0a
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  01
[    0.000000]  a8
[    0.000000]  03
[    0.000000]  40
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  05
[    0.000000]  70
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  fb
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  30
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000] 
                 a0:
[    0.000000]  d4
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  08
[    0.000000]  11
[    0.000000]  1e
[    0.000000]  2c
[    0.000000]  12
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  48
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  11
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:00:1b.0 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  67
[    0.000000]  a1
[    0.000000]  07
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  f1
[    0.000000]  00
[    0.000000]  04
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  81
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  01
[    0.000000]  00
[    0.000000]  e0
[    0.000000]  e0
[    0.000000]  00
[    0.000000]  20
[    0.000000] 
                 20:
[    0.000000]  e0
[    0.000000]  df
[    0.000000]  e0
[    0.000000]  df
[    0.000000]  01
[    0.000000]  c0
[    0.000000]  11
[    0.000000]  d0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  18
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  10
[    0.000000]  80
[    0.000000]  42
[    0.000000]  01
[    0.000000]  01
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  20
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  23
[    0.000000]  40
[    0.000000]  72
[    0.000000]  11
[    0.000000] 
                 50:
[    0.000000]  40
[    0.000000]  00
[    0.000000]  23
[    0.000000]  70
[    0.000000]  00
[    0.000000]  fd
[    0.000000]  a4
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  37
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0e
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  03
[    0.000000]  00
[    0.000000]  1f
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  05
[    0.000000]  90
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  0d
[    0.000000]  a0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  01
[    0.000000]  00
[    0.000000]  03
[    0.000000]  c8
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  11
[    0.000000]  10
[    0.000000]  00
[    0.000000]  07
[    0.000000]  42
[    0.000000]  18
[    0.000000]  00
[    0.000000]  00
[    0.000000]  08
[    0.000000]  00
[    0.000000]  1e
[    0.000000]  8b
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  f7
[    0.000000]  f3
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  06
[    0.000000]  80
[    0.000000]  12
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  50
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  03
[    0.000000]  00
[    0.000000]  40
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  80
[    0.000000]  00
[    0.000000]  01

[    0.000000] pci 0000:00:1c.0 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  10
[    0.000000]  a1
[    0.000000]  07
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  f1
[    0.000000]  00
[    0.000000]  04
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  81
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  02
[    0.000000]  02
[    0.000000]  00
[    0.000000]  f0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  20
[    0.000000] 
                 20:
[    0.000000]  d0
[    0.000000]  df
[    0.000000]  d0
[    0.000000]  df
[    0.000000]  f1
[    0.000000]  ff
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  10
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  10
[    0.000000]  80
[    0.000000]  42
[    0.000000]  01
[    0.000000]  01
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  20
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  23
[    0.000000]  40
[    0.000000]  72
[    0.000000]  01
[    0.000000] 
                 50:
[    0.000000]  40
[    0.000000]  00
[    0.000000]  22
[    0.000000]  70
[    0.000000]  00
[    0.000000]  fd
[    0.000000]  24
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  37
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0e
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  05
[    0.000000]  90
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  0d
[    0.000000]  a0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  01
[    0.000000]  00
[    0.000000]  03
[    0.000000]  c8
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  11
[    0.000000]  10
[    0.000000]  00
[    0.000000]  07
[    0.000000]  42
[    0.000000]  18
[    0.000000]  00
[    0.000000]  00
[    0.000000]  08
[    0.000000]  00
[    0.000000]  1e
[    0.000000]  8b
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  f7
[    0.000000]  f3
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  06
[    0.000000]  80
[    0.000000]  12
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  50
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  03
[    0.000000]  00
[    0.000000]  40
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  40
[    0.000000]  00
[    0.000000]  01

[    0.000000] pci 0000:00:1c.2 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  12
[    0.000000]  a1
[    0.000000]  07
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  f1
[    0.000000]  00
[    0.000000]  04
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  81
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  03
[    0.000000]  04
[    0.000000]  20
[    0.000000]  f0
[    0.000000]  00
[    0.000000]  80
[    0.000000]  22
[    0.000000] 
                 20:
[    0.000000]  f0
[    0.000000]  ff
[    0.000000]  00
[    0.000000]  00
[    0.000000]  f1
[    0.000000]  ff
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  ff
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  10
[    0.000000]  80
[    0.000000]  42
[    0.000000]  01
[    0.000000]  01
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  13
[    0.000000]  40
[    0.000000]  72
[    0.000000]  03
[    0.000000] 
                 50:
[    0.000000]  40
[    0.000000]  00
[    0.000000]  11
[    0.000000]  70
[    0.000000]  00
[    0.000000]  b2
[    0.000000]  34
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  37
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0e
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  03
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  05
[    0.000000]  90
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  0d
[    0.000000]  a0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  01
[    0.000000]  00
[    0.000000]  03
[    0.000000]  c8
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  01
[    0.000000]  10
[    0.000000]  00
[    0.000000]  07
[    0.000000]  42
[    0.000000]  18
[    0.000000]  00
[    0.000000]  00
[    0.000000]  08
[    0.000000]  00
[    0.000000]  1e
[    0.000000]  8f
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  03
[    0.000000]  e3
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  06
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  50
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  4c
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  03

[    0.000000] pci 0000:00:1d.0 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  18
[    0.000000]  a1
[    0.000000]  07
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  f1
[    0.000000]  00
[    0.000000]  04
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  81
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  05
[    0.000000]  05
[    0.000000]  00
[    0.000000]  f0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  20
[    0.000000] 
                 20:
[    0.000000]  f0
[    0.000000]  ff
[    0.000000]  00
[    0.000000]  00
[    0.000000]  f1
[    0.000000]  ff
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  10
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  10
[    0.000000]  80
[    0.000000]  42
[    0.000000]  01
[    0.000000]  01
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  43
[    0.000000]  4c
[    0.000000]  71
[    0.000000]  09
[    0.000000] 
                 50:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  10
[    0.000000]  60
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  37
[    0.000000]  08
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0e
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  03
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  05
[    0.000000]  90
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  0d
[    0.000000]  a0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  01
[    0.000000]  00
[    0.000000]  03
[    0.000000]  c8
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  11
[    0.000000]  00
[    0.000000]  00
[    0.000000]  07
[    0.000000]  00
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  02
[    0.000000]  00
[    0.000000]  91
[    0.000000]  09
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  f7
[    0.000000]  b3
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  04
[    0.000000]  80
[    0.000000]  02
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  50
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  f3
[    0.000000]  00
[    0.000000]  40
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  c0
[    0.000000]  00
[    0.000000]  01

[    0.000000] pci 0000:00:1f.0 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  45
[    0.000000]  a1
[    0.000000]  07
[    0.000000]  00
[    0.000000]  00
[    0.000000]  02
[    0.000000]  31
[    0.000000]  00
[    0.000000]  01
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  d0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  70
[    0.000000]  00
[    0.000000]  03
[    0.000000]  3c
[    0.000000]  91
[    0.000000]  02
[    0.000000]  0c
[    0.000000]  00
[    0.000000]  01
[    0.000000]  0a
[    0.000000]  7c
[    0.000000]  00
[    0.000000]  21
[    0.000000]  ff
[    0.000000]  0c
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  69
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  2b
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  33
[    0.000000]  22
[    0.000000]  11
[    0.000000]  00
[    0.000000]  67
[    0.000000]  45
[    0.000000]  00
[    0.000000]  00
[    0.000000]  cf
[    0.000000]  ff
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  c1
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:00:1f.2 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  21
[    0.000000]  a1
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  31
[    0.000000]  00
[    0.000000]  80
[    0.000000]  05
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  40
[    0.000000]  f4
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  01
[    0.000000]  18
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  fe
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  f8
[    0.000000]  36
[    0.000000]  80
[    0.000000]  98
[    0.000000]  49
[    0.000000]  38
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  46
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:00:1f.3 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  70
[    0.000000]  a1
[    0.000000]  06
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  31
[    0.000000]  00
[    0.000000]  03
[    0.000000]  04
[    0.000000]  00
[    0.000000]  20
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  04
[    0.000000]  00
[    0.000000]  f4
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  04
[    0.000000]  00
[    0.000000]  f2
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  98
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  50
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  ff
[    0.000000]  0d
[    0.000000]  3b
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  01
[    0.000000]  60
[    0.000000]  43
[    0.000000]  c0
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  05
[    0.000000]  00
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  10
[    0.000000]  00
[    0.000000]  91
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  28
[    0.000000]  10
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  08
[    0.000000]  06
[    0.000000]  02
[    0.000000]  01
[    0.000000]  00
[    0.000000]  70
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  0c
[    0.000000]  a5
[    0.000000]  82
[    0.000000]  10
[    0.000000]  00
[    0.000000]  03
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  00
[    0.000000]  0c
[    0.000000]  b5
[    0.000000]  02
[    0.000000]  10
[    0.000000]  00
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:00:1f.4 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  23
[    0.000000]  a1
[    0.000000]  03
[    0.000000]  00
[    0.000000]  80
[    0.000000]  02
[    0.000000]  31
[    0.000000]  00
[    0.000000]  05
[    0.000000]  0c
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  04
[    0.000000]  a0
[    0.000000]  f4
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  01
[    0.000000]  f0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  94
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  11
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  01
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  04
[    0.000000]  05
[    0.000000]  05
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  06
[    0.000000]  0f
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  24
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:00:1f.5 config space:
[    0.000000] 
                 00:
[    0.000000]  ff
[    0.000000]  ff
[    0.000000]  ff
[    0.000000]  ff
[    0.000000]  02
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  31
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  fe
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  02
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  cf
[    0.000000]  ff
[    0.000000]  00
[    0.000000]  00
[    0.000000]  88
[    0.000000]  0a
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  b3
[    0.000000]  0f
[    0.000000]  33
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:00:1f.6 config space:
[    0.000000] 
                 00:
[    0.000000]  86
[    0.000000]  80
[    0.000000]  b8
[    0.000000]  15
[    0.000000]  06
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  31
[    0.000000]  00
[    0.000000]  00
[    0.000000]  02
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  f0
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  72
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  c8
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  28
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0d
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  40
[    0.000000]  02
[    0.000000]  18
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  03
[    0.000000]  10
[    0.000000]  03
[    0.000000]  10
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  d0
[    0.000000]  23
[    0.000000]  c8
[    0.000000]  08
[    0.000000]  20
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  05
[    0.000000]  e0
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  13
[    0.000000]  00
[    0.000000]  06
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:01:00.0 config space:
[    0.000000] 
                 00:
[    0.000000]  02
[    0.000000]  10
[    0.000000]  29
[    0.000000]  69
[    0.000000]  07
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  0c
[    0.000000]  00
[    0.000000]  00
[    0.000000]  c0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0c
[    0.000000]  00
[    0.000000]  00
[    0.000000]  d0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  01
[    0.000000]  e0
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  e0
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  02
[    0.000000]  10
[    0.000000]  0c
[    0.000000]  03
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  e4
[    0.000000]  df
[    0.000000]  48
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  09
[    0.000000]  50
[    0.000000]  08
[    0.000000]  00
[    0.000000]  02
[    0.000000]  10
[    0.000000]  0c
[    0.000000]  03
[    0.000000] 
                 50:
[    0.000000]  01
[    0.000000]  58
[    0.000000]  03
[    0.000000]  f6
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  10
[    0.000000]  a0
[    0.000000]  12
[    0.000000]  00
[    0.000000]  a1
[    0.000000]  8f
[    0.000000]  e8
[    0.000000]  07
[    0.000000] 
                 60:
[    0.000000]  30
[    0.000000]  28
[    0.000000]  0b
[    0.000000]  00
[    0.000000]  03
[    0.000000]  0d
[    0.000000]  40
[    0.000000]  00
[    0.000000]  40
[    0.000000]  00
[    0.000000]  23
[    0.000000]  10
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  70
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0e
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  03
[    0.000000]  00
[    0.000000]  1e
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 90:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  05
[    0.000000]  00
[    0.000000]  84
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:02:00.0 config space:
[    0.000000] 
                 00:
[    0.000000]  21
[    0.000000]  1b
[    0.000000]  42
[    0.000000]  12
[    0.000000]  06
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  00
[    0.000000]  30
[    0.000000]  03
[    0.000000]  0c
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  04
[    0.000000]  00
[    0.000000]  d0
[    0.000000]  df
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 20:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  43
[    0.000000]  10
[    0.000000]  75
[    0.000000]  86
[    0.000000] 
                 30:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  50
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0b
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  05
[    0.000000]  68
[    0.000000]  86
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  30
[    0.000000]  20
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  11
[    0.000000]  78
[    0.000000]  07
[    0.000000]  00
[    0.000000]  00
[    0.000000]  20
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  80
[    0.000000]  20
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  80
[    0.000000]  43
[    0.000000]  c0
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  10
[    0.000000]  00
[    0.000000]  02
[    0.000000]  00
[    0.000000]  22
[    0.000000]  82
[    0.000000]  e8
[    0.000000]  07
[    0.000000]  30
[    0.000000]  28
[    0.000000]  10
[    0.000000]  00
[    0.000000]  22
[    0.000000]  dc
[    0.000000]  43
[    0.000000]  01
[    0.000000] 
                 90:
[    0.000000]  40
[    0.000000]  00
[    0.000000]  22
[    0.000000]  10
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  00
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  21
[    0.000000]  1b
[    0.000000]  42
[    0.000000]  12
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  28
[    0.000000]  4f
[    0.000000]  01
[    0.000000]  32
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  ff
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] pci 0000:03:00.0 config space:
[    0.000000] 
                 00:
[    0.000000]  21
[    0.000000]  1b
[    0.000000]  80
[    0.000000]  10
[    0.000000]  07
[    0.000000]  00
[    0.000000]  10
[    0.000000]  00
[    0.000000]  04
[    0.000000]  00
[    0.000000]  04
[    0.000000]  06
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  00
[    0.000000] 
                 10:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  03
[    0.000000]  04
[    0.000000]  04
[    0.000000]  20
[    0.000000]  f1
[    0.000000]  01
[    0.000000]  20
[    0.000000]  20
[    0.000000] 
                 20:
[    0.000000]  f0
[    0.000000]  ff
[    0.000000]  00
[    0.000000]  00
[    0.000000]  f1
[    0.000000]  ff
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 30:
[    0.000000]  ff
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  50
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  0f
[    0.000000]  01
[    0.000000]  10
[    0.000000]  00
[    0.000000] 
                 40:
[    0.000000]  20
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  07
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  a7
[    0.000000]  27
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 50:
[    0.000000]  05
[    0.000000]  78
[    0.000000]  80
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 60:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  11
[    0.000000]  78
[    0.000000]  01
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 70:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  01
[    0.000000]  80
[    0.000000]  23
[    0.000000]  fe
[    0.000000]  08
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 80:
[    0.000000]  10
[    0.000000]  c0
[    0.000000]  71
[    0.000000]  00
[    0.000000]  00
[    0.000000]  80
[    0.000000]  90
[    0.000000]  05
[    0.000000]  10
[    0.000000]  28
[    0.000000]  02
[    0.000000]  00
[    0.000000]  11
[    0.000000]  bc
[    0.000000]  00
[    0.000000]  01
[    0.000000] 
                 90:
[    0.000000]  40
[    0.000000]  00
[    0.000000]  11
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  c0
[    0.000000]  03
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 a0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 b0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 c0:
[    0.000000]  0d
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 d0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 e0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000] 
                 f0:
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00
[    0.000000]  00

[    0.000000] SMBIOS 3.0 present.
[    0.000000] DMI: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x476800 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 00C0000000 mask 7FC0000000 uncachable
[    0.000000]   1 base 00A0000000 mask 7FE0000000 uncachable
[    0.000000]   2 base 0090000000 mask 7FF0000000 uncachable
[    0.000000]   3 base 008C000000 mask 7FFC000000 uncachable
[    0.000000]   4 base 008A000000 mask 7FFE000000 uncachable
[    0.000000]   5 base 0089000000 mask 7FFF000000 uncachable
[    0.000000]   6 base 0088800000 mask 7FFF800000 uncachable
[    0.000000]   7 disabled
[    0.000000]   8 disabled
[    0.000000]   9 disabled
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT  
[    0.000000] e820: last_pfn = 0x88000 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000fcc50-0x000fcc5f] mapped at [ffff8800000fcc50]
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff880000096000] 96000 size 24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] BRK [0x020c7000, 0x020c7fff] PGTABLE
[    0.000000] BRK [0x020c8000, 0x020c8fff] PGTABLE
[    0.000000] BRK [0x020c9000, 0x020c9fff] PGTABLE
[    0.000000] BRK [0x020ca000, 0x020cafff] PGTABLE
[    0.000000] BRK [0x020cb000, 0x020cbfff] PGTABLE
[    0.000000] BRK [0x020cc000, 0x020ccfff] PGTABLE
[    0.000000] RAMDISK: [mem 0x2f6de000-0x33b66fff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000F0580 000024 (v02 ALASKA)
[    0.000000] ACPI: XSDT 0x0000000086DC2098 0000AC (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FACP 0x0000000086DE5038 00010C (v05 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x0000000086DC21E0 022E52 (v02 ALASKA A M I    01072009 INTL 20120913)
[    0.000000] ACPI: FACS 0x000000008774BF80 000040
[    0.000000] ACPI: APIC 0x0000000086DE5148 0000BC (v03 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FPDT 0x0000000086DE5208 000044 (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FIDT 0x0000000086DE5250 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: MCFG 0x0000000086DE52F0 00003C (v01 ALASKA A M I    01072009 MSFT 00000097)
[    0.000000] ACPI: HPET 0x0000000086DE5330 000038 (v01 ALASKA A M I    01072009 AMI. 0005000B)
[    0.000000] ACPI: SSDT 0x0000000086DE5368 00036D (v01 SataRe SataTabl 00001000 INTL 20120913)
[    0.000000] ACPI: LPIT 0x0000000086DE56D8 000094 (v01 INTEL  SKL      00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000086DE5770 000248 (v02 INTEL  sensrhub 00000000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE59B8 002BAE (v02 INTEL  PtidDevc 00001000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE8568 000C45 (v02 INTEL  Ther_Rvp 00001000 INTL 20120913)
[    0.000000] ACPI: DBGP 0x0000000086DE91B0 000034 (v01 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: DBG2 0x0000000086DE91E8 000054 (v00 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000086DE9240 0006FD (v02 INTEL  xh_rvp08 00000000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE9940 005271 (v02 SaSsdt SaSsdt   00003000 INTL 20120913)
[    0.000000] ACPI: UEFI 0x0000000086DEEBB8 000042 (v01                 00000000      00000000)
[    0.000000] ACPI: SSDT 0x0000000086DEEC00 000E58 (v02 CpuRef CpuSsdt  00003000 INTL 20120913)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x00000004767fffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x4767f8000-0x4767fbfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000004767fffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009bfff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x000000007d438fff]
[    0.000000]   node   0: [mem 0x000000007d484000-0x000000007d4c4fff]
[    0.000000]   node   0: [mem 0x000000007db4a000-0x00000000858a7fff]
[    0.000000]   node   0: [mem 0x0000000087fff000-0x0000000087ffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x00000004767fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000004767fffff]
[    0.000000] On node 0 totalpages: 4176244
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3995 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 8456 pages used for memmap
[    0.000000]   DMA32 zone: 541145 pages, LIFO batch:31
[    0.000000]   Normal zone: 56736 pages used for memmap
[    0.000000]   Normal zone: 3631104 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x1808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009c000-0x0009cfff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d439000-0x7d439fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d43a000-0x7d483fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d4c5000-0x7db49fff]
[    0.000000] PM: Registered nosave memory: [mem 0x858a8000-0x86db6fff]
[    0.000000] PM: Registered nosave memory: [mem 0x86db7000-0x86deffff]
[    0.000000] PM: Registered nosave memory: [mem 0x86df0000-0x8774bfff]
[    0.000000] PM: Registered nosave memory: [mem 0x8774c000-0x87ffefff]
[    0.000000] PM: Registered nosave memory: [mem 0x88000000-0x880fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x88100000-0xdfffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xe0000000-0xefffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfdffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe000000-0xfe010fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe011000-0xfebfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfedfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfeffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff]
[    0.000000] e820: [mem 0x88100000-0xdfffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:8 nr_node_ids:1
[    0.000000] percpu: Embedded 36 pages/cpu @ffff880476400000 s108056 r8192 d31208 u262144
[    0.000000] pcpu-alloc: s108056 r8192 d31208 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 4110967
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-custom root=UUID=900c4a92-9f8e-44be-86bc-9673dd3b0d78 ro console=ttyS0,115200,8N1 console=tty0 no_console_suspend pci=earlydump quiet splash vt.handoff=7
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 16288164K/16704976K available (8091K kernel code, 1445K rwdata, 3668K rodata, 2220K init, 1180K bss, 416812K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=8.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=8
[    0.000000] NR_IRQS:16640 nr_irqs:2048 16
[    0.000000] 	Offload RCU callbacks from all CPUs
[    0.000000] 	Offload RCU callbacks from CPUs: 0-7.
[    0.000000] spurious 8259A interrupt: IRQ7.
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [ttyS0] enabled
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Detected 3400.000 MHz processor
[    0.000015] Calibrating delay loop (skipped), value calculated using timer frequency.. 6816.00 BogoMIPS (lpj=13632000)
[    0.000016] pid_max: default: 32768 minimum: 301
[    0.000035] ACPI: Core revision 20160831
[    0.049176] ACPI Error: [\_SB_.PCI0.XHC_.RHUB.HS11] Namespace lookup failure, AE_NOT_FOUND (20160831/dswload-210)
[    0.059575] ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20160831/psobject-227)
[    0.068267] ACPI Exception: AE_NOT_FOUND, (SSDT:xh_rvp08) while loading table (20160831/tbxfload-228)
[    0.089637] ACPI Error: 1 table load failures, 7 successful (20160831/tbxfload-246)
[    0.097470] Security Framework initialized
[    0.097471] Yama: becoming mindful.
[    0.097482] AppArmor: AppArmor initialized
[    0.098215] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.100192] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.101047] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.101057] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.101429] CPU: Physical Processor ID: 0
[    0.101429] CPU: Processor Core ID: 0
[    0.101433] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.101433] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.101436] mce: CPU supports 10 MCE banks
[    0.101444] CPU0: Thermal monitoring enabled (TM1)
[    0.101458] process: using mwait in idle threads
[    0.101460] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.101460] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.101715] Freeing SMP alternatives memory: 32K (ffffffff81f96000 - ffffffff81f9e000)
[    0.131902] ftrace: allocating 32207 entries in 126 pages
[    0.140945] smpboot: APIC(0) Converting physical 0 to logical package 0
[    0.140946] smpboot: Max logical packages: 2
[    0.142146] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.145288] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[    0.184975] TSC deadline timer enabled
[    0.184980] smpboot: CPU0: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz (family: 0x6, model: 0x5e, stepping: 0x3)
[    0.184983] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver.
[    0.185003] ... version:                4
[    0.185003] ... bit width:              48
[    0.185004] ... generic registers:      4
[    0.185004] ... value mask:             0000ffffffffffff
[    0.185004] ... max period:             00007fffffffffff
[    0.185005] ... fixed-purpose events:   3
[    0.185005] ... event mask:             000000070000000f
[    0.185710] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    0.185820] x86: Booting SMP configuration:
[    0.185821] .... node  #0, CPUs:      #1 #2 #3 #4 #5 #6 #7
[    0.747132] x86: Booted up 1 node, 8 CPUs
[    0.747134] smpboot: Total of 8 processors activated (54535.50 BogoMIPS)
[    0.754036] devtmpfs: initialized
[    0.754148] x86/mm: Memory block size: 128MB
[    0.758694] evm: security.selinux
[    0.758695] evm: security.SMACK64
[    0.758695] evm: security.ima
[    0.758695] evm: security.capability
[    0.758748] PM: Registering ACPI NVS region [mem 0x7d439000-0x7d439fff] (4096 bytes)
[    0.758750] PM: Registering ACPI NVS region [mem 0x86df0000-0x8774bfff] (9814016 bytes)
[    0.759537] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.759583] pinctrl core: initialized pinctrl subsystem
[    0.759817] RTC time:  3:09:25, date: 05/23/17
[    0.759927] NET: Registered protocol family 16
[    0.785755] cpuidle: using governor ladder
[    0.793550] cpuidle: using governor menu
[    0.793641] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.793642] ACPI: bus type PCI registered
[    0.793642] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.793763] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.793764] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[    0.793774] PCI: Using configuration type 1 for base access
[    0.813895] HugeTLB registered 1 GB page size, pre-allocated 0 pages
[    0.813896] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.814197] ACPI: Added _OSI(Module Device)
[    0.814198] ACPI: Added _OSI(Processor Device)
[    0.814198] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.814199] ACPI: Added _OSI(Processor Aggregator Device)
[    0.816499] ACPI: Executed 21 blocks of module-level executable AML code
[    0.835899] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    0.839497] ACPI: Dynamic OEM Table Load:
[    0.839502] ACPI: SSDT 0xFFFF8804625B9000 0006E4 (v02 PmRef  Cpu0Ist  00003000 INTL 20120913)
[    0.840012] ACPI: \_PR_.CPU0: _OSC native thermal LVT Acked
[    0.842698] ACPI: Dynamic OEM Table Load:
[    0.842702] ACPI: SSDT 0xFFFF880464BC8800 00037F (v02 PmRef  Cpu0Cst  00003001 INTL 20120913)
[    0.843852] ACPI: Dynamic OEM Table Load:
[    0.843856] ACPI: SSDT 0xFFFF880464042800 0005AA (v02 PmRef  ApIst    00003000 INTL 20120913)
[    0.844843] ACPI: Dynamic OEM Table Load:
[    0.844847] ACPI: SSDT 0xFFFF880462178000 000119 (v02 PmRef  ApCst    00003000 INTL 20120913)
[    0.853361] ACPI: Interpreter enabled
[    0.853387] ACPI: (supports S0 S3 S4 S5)
[    0.853388] ACPI: Using IOAPIC for interrupt routing
[    0.853431] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.856700] ACPI: Power Resource [PG00] (on)
[    0.857464] ACPI: Power Resource [PG01] (on)
[    0.858179] ACPI: Power Resource [PG02] (on)
[    0.862415] ACPI: Power Resource [WRST] (off)
[    0.863136] ACPI: Power Resource [WRST] (off)
[    0.863852] ACPI: Power Resource [WRST] (off)
[    0.864571] ACPI: Power Resource [WRST] (off)
[    0.865291] ACPI: Power Resource [WRST] (off)
[    0.866019] ACPI: Power Resource [WRST] (off)
[    0.866735] ACPI: Power Resource [WRST] (off)
[    0.867466] ACPI: Power Resource [WRST] (off)
[    0.868181] ACPI: Power Resource [WRST] (off)
[    0.868897] ACPI: Power Resource [WRST] (off)
[    0.869614] ACPI: Power Resource [WRST] (off)
[    0.870335] ACPI: Power Resource [WRST] (off)
[    0.871051] ACPI: Power Resource [WRST] (off)
[    0.871774] ACPI: Power Resource [WRST] (off)
[    0.872496] ACPI: Power Resource [WRST] (off)
[    0.873215] ACPI: Power Resource [WRST] (off)
[    0.873931] ACPI: Power Resource [WRST] (off)
[    0.874645] ACPI: Power Resource [WRST] (off)
[    0.875360] ACPI: Power Resource [WRST] (off)
[    0.876074] ACPI: Power Resource [WRST] (off)
[    0.890607] ACPI: Power Resource [FN00] (off)
[    0.890767] ACPI: Power Resource [FN01] (off)
[    0.890928] ACPI: Power Resource [FN02] (off)
[    0.891086] ACPI: Power Resource [FN03] (off)
[    0.891244] ACPI: Power Resource [FN04] (off)
[    0.893204] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe])
[    0.893208] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    0.893287] acpi PNP0A08:00: _OSC failed (AE_ERROR); disabling ASPM
[    0.894651] PCI host bridge to bus 0000:00
[    0.894652] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.894654] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.894655] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.894656] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
[    0.894657] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
[    0.894658] pci_bus 0000:00: root bus resource [bus 00-fe]
[    0.894664] pci 0000:00:00.0: [8086:191f] type 00 class 0x060000
[    0.895205] pci 0000:00:14.0: [8086:a12f] type 00 class 0x0c0330
[    0.895221] pci 0000:00:14.0: reg 0x10: [mem 0xdff30000-0xdff3ffff 64bit]
[    0.895277] pci 0000:00:14.0: PME# supported from D3hot D3cold
[    0.895495] pci 0000:00:14.0: System wakeup disabled by ACPI
[    0.895580] pci 0000:00:16.0: [8086:a13a] type 00 class 0x078000
[    0.895593] pci 0000:00:16.0: reg 0x10: [mem 0xdff4d000-0xdff4dfff 64bit]
[    0.895639] pci 0000:00:16.0: PME# supported from D3hot
[    0.895901] pci 0000:00:17.0: [8086:a102] type 00 class 0x010601
[    0.895912] pci 0000:00:17.0: reg 0x10: [mem 0xdff48000-0xdff49fff]
[    0.895918] pci 0000:00:17.0: reg 0x14: [mem 0xdff4c000-0xdff4c0ff]
[    0.895923] pci 0000:00:17.0: reg 0x18: [io  0xf050-0xf057]
[    0.895929] pci 0000:00:17.0: reg 0x1c: [io  0xf040-0xf043]
[    0.895934] pci 0000:00:17.0: reg 0x20: [io  0xf020-0xf03f]
[    0.895939] pci 0000:00:17.0: reg 0x24: [mem 0xdff4b000-0xdff4b7ff]
[    0.895972] pci 0000:00:17.0: PME# supported from D3hot
[    0.896230] pci 0000:00:1b.0: [8086:a167] type 01 class 0x060400
[    0.896278] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
[    0.896547] pci 0000:00:1b.0: System wakeup disabled by ACPI
[    0.896630] pci 0000:00:1c.0: [8086:a110] type 01 class 0x060400
[    0.896679] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    0.896946] pci 0000:00:1c.0: System wakeup disabled by ACPI
[    0.897007] pci 0000:00:1c.2: [8086:a112] type 01 class 0x060400
[    0.897056] pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold
[    0.897323] pci 0000:00:1c.2: System wakeup disabled by ACPI
[    0.897399] pci 0000:00:1d.0: [8086:a118] type 01 class 0x060400
[    0.897449] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[    0.897711] pci 0000:00:1d.0: System wakeup disabled by ACPI
[    0.897797] pci 0000:00:1f.0: [8086:a145] type 00 class 0x060100
[    0.898127] pci 0000:00:1f.2: [8086:a121] type 00 class 0x058000
[    0.898137] pci 0000:00:1f.2: reg 0x10: [mem 0xdff44000-0xdff47fff]
[    0.898418] pci 0000:00:1f.3: [8086:a170] type 00 class 0x040300
[    0.898438] pci 0000:00:1f.3: reg 0x10: [mem 0xdff40000-0xdff43fff 64bit]
[    0.898465] pci 0000:00:1f.3: reg 0x20: [mem 0xdff20000-0xdff2ffff 64bit]
[    0.898510] pci 0000:00:1f.3: PME# supported from D3hot D3cold
[    0.898780] pci 0000:00:1f.3: System wakeup disabled by ACPI
[    0.898846] pci 0000:00:1f.4: [8086:a123] type 00 class 0x0c0500
[    0.898893] pci 0000:00:1f.4: reg 0x10: [mem 0xdff4a000-0xdff4a0ff 64bit]
[    0.898962] pci 0000:00:1f.4: reg 0x20: [io  0xf000-0xf01f]
[    0.899275] pci 0000:00:1f.6: [8086:15b8] type 00 class 0x020000
[    0.899294] pci 0000:00:1f.6: reg 0x10: [mem 0xdff00000-0xdff1ffff]
[    0.899395] pci 0000:00:1f.6: PME# supported from D0 D3hot D3cold
[    0.899600] pci 0000:00:1f.6: System wakeup disabled by ACPI
[    0.899769] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.899784] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.899794] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.899800] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.899807] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.899813] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.899877] pci 0000:01:00.0: supports D1 D2
[    0.899877] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.899911] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
[    0.899912] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.899921] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
[    0.899921] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
[    0.899933] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.899933] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
[    0.899985] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.909949] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.909954] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.909959] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.909967] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.909984] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
[    0.910106] pci 0000:02:00.0: [1b21:1242] type 00 class 0x0c0330
[    0.910126] pci 0000:02:00.0: reg 0x10: [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.910232] pci 0000:02:00.0: PME# supported from D3hot D3cold
[    0.910285] pci 0000:02:00.0: System wakeup disabled by ACPI
[    0.921964] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.921972] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.922123] pci 0000:03:00.0: [1b21:1080] type 01 class 0x060400
[    0.922240] pci 0000:03:00.0: supports D1 D2
[    0.922240] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.922286] pci 0000:03:00.0: System wakeup disabled by ACPI
[    0.933966] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.934115] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.934264] acpiphp: Slot [1] registered
[    0.934267] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.937083] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.937197] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.937309] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 10 11 12 14 *15)
[    0.937420] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.937531] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.937643] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.937754] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.937865] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
[    0.938691] ACPI: Enabled 6 GPEs in block 00 to 7F
[    0.938968] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.938969] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.938969] vgaarb: loaded
[    0.938970] vgaarb: bridge control possible 0000:01:00.0
[    0.939235] SCSI subsystem initialized
[    0.939288] libata version 3.00 loaded.
[    0.939318] ACPI: bus type USB registered
[    0.939348] usbcore: registered new interface driver usbfs
[    0.939363] usbcore: registered new interface driver hub
[    0.939384] usbcore: registered new device driver usb
[    0.939516] PCI: Using ACPI for IRQ routing
[    0.967348] PCI: pci_cache_line_size set to 64 bytes
[    0.967385] e820: reserve RAM buffer [mem 0x0009c800-0x0009ffff]
[    0.967386] e820: reserve RAM buffer [mem 0x7d439000-0x7fffffff]
[    0.967387] e820: reserve RAM buffer [mem 0x7d4c5000-0x7fffffff]
[    0.967388] e820: reserve RAM buffer [mem 0x858a8000-0x87ffffff]
[    0.967390] e820: reserve RAM buffer [mem 0x476800000-0x477ffffff]
[    0.967546] NetLabel: Initializing
[    0.967547] NetLabel:  domain hash size = 128
[    0.967547] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.967565] NetLabel:  unlabeled traffic allowed by default
[    0.967695] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    0.967698] hpet0: 8 comparators, 64-bit 24.000000 MHz counter
[    0.969739] clocksource: Switched to clocksource hpet
[    0.979775] VFS: Disk quotas dquot_6.6.0
[    0.979810] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.979940] AppArmor: AppArmor Filesystem Enabled
[    0.979994] pnp: PnP ACPI init
[    0.980278] system 00:00: [io  0x0290-0x029f] has been reserved
[    0.980280] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.980751] pnp 00:01: [dma 0 disabled]
[    0.980815] pnp 00:01: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.981055] system 00:02: [io  0x0680-0x069f] has been reserved
[    0.981057] system 00:02: [io  0xffff] has been reserved
[    0.981058] system 00:02: [io  0xffff] has been reserved
[    0.981060] system 00:02: [io  0xffff] has been reserved
[    0.981061] system 00:02: [io  0x1800-0x18fe] has been reserved
[    0.981063] system 00:02: [io  0x164e-0x164f] has been reserved
[    0.981064] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.981173] system 00:03: [io  0x0800-0x087f] has been reserved
[    0.981174] system 00:03: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.981213] pnp 00:04: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.981283] system 00:05: [io  0x1854-0x1857] has been reserved
[    0.981284] system 00:05: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
[    0.981675] system 00:06: [mem 0xfed10000-0xfed17fff] has been reserved
[    0.981677] system 00:06: [mem 0xfed18000-0xfed18fff] has been reserved
[    0.981678] system 00:06: [mem 0xfed19000-0xfed19fff] has been reserved
[    0.981679] system 00:06: [mem 0xe0000000-0xefffffff] has been reserved
[    0.981681] system 00:06: [mem 0xfed20000-0xfed3ffff] has been reserved
[    0.981682] system 00:06: [mem 0xfed90000-0xfed93fff] has been reserved
[    0.981684] system 00:06: [mem 0xfed45000-0xfed8ffff] has been reserved
[    0.981686] system 00:06: [mem 0xff000000-0xffffffff] has been reserved
[    0.981688] system 00:06: [mem 0xfee00000-0xfeefffff] could not be reserved
[    0.981689] system 00:06: [mem 0xdffc0000-0xdffdffff] has been reserved
[    0.981690] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.981791] system 00:07: [mem 0xfd000000-0xfdabffff] has been reserved
[    0.981793] system 00:07: [mem 0xfdad0000-0xfdadffff] has been reserved
[    0.981794] system 00:07: [mem 0xfdb00000-0xfdffffff] has been reserved
[    0.981796] system 00:07: [mem 0xfe000000-0xfe01ffff] could not be reserved
[    0.981798] system 00:07: [mem 0xfe036000-0xfe03bfff] has been reserved
[    0.981800] system 00:07: [mem 0xfe03d000-0xfe3fffff] has been reserved
[    0.981801] system 00:07: [mem 0xfe410000-0xfe7fffff] has been reserved
[    0.981803] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.983536] system 00:08: [mem 0xfdaf0000-0xfdafffff] has been reserved
[    0.983538] system 00:08: [mem 0xfdae0000-0xfdaeffff] has been reserved
[    0.983540] system 00:08: [mem 0xfdac0000-0xfdacffff] has been reserved
[    0.983541] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.984902] pnp: PnP ACPI: found 9 devices
[    0.992842] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.992845] pci_bus 0000:00: max bus depth: 2 pci_try_num: 3
[    0.992875] pci 0000:00:1d.0: BAR 14: assigned [mem 0x88800000-0x889fffff]
[    0.992878] pci 0000:00:1d.0: BAR 15: assigned [mem 0x88a00000-0x88bfffff 64bit pref]
[    0.992879] pci 0000:00:1d.0: BAR 13: assigned [io  0x2000-0x2fff]
[    0.992884] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.992885] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.992886] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
[    0.992887] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
[    0.992888] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.992889] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.992890] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.992892] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.992894] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.992896] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.992899] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.992901] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.992905] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.992918] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.992924] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.992925] pci 0000:00:1d.0:   bridge window [io  0x2000-0x2fff]
[    0.992928] pci 0000:00:1d.0:   bridge window [mem 0x88800000-0x889fffff]
[    0.992929] pci 0000:00:1d.0:   bridge window [mem 0x88a00000-0x88bfffff 64bit pref]
[    0.992933] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.992933] release child resource [mem 0xdfe00000-0xdfe3ffff]
[    0.992934] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
[    0.992934] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] released
[    0.992935] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.992936] release child resource [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.992937] release child resource [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.992937] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64bit pref] released
[    0.992938] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.992969] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.992970] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.992971] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adfffff]
[    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
[    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
[    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
[    0.992998] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.993000] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.993002] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff]
[    0.993006] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.993008] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.993012] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.993025] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.993031] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.993032] pci 0000:00:1d.0:   bridge window [io  0x2000-0x2fff]
[    0.993035] pci 0000:00:1d.0:   bridge window [mem 0x88800000-0x889fffff]
[    0.993036] pci 0000:00:1d.0:   bridge window [mem 0x88a00000-0x88bfffff 64bit pref]
[    0.993040] pci_bus 0000:00: No. 3 try to assign unassigned res
[    0.993040] release child resource [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.993040] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.993041] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] released
[    0.993042] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.993043] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.993044] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] released
[    0.993044] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.993046] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64bit pref] released
[    0.993047] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.993057] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 64bit pref] to [bus 01] add_size 48000000 add_align 8000000
[    0.993058] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] to [bus 01] add_size 2200000 add_align 400000
[    0.993080] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
[    0.993084] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0x5fffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.993085] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0xa7ffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.993086] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x022fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.993087] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x044fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.993088] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.993088] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x004fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.993091] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 64bit pref]
[    0.993091] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000000 64bit pref]
[    0.993092] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cffffff]
[    0.993093] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88afffff]
[    0.993095] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3fffff 64bit pref]
[    0.993098] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.993098] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.993099] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abfffff]
[    0.993100] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acfffff]
[    0.993102] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aefffff 64bit pref]
[    0.993104] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0fffff 64bit pref] (expanded by 0x200000)
[    0.993105] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7fffff] (expanded by 0x2200000)
[    0.993112] pci 0000:01:00.0: res[7]=[mem size 0x00000000 64bit pref] res_to_dev_res add_size 40000000 min_align 0
[    0.993113] pci 0000:01:00.0: res[9]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 2000000 min_align 0
[    0.993113] pci 0000:01:00.0: res[12]=[mem size 0x00000000] res_to_dev_res add_size 2000000 min_align 0
[    0.993114] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.993115] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.993116] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.993117] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.993118] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff 64bit pref]
[    0.993124] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3fffff]
[    0.993127] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff 64bit pref]
[    0.993133] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff]
[    0.993137] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.993138] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.993139] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff 64bit pref]
[    0.993146] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff]
[    0.993148] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7fffff]
[    0.993151] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff 64bit pref]
[    0.993157] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.993158] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.993160] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.993161] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.993163] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff]
[    0.993168] pci 0000:02:00.0: BAR 0: assigned [mem 0x8ac00000-0x8ac07fff 64bit]
[    0.993177] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.993179] pci 0000:00:1c.0:   bridge window [mem 0x8ac00000-0x8acfffff]
[    0.993183] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.993196] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.993202] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.993203] pci 0000:00:1d.0:   bridge window [io  0x2000-0x2fff]
[    0.993205] pci 0000:00:1d.0:   bridge window [mem 0x88800000-0x889fffff]
[    0.993207] pci 0000:00:1d.0:   bridge window [mem 0x8ad00000-0x8b0fffff 64bit pref]
[    0.993210] pci_bus 0000:00: Automatically enabled pci realloc, if you have problem, try booting with pci=realloc=off
[    0.993212] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.993212] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.993213] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.993214] pci_bus 0000:00: resource 7 [mem 0x88800000-0xdfffffff window]
[    0.993214] pci_bus 0000:00: resource 8 [mem 0xfd000000-0xfe7fffff window]
[    0.993215] pci_bus 0000:01: resource 0 [io  0xe000-0xefff]
[    0.993215] pci_bus 0000:01: resource 1 [mem 0x8b400000-0x8f7fffff]
[    0.993216] pci_bus 0000:02: resource 1 [mem 0x8ac00000-0x8acfffff]
[    0.993217] pci_bus 0000:05: resource 0 [io  0x2000-0x2fff]
[    0.993217] pci_bus 0000:05: resource 1 [mem 0x88800000-0x889fffff]
[    0.993218] pci_bus 0000:05: resource 2 [mem 0x8ad00000-0x8b0fffff 64bit pref]
[    0.993408] NET: Registered protocol family 2
[    0.993609] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.993728] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.993823] TCP: Hash tables configured (established 131072 bind 65536)
[    0.993845] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    0.993874] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    0.993964] NET: Registered protocol family 1
[    0.994633] pci 0000:01:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.995308] PCI: CLS 0 bytes, default 64
[    0.995353] Trying to unpack rootfs image as initramfs...
[    1.722933] Freeing initrd memory: 70180K (ffff88002f6de000 - ffff880033b67000)
[    1.722935] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.722937] software IO TLB [mem 0x818a8000-0x858a8000] (64MB) mapped at [ffff8800818a8000-ffff8800858a7fff]
[    1.723156] RAPL PMU: API unit is 2^-32 Joules, 5 fixed counters, 655360 ms ovfl timer
[    1.723157] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
[    1.723157] RAPL PMU: hw unit of domain package 2^-14 Joules
[    1.723157] RAPL PMU: hw unit of domain dram 2^-14 Joules
[    1.723158] RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules
[    1.723158] RAPL PMU: hw unit of domain psys 2^-14 Joules
[    1.723731] Scanning for low memory corruption every 60 seconds
[    1.724315] futex hash table entries: 2048 (order: 5, 131072 bytes)
[    1.724341] audit: initializing netlink subsys (disabled)
[    1.724358] audit: type=2000 audit(1495508965.692:1): initialized
[    1.725131] Initialise system trusted keyrings
[    1.725270] workingset: timestamp_bits=40 max_order=22 bucket_order=0
[    1.729709] fuse init (API version 7.26)
[    1.730979] Key type asymmetric registered
[    1.730980] Asymmetric key parser 'x509' registered
[    1.731041] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    1.731094] io scheduler noop registered
[    1.731095] io scheduler deadline registered (default)
[    1.731102] io scheduler cfq registered
[    1.731590] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    1.731597] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    1.731638] efifb: probing for efifb
[    1.731641] efifb: framebuffer at 0xa0000, using 64k, total 64k
[    1.731641] efifb: mode is 640x480x1, linelength=80, pages=1
[    1.731642] efifb: scrolling: redraw
[    1.731642] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.732374] Console: switching to colour frame buffer device 80x30
[    1.732982] fb0: EFI VGA frame buffer device
[    1.732990] intel_idle: MWAIT substates: 0x142120
[    1.732991] intel_idle: v0.4.1 model 0x5E
[    1.733723] intel_idle: lapic_timer_reliable_states 0xffffffff
[    1.733734] ipmi message handler version 39.2
[    1.733864] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input0
[    1.733866] ACPI: Sleep Button [SLPB]
[    1.733927] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[    1.733929] ACPI: Power Button [PWRB]
[    1.733990] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[    1.733992] ACPI: Power Button [PWRF]
[    1.735745] thermal LNXTHERM:00: registered as thermal_zone0
[    1.735746] ACPI: Thermal Zone [TZ00] (28 C)
[    1.736097] thermal LNXTHERM:01: registered as thermal_zone1
[    1.736098] ACPI: Thermal Zone [TZ01] (30 C)
[    1.736129] GHES: HEST is not enabled!
[    1.736269] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    1.756811] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.759871] Linux agpgart interface v0.103
[    1.766022] brd: module loaded
[    1.769313] loop: module loaded
[    1.769764] libphy: Fixed MDIO Bus: probed
[    1.770200] tun: Universal TUN/TAP device driver, 1.6
[    1.770201] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    1.770297] PPP generic driver version 2.4.2
[    1.770387] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.770390] ehci-pci: EHCI PCI platform driver
[    1.770403] ehci-platform: EHCI generic platform driver
[    1.770415] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.770417] ohci-pci: OHCI PCI platform driver
[    1.770428] ohci-platform: OHCI generic platform driver
[    1.770436] uhci_hcd: USB Universal Host Controller Interface driver
[    1.770631] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.770637] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[    1.771719] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x00109810
[    1.771723] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
[    1.771837] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    1.771838] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.771839] usb usb1: Product: xHCI Host Controller
[    1.771839] usb usb1: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.771840] usb usb1: SerialNumber: 0000:00:14.0
[    1.772062] hub 1-0:1.0: USB hub found
[    1.772080] hub 1-0:1.0: 16 ports detected
[    1.786880] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.786883] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[    1.786926] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[    1.786927] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.786927] usb usb2: Product: xHCI Host Controller
[    1.786928] usb usb2: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.786928] usb usb2: SerialNumber: 0000:00:14.0
[    1.787132] hub 2-0:1.0: USB hub found
[    1.787146] hub 2-0:1.0: 10 ports detected
[    1.796782] xhci_hcd 0000:02:00.0: xHCI Host Controller
[    1.796788] xhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 3
[    1.855549] xhci_hcd 0000:02:00.0: hcc params 0x0200eec1 hci version 0x110 quirks 0x00000010
[    1.855867] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
[    1.855868] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.855868] usb usb3: Product: xHCI Host Controller
[    1.855869] usb usb3: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.855869] usb usb3: SerialNumber: 0000:02:00.0
[    1.856068] hub 3-0:1.0: USB hub found
[    1.856075] hub 3-0:1.0: 2 ports detected
[    1.856196] xhci_hcd 0000:02:00.0: xHCI Host Controller
[    1.856199] xhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 4
[    1.856228] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    1.856252] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003
[    1.856253] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.856253] usb usb4: Product: xHCI Host Controller
[    1.856254] usb usb4: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.856254] usb usb4: SerialNumber: 0000:02:00.0
[    1.856449] hub 4-0:1.0: USB hub found
[    1.856457] hub 4-0:1.0: 2 ports detected
[    1.856615] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.859955] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.859958] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.860246] mousedev: PS/2 mouse device common for all mice
[    1.860616] rtc_cmos 00:04: RTC can wake from S4
[    1.861033] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
[    1.861113] rtc_cmos 00:04: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
[    1.861185] device-mapper: uevent: version 1.0.3
[    1.861338] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
[    1.861340] intel_pstate: Intel P-state driver initializing
[    1.862180] intel_pstate: HWP enabled
[    1.862594] ledtrig-cpu: registered to indicate activity on CPUs
[    1.862884] NET: Registered protocol family 10
[    1.863303] NET: Registered protocol family 17
[    1.863309] Key type dns_resolver registered
[    1.863832] microcode: sig=0x506e3, pf=0x2, revision=0x33
[    1.864201] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    1.864586] registered taskstats version 1
[    1.864587] Loading compiled-in X.509 certificates
[    1.867068] Loaded X.509 cert 'Build time autogenerated kernel key: 425e568c95731e69b073bb62214af700479a67a7'
[    1.867273] kmemleak: Kernel memory leak detector initialized
[    1.867306] kmemleak: Automatic memory scanning thread started
[    1.869269] Key type trusted registered
[    1.873218] Key type encrypted registered
[    1.873220] AppArmor: AppArmor sha1 policy hashing enabled
[    1.873221] ima: No TPM chip found, activating TPM-bypass!
[    1.873238] evm: HMAC attrs: 0x1
[    1.874566]   Magic number: 1:653:159
[    1.874607] acpi device:26: hash matches
[    1.874781] rtc_cmos 00:04: setting system clock to 2017-05-23 03:09:26 UTC (1495508966)
[    1.875067] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    1.875067] EDD information not available.
[    1.875199] PM: Hibernation image not present or could not be loaded.
[    1.875981] Freeing unused kernel memory: 2220K (ffffffff81d6b000 - ffffffff81f96000)
[    1.875982] Write protecting the kernel read-only data: 12288k
[    1.876184] Freeing unused kernel memory: 84K (ffff8800017eb000 - ffff880001800000)
[    1.876809] Freeing unused kernel memory: 428K (ffff880001b95000 - ffff880001c00000)
[    1.885819] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.885884] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.885890] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.886260] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.886285] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.887446] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.887489] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.888055] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.888096] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.888136] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.917901] hid: module verification failed: signature and/or required key missing - tainting kernel
[    1.918301] hidraw: raw HID events driver (C) Jiri Kosina
[    1.922762] wmi: Mapper loaded
[    1.938531] ahci 0000:00:17.0: version 3.0
[    1.940491] pps_core: LinuxPPS API ver. 1 registered
[    1.940491] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.940806] PTP clock support registered
[    1.941521] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.941521] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.949165] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[    1.949166] ahci 0000:00:17.0: flags: 64bit ncq sntf led clo only pio slum part ems deso sadm sds apst 
[    2.007796] scsi host0: ahci
[    2.008128] scsi host1: ahci
[    2.008456] scsi host2: ahci
[    2.008778] scsi host3: ahci
[    2.009104] scsi host4: ahci
[    2.009432] scsi host5: ahci
[    2.009551] ata1: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b100 irq 129
[    2.009556] ata2: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b180 irq 129
[    2.009557] ata3: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b200 irq 129
[    2.009558] ata4: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b280 irq 129
[    2.009559] ata5: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b300 irq 129
[    2.009560] ata6: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b380 irq 129
[    2.010176] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    2.113870] usb 1-10: new full-speed USB device number 2 using xhci_hcd
[    2.245903] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    2.256463] usb 1-10: New USB device found, idVendor=046d, idProduct=c534
[    2.256464] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.256465] usb 1-10: Product: USB Receiver
[    2.256465] usb 1-10: Manufacturer: Logitech
[    2.261440] usbcore: registered new interface driver usbhid
[    2.261440] usbhid: USB HID core driver
[    2.262335] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C534.0001/input/input6
[    2.322250] hid-generic 0003:046D:C534.0001: input,hidraw0: USB HID v1.11 Keyboard [Logitech USB Receiver] on usb-0000:00:14.0-10/input0
[    2.322431] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.1/0003:046D:C534.0002/input/input7
[    2.326537] CPU: 6 PID: 214 Comm: scsi_eh_5 Tainted: G            E   4.9.0-custom #2
[    2.326538] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326538]  ffffc9000211fae8 ffffffff813c2d3f ffff880452b6de80 ffff880452b6e2b0
[    2.326540]  ffffc9000211fb10 ffffffff815a7226 ffffc9000211fb10 0000000000000286
[    2.326541]  ffffc90000029380 ffffc9000211fb30 ffffffffa003f634 ffff880452b6de80
[    2.326543] Call Trace:
[    2.326546]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.326548]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.326550]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.326551]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.326553]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326555]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326557]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326558]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326559]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326561]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326562]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.326563]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326565]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326566]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326567]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.326568]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.326570]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.326572]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.326573]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.326574]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.326575]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.326577]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326578]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.326579]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326580]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326581]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.326582]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326584]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.326590] ata6: SATA link down (SStatus 4 SControl 300)
[    2.326611] CPU: 2 PID: 204 Comm: scsi_eh_0 Tainted: G            E   4.9.0-custom #2
[    2.326611] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326612]  ffffc90001ef3ae8 ffffffff813c2d3f ffff880452b59e80 ffff880452b5a2b0
[    2.326613]  ffffc90001ef3b10 ffffffff815a7226 ffffc90001ef3b10 0000000000000286
[    2.326614]  ffffc90000029100 ffffc90001ef3b30 ffffffffa003f634 ffff880452b59e80
[    2.326615] Call Trace:
[    2.326617]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.326618]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.326619]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.326620]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.326621]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326622]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326623]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326624]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326625]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326626]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326627]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.326628]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326629]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326630]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326631]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.326632]  [<ffffffff813cc32a>] ? put_dec+0x1a/0x80
[    2.326633]  [<ffffffff813cc65e>] ? number+0x2ce/0x300
[    2.326634]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.326635]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.326636]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.326637]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.326638]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.326639]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.326640]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326641]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.326642]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326643]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326644]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.326645]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326646]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326647]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.326650] ata1: SATA link down (SStatus 4 SControl 300)
[    2.326670] CPU: 1 PID: 206 Comm: scsi_eh_1 Tainted: G            E   4.9.0-custom #2
[    2.326671] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326671]  ffffc90001f3bae8 ffffffff813c2d3f ffff880452b5de80 ffff880452b5e2b0
[    2.326673]  ffffc90001f3bb10 ffffffff815a7226 ffffc90001f3bb10 0000000000000286
[    2.326674]  ffffc90000029180 ffffc90001f3bb30 ffffffffa003f634 ffff880452b5de80
[    2.326675] Call Trace:
[    2.326677]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.326678]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.326680]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.326681]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.326683]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326684]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326686]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326687]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326688]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326689]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326691]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.326692]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326693]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326694]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326695]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.326696]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.326698]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.326699]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.326700]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.326701]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.326703]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.326704]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326705]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.326706]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326707]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326708]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.326709]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326710]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.326713] ata2: SATA link down (SStatus 4 SControl 300)
[    2.326746] CPU: 5 PID: 212 Comm: scsi_eh_4 Tainted: G            E   4.9.0-custom #2
[    2.326747] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326747]  ffffc90002003ae8 ffffffff813c2d3f ffff880452b69e80 ffff880452b6a2b0
[    2.326748]  ffffc90002003b10 ffffffff815a7226 ffffc90002003b10 0000000000000286
[    2.326749]  ffffc90000029300 ffffc90002003b30 ffffffffa003f634 ffff880452b69e80
[    2.326750] Call Trace:
[    2.326751]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.326752]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.326753]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.326754]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.326756]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326756]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326757]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326758]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326759]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326760]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326761]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.326762]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326763]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326764]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326765]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.326766]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.326767]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.326768]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.326769]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.326770]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.326771]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.326772]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326773]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.326774]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326775]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326776]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.326777]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326778]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.326781] ata5: SATA link down (SStatus 4 SControl 300)
[    2.326803] CPU: 4 PID: 208 Comm: scsi_eh_2 Tainted: G            E   4.9.0-custom #2
[    2.326804] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326804]  ffffc90001f4bae8 ffffffff813c2d3f ffff880452b61e80 ffff880452b622b0
[    2.326806]  ffffc90001f4bb10 ffffffff815a7226 ffffc90001f4bb10 0000000000000286
[    2.326807]  ffffc90000029200 ffffc90001f4bb30 ffffffffa003f634 ffff880452b61e80
[    2.326808] Call Trace:
[    2.326810]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.326811]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.326813]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.326814]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.326816]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326817]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326818]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326820]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326821]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326822]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326824]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.326825]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326826]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326839]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326840]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.326841]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.326842]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.326843]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.326845]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.326846]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.326847]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.326848]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326849]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.326850]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326851]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326852]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.326853]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326854]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.326857] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.326873] CPU: 0 PID: 210 Comm: scsi_eh_3 Tainted: G            E   4.9.0-custom #2
[    2.326874] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.326874]  ffffc90001febae8 ffffffff813c2d3f ffff880452b65e80 ffff880452b662b0
[    2.326875]  ffffc90001febb10 ffffffff815a7226 ffffc90001febb10 0000000000000286
[    2.326876]  ffffc90000029280 ffffc90001febb30 ffffffffa003f634 ffff880452b65e80
[    2.326877] Call Trace:
[    2.326879]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.326879]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.326881]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.326882]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.326883]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326884]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326885]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326886]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326886]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326887]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326888]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.326889]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.326890]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.326891]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.326892]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.326893]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.326894]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.326895]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.326896]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.326897]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.326898]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.326899]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326900]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.326901]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326902]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.326903]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.326904]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.326905]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.326908] ata4: SATA link down (SStatus 4 SControl 300)
[    2.327973] ata3.00: ATA-8: ST3500413AS, JC45, max UDMA/133
[    2.327974] ata3.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32)
[    2.329553] ata3.00: configured for UDMA/133
[    2.330033] scsi 2:0:0:0: Direct-Access     ATA      ST3500413AS      JC45 PQ: 0 ANSI: 5
[    2.330224] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 30:5a:3a:e0:7f:4d
[    2.330225] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    2.330291] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: FFFFFF-0FF
[    2.331018] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
[    2.382480] hid-generic 0003:046D:C534.0002: input,hiddev0,hidraw1: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-0000:00:14.0-10/input1
[    2.386345] sd 2:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/466 GiB)
[    2.386374] sd 2:0:0:0: Attached scsi generic sg0 type 0
[    2.386556] sd 2:0:0:0: [sda] Write Protect is off
[    2.386557] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.386708] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.415045]  sda: sda1 sda2 < sda5 >
[    2.416527] sd 2:0:0:0: [sda] Attached SCSI disk
[    2.745882] tsc: Refined TSC clocksource calibration: 3407.997 MHz
[    2.745886] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x311fd171fc9, max_idle_ns: 440795303639 ns
[    2.761152] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[    2.833209] random: fast init done
[    3.697270] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[    3.697402] systemd[1]: Detected architecture x86-64.
[    3.703893] systemd[1]: Set hostname to <jenkins-MS-7984>.
[    3.769934] clocksource: Switched to clocksource tsc
[    4.353732] systemd[1]: Listening on Journal Socket (/dev/log).
[    4.353820] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
[    4.353876] systemd[1]: Listening on Journal Audit Socket.
[    4.353899] systemd[1]: Listening on udev Control Socket.
[    4.353913] systemd[1]: Listening on udev Kernel Socket.
[    4.353938] systemd[1]: Listening on Journal Socket.
[    4.353947] systemd[1]: PNFS blkmaping enablement. is not active.
[    4.754407] random: crng init done
[    5.232246] lp: driver loaded but no devices found
[    5.282967] RPC: Registered named UNIX socket transport module.
[    5.282967] RPC: Registered udp transport module.
[    5.282968] RPC: Registered tcp transport module.
[    5.282968] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    5.314923] ppdev: user-space parallel port driver
[    5.757387] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    5.951868] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro
[    6.210363] systemd-journald[291]: Received request to flush runtime journal from PID 1
[    6.395294] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    6.677090] mei_me 0000:00:16.0: enabling device (0000 -> 0002)
[    7.194322] AVX2 version of gcm_enc/dec engaged.
[    7.194322] AES CTR mode by8 optimization enabled
[    7.802905] intel_rapl: Found RAPL domain package
[    7.802906] intel_rapl: Found RAPL domain core
[    7.802907] intel_rapl: Found RAPL domain dram
[    7.992002] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC892: line_outs=3 (0x14/0x15/0x16/0x0/0x0) type:line
[    7.992003] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    7.992004] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    7.992005] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    7.992005] snd_hda_codec_realtek hdaudioC0D0:    dig-out=0x11/0x1e
[    7.992006] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    7.992006] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[    7.992007] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[    7.992008] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a
[    8.004083] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input8
[    8.004191] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9
[    8.004295] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
[    8.004404] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
[    8.004515] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1f.3/sound/card0/input12
[    8.004621] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13
[    8.004725] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input14
[    9.283193] asus_wmi: ASUS WMI generic driver loaded
[    9.366216] audit: type=1400 audit(1495508973.987:2): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine" pid=701 comm="apparmor_parser"
[    9.366235] audit: type=1400 audit(1495508973.987:3): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=701 comm="apparmor_parser"
[    9.367373] audit: type=1400 audit(1495508973.987:4): apparmor="STATUS" operation="profile_load" name="/usr/sbin/ippusbxd" pid=704 comm="apparmor_parser"
[    9.387604] asus_wmi: Initialization: 0x0
[    9.387681] asus_wmi: BIOS WMI version: 0.9
[    9.387810] asus_wmi: SFUN value: 0x0
[    9.388864] input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input15
[    9.389111] asus_wmi: Number of fans: 1
[    9.429704] audit: type=1400 audit(1495508974.051:5): apparmor="STATUS" operation="profile_load" name="/sbin/dhclient" pid=697 comm="apparmor_parser"
[    9.429710] audit: type=1400 audit(1495508974.051:6): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=697 comm="apparmor_parser"
[    9.429714] audit: type=1400 audit(1495508974.051:7): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=697 comm="apparmor_parser"
[    9.429718] audit: type=1400 audit(1495508974.051:8): apparmor="STATUS" operation="profile_load" name="/usr/lib/connman/scripts/dhclient-script" pid=697 comm="apparmor_parser"
[    9.431296] audit: type=1400 audit(1495508974.051:9): apparmor="STATUS" operation="profile_load" name="/usr/lib/cups/backend/cups-pdf" pid=703 comm="apparmor_parser"
[    9.431302] audit: type=1400 audit(1495508974.051:10): apparmor="STATUS" operation="profile_load" name="/usr/sbin/cupsd" pid=703 comm="apparmor_parser"
[    9.431319] audit: type=1400 audit(1495508974.051:11): apparmor="STATUS" operation="profile_load" name="/usr/sbin/cupsd//third_party" pid=703 comm="apparmor_parser"
[   10.025664] Adding 8348668k swap on /dev/sda5.  Priority:-1 extents:1 across:8348668k FS
[   13.998301] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[   14.017392] NFSD: starting 90-second grace period (net ffffffff81d06400)
[   14.637236] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   14.850054] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   17.689488] e1000e: enp0s31f6 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx
[   17.689490] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[   17.689534] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready
===================================================================================================
root@jenkins-MS-7984:~# lspci -s 01:00.0 -vvv 
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at <ignored> (64-bit, prefetchable)
	Region 2: Memory at 8b400000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at e000 [size=256]
	Region 5: Memory at 8b600000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [200 v1] #15
	Capabilities: [270 v1] #19
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] #13
	Capabilities: [2d0 v1] #1b
	Capabilities: [328 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [330 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 256, stride: 1, Device ID: 692f
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
		Region 2: Memory at 000000008d800000 (64-bit, prefetchable)
		Region 5: Memory at 8b800000 (32-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [400 v1] Vendor Specific Information: ID=0002 Rev=1 Len=070 <?>
	Kernel modules: amdgpu

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Deucher, Alexander]

> -----Original Message-----
> From: Cheng, Collins
> Sent: Thursday, May 11, 2017 10:51 PM
> To: Bjorn Helgaas; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org
> Cc: Deucher, Alexander; Zytaruk, Kelly
> Subject: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
> incapable platform
> 
> Hi Helgaas,
> 
> Some AMD GPUs have hardware support for graphics SR-IOV.
> If the SR-IOV capable GPU is plugged into the SR-IOV incapable
> platform. It would cause a problem on PCI resource allocation in
> current Linux kernel.
> 
> Therefore in order to allow the PF (Physical Function) device of
> SR-IOV capable GPU to work on the SR-IOV incapable platform,
> it is required to verify conditions for initializing BAR resources
> on AMD SR-IOV capable GPUs.
> 
> If the device is an AMD graphics device and it supports
> SR-IOV it will require a large amount of resources.
> Before calling sriov_init() must ensure that the system
> BIOS also supports SR-IOV and that system BIOS has been
> able to allocate enough resources.
> If the VF BARs are zero then the system BIOS does not
> support SR-IOV or it could not allocate the resources
> and this platform will not support AMD graphics SR-IOV.
> Therefore do not call sriov_init().
> If the system BIOS does support SR-IOV then the VF BARs
> will be properly initialized to non-zero values.
> 
> Below is the patch against to Kernel 4.8 & 4.9. Please review.

For upstream, the patch should be against Linus' master or the Bjorn's pci-next tree.

Alex

> 
> I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in
> quirks.c are for specific devices and one or more device ID are defined
> for the specific devices. However my patch is for all AMD SR-IOV
> capable GPUs, that includes all existing and future AMD server GPUs.
> So it doesn't seem like a good fit to put the fix in quirks.c.
> 
> 
> 
> Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> ---
>  drivers/pci/iov.c | 63
> ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 60 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index e30f05c..e4f1405 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
>  		msleep(100);
>  }
> 
> +/*
> + * pci_vf_bar_valid - check if VF BARs have resource allocated
> + * @dev: the PCI device
> + * @pos: register offset of SR-IOV capability in PCI config space
> + * Returns true any VF BAR has resource allocated, false
> + * if all VF BARs are empty.
> + */
> +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos)
> +{
> +	int i;
> +	u32 bar_value;
> +	u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> +
> +	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4,
> &bar_value);
> +		if (bar_value & bar_size_mask)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
> +/*
> + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> + * @dev: the PCI device
> + *
> + * Returns true if device is an AMD/ATI display adapter,
> + * otherwise return false.
> + */
> +
> +static bool is_amd_display_adapter(struct pci_dev *dev)
> +{
> +	return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> +		(dev->vendor == PCI_VENDOR_ID_ATI ||
> +		dev->vendor == PCI_VENDOR_ID_AMD));
> +}
> +
>  /**
>   * pci_iov_init - initialize the IOV capability
>   * @dev: the PCI device
> @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>  		return -ENODEV;
> 
>  	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> -	if (pos)
> -		return sriov_init(dev, pos);
> -
> +	if (pos) {
> +	/*
> +	 * If the device is an AMD graphics device and it supports
> +	 * SR-IOV it will require a large amount of resources.
> +	 * Before calling sriov_init() must ensure that the system
> +	 * BIOS also supports SR-IOV and that system BIOS has been
> +	 * able to allocate enough resources.
> +	 * If the VF BARs are zero then the system BIOS does not
> +	 * support SR-IOV or it could not allocate the resources
> +	 * and this platform will not support AMD graphics SR-IOV.
> +	 * Therefore do not call sriov_init().
> +	 * If the system BIOS does support SR-IOV then the VF BARs
> +	 * will be properly initialized to non-zero values.
> +	 */
> +		if (is_amd_display_adapter(dev)) {
> +			if (pci_vf_bar_valid(dev, pos))
> +				return sriov_init(dev, pos);
> +		} else {
> +			return sriov_init(dev, pos);
> +		}
> +	}
>  	return -ENODEV;
>  }
> 
> --
> 1.9.1
> 
> 
> 
> -Collins Cheng

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Deucher, Alexander]

> -----Original Message-----
> From: Cheng, Collins
> Sent: Thursday, May 11, 2017 10:51 PM
> To: Bjorn Helgaas; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.or=
g
> Cc: Deucher, Alexander; Zytaruk, Kelly
> Subject: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV
> incapable platform
>=20
> Hi Helgaas,
>=20
> Some AMD GPUs have hardware support for graphics SR-IOV.
> If the SR-IOV capable GPU is plugged into the SR-IOV incapable
> platform. It would cause a problem on PCI resource allocation in
> current Linux kernel.
>=20
> Therefore in order to allow the PF (Physical Function) device of
> SR-IOV capable GPU to work on the SR-IOV incapable platform,
> it is required to verify conditions for initializing BAR resources
> on AMD SR-IOV capable GPUs.
>=20
> If the device is an AMD graphics device and it supports
> SR-IOV it will require a large amount of resources.
> Before calling sriov_init() must ensure that the system
> BIOS also supports SR-IOV and that system BIOS has been
> able to allocate enough resources.
> If the VF BARs are zero then the system BIOS does not
> support SR-IOV or it could not allocate the resources
> and this platform will not support AMD graphics SR-IOV.
> Therefore do not call sriov_init().
> If the system BIOS does support SR-IOV then the VF BARs
> will be properly initialized to non-zero values.
>=20
> Below is the patch against to Kernel 4.8 & 4.9. Please review.

For upstream, the patch should be against Linus' master or the Bjorn's pci-=
next tree.

Alex

>=20
> I checked the drivers/pci/quirks.c, it looks the workarounds/fixes in
> quirks.c are for specific devices and one or more device ID are defined
> for the specific devices. However my patch is for all AMD SR-IOV
> capable GPUs, that includes all existing and future AMD server GPUs.
> So it doesn't seem like a good fit to put the fix in quirks.c.
>=20
>=20
>=20
> Signed-off-by: Collins Cheng <collins.cheng@amd.com>
> ---
>  drivers/pci/iov.c | 63
> ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 60 insertions(+), 3 deletions(-)
>=20
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index e30f05c..e4f1405 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
>  		msleep(100);
>  }
>=20
> +/*
> + * pci_vf_bar_valid - check if VF BARs have resource allocated
> + * @dev: the PCI device
> + * @pos: register offset of SR-IOV capability in PCI config space
> + * Returns true any VF BAR has resource allocated, false
> + * if all VF BARs are empty.
> + */
> +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos)
> +{
> +	int i;
> +	u32 bar_value;
> +	u32 bar_size_mask =3D ~(PCI_BASE_ADDRESS_SPACE |
> +			PCI_BASE_ADDRESS_MEM_TYPE_64 |
> +			PCI_BASE_ADDRESS_MEM_PREFETCH);
> +
> +	for (i =3D 0; i < PCI_SRIOV_NUM_BARS; i++) {
> +		pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4,
> &bar_value);
> +		if (bar_value & bar_size_mask)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
> +/*
> + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> + * @dev: the PCI device
> + *
> + * Returns true if device is an AMD/ATI display adapter,
> + * otherwise return false.
> + */
> +
> +static bool is_amd_display_adapter(struct pci_dev *dev)
> +{
> +	return (((dev->class >> 16) =3D=3D PCI_BASE_CLASS_DISPLAY) &&
> +		(dev->vendor =3D=3D PCI_VENDOR_ID_ATI ||
> +		dev->vendor =3D=3D PCI_VENDOR_ID_AMD));
> +}
> +
>  /**
>   * pci_iov_init - initialize the IOV capability
>   * @dev: the PCI device
> @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>  		return -ENODEV;
>=20
>  	pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> -	if (pos)
> -		return sriov_init(dev, pos);
> -
> +	if (pos) {
> +	/*
> +	 * If the device is an AMD graphics device and it supports
> +	 * SR-IOV it will require a large amount of resources.
> +	 * Before calling sriov_init() must ensure that the system
> +	 * BIOS also supports SR-IOV and that system BIOS has been
> +	 * able to allocate enough resources.
> +	 * If the VF BARs are zero then the system BIOS does not
> +	 * support SR-IOV or it could not allocate the resources
> +	 * and this platform will not support AMD graphics SR-IOV.
> +	 * Therefore do not call sriov_init().
> +	 * If the system BIOS does support SR-IOV then the VF BARs
> +	 * will be properly initialized to non-zero values.
> +	 */
> +		if (is_amd_display_adapter(dev)) {
> +			if (pci_vf_bar_valid(dev, pos))
> +				return sriov_init(dev, pos);
> +		} else {
> +			return sriov_init(dev, pos);
> +		}
> +	}
>  	return -ENODEV;
>  }
>=20
> --
> 1.9.1
>=20
>=20
>=20
> -Collins Cheng

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alexander Duyck]

So Alex Williamson brought up an interesting point. What happens if
you boot with "pci=realloc=off"? Do you see the same issue with it
attempting to reallocate resources? I'm just wondering what the state
of things is if we don't attempt to reallocate resources after the
BIOS has configured them?

- Alex

On Mon, May 22, 2017 at 8:41 PM, Cheng, Collins <Collins.Cheng@amd.com> wrote:
> Hi Alex,
>
> I owe you a dmesg log. Attachment are two log files. 1.txt is without "pci=earlydump", 2.txt is with "pci=earlydump". The platform is an ASUS Z170-A motherboard that doesn't support SR-IOV. The graphics card is AMD FirePro S7150 card which enabled SR-IOV.
>
> You could find the error info like below in both logs. From the log, kernel failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (256MB needed), but kernel reallocated resource for BAR9 which is for VF. You are right, the real bug that is something goes wrong with the reallocation leaving the PF without resources.
>
> [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
> [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
> [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
> [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
> [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
> [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
>
> -Collins Cheng
>
> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Monday, May 22, 2017 11:44 PM
> To: Alexander Duyck <alexander.duyck@gmail.com>
> Cc: Cheng, Collins <Collins.Cheng@amd.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform
>
> On Fri, 19 May 2017 08:43:38 -0700
> Alexander Duyck <alexander.duyck@gmail.com> wrote:
>
>> On Mon, May 15, 2017 at 10:53 AM, Alex Williamson
>> <alex.williamson@redhat.com> wrote:
>> > On Mon, 15 May 2017 08:19:28 +0000
>> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>> >
>> >> Hi Williamson,
>> >>
>> >> We cannot assume BIOS supports SR-IOV, actually only newer server motherboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older server motherboard BIOS doesn't support SR-IOV. This issue would happen if an user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.
>> >
>> > Servers should be supporting SR-IOV for a long time now.  What
>> > really is there to a BIOS supporting SR-IOV anyway, it's simply
>> > reserving sufficient bus number and MMIO resources such that we can
>> > enable the VFs.  This process isn't exclusively reserved for the
>> > BIOS.  Some platforms may choose to only initialize boot devices,
>> > leaving the rest for the OS to program.  The initial proposal here
>> > to disable SR-IOV if not programmed at OS hand-off disables even the
>> > possibility of the OS reallocating resources for this device.
>>
>> There are differences between supporting SR-IOV and supporting SR-IOV
>> on devices with massive resources. I know I have seen NICs that will
>> keep a system from completing POST if SR-IOV is enabled, and MMIO
>> beyond 4G is not. My guess would be that the issues being seen are
>> probably that they disable SR-IOV in the BIOS in such a setup and end
>> up running into issues when they try to boot into the Linux kernel as
>> it goes through and tries to allocate resources for SR-IOV even though
>> it was disabled in the BIOS.
>>
>> It might make sense to add a kernel parameter something like a
>> "pci=nosriov" that would allow for disabling SR-IOV and related
>> resource allocation if that is what we are talking about. That way you
>> could plug in these types of devices into a system with a legacy bios
>> or that doesn't wan to allocate addresses above 32b for MMIO, and this
>> parameter would be all that is needed to disable SR-IOV so you could
>> plug in a NIC that has SR-IOV associated with it.
>
> Hi,
>
> a) I think we're still ignoring the real bug that is something goes wrong with the reallocation leaving the PF without resources.
>
> b) Why does an option to avoid re-allocation need to be sr-iov specific?  Shouldn't pci=realloc=off cover this?
>
> Thanks,
> Alex
>
>> >> I agree that failure to allocate VF resources should leave the device in no worse condition than before it tried. I hope kernel could allocate PF device resource before allocating VF device resource, and keep PF device resource valid and functional if failed to allocate VF device resource.
>> >>
>> >> I will send out dmesg log lspci info tomorrow. Thanks.
>> >
>> > Thanks,
>> > Alex
>> >
>> >> -----Original Message-----
>> >> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> >> Sent: Friday, May 12, 2017 10:43 PM
>> >> To: Cheng, Collins <Collins.Cheng@amd.com>
>> >> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>> >> linux-kernel@vger.kernel.org; Deucher, Alexander
>> >> <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>> >> <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
>> >> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>> >> SR-IOV incapable platform
>> >>
>> >> On Fri, 12 May 2017 04:51:43 +0000
>> >> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>> >>
>> >> > Hi Williamson,
>> >> >
>> >> > I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.
>> >>
>> >> It also masks an underlying bug and creates a maintenance issue that we won't know when it's safe to remove this workaround.  I don't think faster boot is valid rationale, in one case SR-IOV is completely disabled, the other we attempt to allocate the resources the BIOS failed to provide.  I expect this is also a corner case, the BIOS should typically support SR-IOV, therefore this situation should be an exception.
>> >>
>> >> > I did not try to fix the issue from the kernel resource allocation perspective, it is because:
>> >> >     1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
>> >> >     2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.
>> >>
>> >> I think these are the issues we should be trying to solve and I'm
>> >> sure folks on the linux-pci list can help us identify the bug.
>> >> Minimally, failure to allocate VF resources should leave the device
>> >> in no worse condition than before it tried.  Perhaps you could post
>> >> more details about the issue, boot with pci=earlydump, post dmesg
>> >> of a boot where the PF resources are incorrectly re-allocated, and
>> >> include lspci -vvv for the SR-IOV device.  Also, please test with
>> >> the latest upstream kernel, upstream only patches old kernels
>> >> through stable backports of commits to the latest kernel.  Adding
>> >> Yinghai as a resource allocation expert. Thanks,
>> >>
>> >> Alex
>> >>
>> >> > -----Original Message-----
>> >> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> >> > Sent: Friday, May 12, 2017 12:01 PM
>> >> > To: Cheng, Collins <Collins.Cheng@amd.com>
>> >> > Cc: Bjorn Helgaas <bhelgaas@google.com>;
>> >> > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher,
>> >> > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>> >> > <Kelly.Zytaruk@amd.com>
>> >> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>> >> > SR-IOV incapable platform
>> >> >
>> >> > On Fri, 12 May 2017 03:42:46 +0000 "Cheng, Collins"
>> >> > <Collins.Cheng@amd.com> wrote:
>> >> >
>> >> > > Hi Williamson,
>> >> > >
>> >> > > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
>> >> > >
>> >> > > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.
>> >> >
>> >> > So the problem is that something bad happens when the kernel is
>> >> > trying to reallocate resources in order to fulfill the
>> >> > requirements of the VFs, leaving the PF resources incorrectly
>> >> > programmed?  Why not just fix that bug rather than creating
>> >> > special handling for this vendor/class of device which disables
>> >> > any attempt to fixup resources for SR-IOV?  IOW, this patch just
>> >> > avoids the problem for your devices rather than fixing the bug.
>> >> > I'd suggest fixing the bug such that the PF is left in a
>> >> > functional state if the kernel is unable to allocate sufficient
>> >> > resources for the VFs.  Thanks,
>> >> >
>> >> > Alex
>> >> >
>> >> > > -----Original Message-----
>> >> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> >> > > Sent: Friday, May 12, 2017 11:21 AM
>> >> > > To: Cheng, Collins <Collins.Cheng@amd.com>
>> >> > > Cc: Bjorn Helgaas <bhelgaas@google.com>;
>> >> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
>> >> > > Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>> >> > > <Kelly.Zytaruk@amd.com>
>> >> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on
>> >> > > the SR-IOV incapable platform
>> >> > >
>> >> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins"
>> >> > > <Collins.Cheng@amd.com> wrote:
>> >> > >
>> >> > > > Hi Helgaas,
>> >> > > >
>> >> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
>> >> > > > If the SR-IOV capable GPU is plugged into the SR-IOV
>> >> > > > incapable platform. It would cause a problem on PCI resource
>> >> > > > allocation in current Linux kernel.
>> >> > > >
>> >> > > > Therefore in order to allow the PF (Physical Function) device
>> >> > > > of SR-IOV capable GPU to work on the SR-IOV incapable
>> >> > > > platform, it is required to verify conditions for
>> >> > > > initializing BAR resources on AMD SR-IOV capable GPUs.
>> >> > > >
>> >> > > > If the device is an AMD graphics device and it supports
>> >> > > > SR-IOV it will require a large amount of resources.
>> >> > > > Before calling sriov_init() must ensure that the system BIOS
>> >> > > > also supports SR-IOV and that system BIOS has been able to
>> >> > > > allocate enough resources.
>> >> > > > If the VF BARs are zero then the system BIOS does not support
>> >> > > > SR-IOV or it could not allocate the resources and this
>> >> > > > platform will not support AMD graphics SR-IOV.
>> >> > > > Therefore do not call sriov_init().
>> >> > > > If the system BIOS does support SR-IOV then the VF BARs will
>> >> > > > be properly initialized to non-zero values.
>> >> > > >
>> >> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
>> >> > > >
>> >> > > > I checked the drivers/pci/quirks.c, it looks the
>> >> > > > workarounds/fixes in quirks.c are for specific devices and
>> >> > > > one or more device ID are defined for the specific devices.
>> >> > > > However my patch is for all AMD SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
>> >> > > > So it doesn't seem like a good fit to put the fix in quirks.c.
>> >> > >
>> >> > >
>> >> > > Why is an AMD graphics card unique here?  Doesn't sriov_init()
>> >> > > always need to be able to deal with devices of any type where
>> >> > > the BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV
>> >> > > devices can fit their VFs within a minimum bridge aperture,
>> >> > > most cannot.  I don't understand why the VF resource
>> >> > > requirements being exceptionally large dictates that they receive special handling.
>> >> > > Thanks,
>> >> > >
>> >> > > Alex
>> >> > >
>> >> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>> >> > > > ---
>> >> > > >  drivers/pci/iov.c | 63
>> >> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>> >> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
>> >> > > >
>> >> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>> >> > > > e30f05c..e4f1405 100644
>> >> > > > --- a/drivers/pci/iov.c
>> >> > > > +++ b/drivers/pci/iov.c
>> >> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
>> >> > > >                 msleep(100);
>> >> > > >  }
>> >> > > >
>> >> > > > +/*
>> >> > > > + * pci_vf_bar_valid - check if VF BARs have resource
>> >> > > > +allocated
>> >> > > > + * @dev: the PCI device
>> >> > > > + * @pos: register offset of SR-IOV capability in PCI config
>> >> > > > +space
>> >> > > > + * Returns true any VF BAR has resource allocated, false
>> >> > > > + * if all VF BARs are empty.
>> >> > > > + */
>> >> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>> >> > > > +       int i;
>> >> > > > +       u32 bar_value;
>> >> > > > +       u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
>> >> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
>> >> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
>> >> > > > +
>> >> > > > +       for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>> >> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
>> >> > > > +               if (bar_value & bar_size_mask)
>> >> > > > +                       return true;
>> >> > > > +       }
>> >> > > > +
>> >> > > > +       return false;
>> >> > > > +}
>> >> > > > +
>> >> > > > +/*
>> >> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU
>> >> > > > +device
>> >> > > > + * @dev: the PCI device
>> >> > > > + *
>> >> > > > + * Returns true if device is an AMD/ATI display adapter,
>> >> > > > + * otherwise return false.
>> >> > > > + */
>> >> > > > +
>> >> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
>> >> > > > +       return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
>> >> > > > +               (dev->vendor == PCI_VENDOR_ID_ATI ||
>> >> > > > +               dev->vendor == PCI_VENDOR_ID_AMD)); }
>> >> > > > +
>> >> > > >  /**
>> >> > > >   * pci_iov_init - initialize the IOV capability
>> >> > > >   * @dev: the PCI device
>> >> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>> >> > > >                 return -ENODEV;
>> >> > > >
>> >> > > >         pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
>> >> > > > -       if (pos)
>> >> > > > -               return sriov_init(dev, pos);
>> >> > > > -
>> >> > > > +       if (pos) {
>> >> > > > +       /*
>> >> > > > +        * If the device is an AMD graphics device and it supports
>> >> > > > +        * SR-IOV it will require a large amount of resources.
>> >> > > > +        * Before calling sriov_init() must ensure that the system
>> >> > > > +        * BIOS also supports SR-IOV and that system BIOS has been
>> >> > > > +        * able to allocate enough resources.
>> >> > > > +        * If the VF BARs are zero then the system BIOS does not
>> >> > > > +        * support SR-IOV or it could not allocate the resources
>> >> > > > +        * and this platform will not support AMD graphics SR-IOV.
>> >> > > > +        * Therefore do not call sriov_init().
>> >> > > > +        * If the system BIOS does support SR-IOV then the VF BARs
>> >> > > > +        * will be properly initialized to non-zero values.
>> >> > > > +        */
>> >> > > > +               if (is_amd_display_adapter(dev)) {
>> >> > > > +                       if (pci_vf_bar_valid(dev, pos))
>> >> > > > +                               return sriov_init(dev, pos);
>> >> > > > +               } else {
>> >> > > > +                       return sriov_init(dev, pos);
>> >> > > > +               }
>> >> > > > +       }
>> >> > > >         return -ENODEV;
>> >> > > >  }
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alexander Duyck]

So Alex Williamson brought up an interesting point. What happens if
you boot with "pci=3Drealloc=3Doff"? Do you see the same issue with it
attempting to reallocate resources? I'm just wondering what the state
of things is if we don't attempt to reallocate resources after the
BIOS has configured them?

- Alex

On Mon, May 22, 2017 at 8:41 PM, Cheng, Collins <Collins.Cheng@amd.com> wro=
te:
> Hi Alex,
>
> I owe you a dmesg log. Attachment are two log files. 1.txt is without "pc=
i=3Dearlydump", 2.txt is with "pci=3Dearlydump". The platform is an ASUS Z1=
70-A motherboard that doesn't support SR-IOV. The graphics card is AMD Fire=
Pro S7150 card which enabled SR-IOV.
>
> You could find the error info like below in both logs. From the log, kern=
el failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (2=
56MB needed), but kernel reallocated resource for BAR9 which is for VF. You=
 are right, the real bug that is something goes wrong with the reallocation=
 leaving the PF without resources.
>
> [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000=
 64bit pref]
> [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x1000=
0000 64bit pref]
> [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000=
 64bit pref]
> [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x4000=
0000 64bit pref]
> [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfff=
ff 64bit pref]
> [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x0200000=
0]
> [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x020=
00000]
> [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfff=
ff 64bit pref]
> [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000=
]
> [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x0004=
0000]
> [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000=
 pref]
> [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x0002=
0000 pref]
>
> -Collins Cheng
>
> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Monday, May 22, 2017 11:44 PM
> To: Alexander Duyck <alexander.duyck@gmail.com>
> Cc: Cheng, Collins <Collins.Cheng@amd.com>; Bjorn Helgaas <bhelgaas@googl=
e.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, A=
lexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com=
>; Yinghai Lu <yinghai@kernel.org>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV i=
ncapable platform
>
> On Fri, 19 May 2017 08:43:38 -0700
> Alexander Duyck <alexander.duyck@gmail.com> wrote:
>
>> On Mon, May 15, 2017 at 10:53 AM, Alex Williamson
>> <alex.williamson@redhat.com> wrote:
>> > On Mon, 15 May 2017 08:19:28 +0000
>> > "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>> >
>> >> Hi Williamson,
>> >>
>> >> We cannot assume BIOS supports SR-IOV, actually only newer server mot=
herboard BIOS supports SR-IOV. Normal desktop motherboard BIOS or older ser=
ver motherboard BIOS doesn't support SR-IOV. This issue would happen if an =
user plugs our AMD SR-IOV capable GPU card to a normal desktop motherboard.
>> >
>> > Servers should be supporting SR-IOV for a long time now.  What
>> > really is there to a BIOS supporting SR-IOV anyway, it's simply
>> > reserving sufficient bus number and MMIO resources such that we can
>> > enable the VFs.  This process isn't exclusively reserved for the
>> > BIOS.  Some platforms may choose to only initialize boot devices,
>> > leaving the rest for the OS to program.  The initial proposal here
>> > to disable SR-IOV if not programmed at OS hand-off disables even the
>> > possibility of the OS reallocating resources for this device.
>>
>> There are differences between supporting SR-IOV and supporting SR-IOV
>> on devices with massive resources. I know I have seen NICs that will
>> keep a system from completing POST if SR-IOV is enabled, and MMIO
>> beyond 4G is not. My guess would be that the issues being seen are
>> probably that they disable SR-IOV in the BIOS in such a setup and end
>> up running into issues when they try to boot into the Linux kernel as
>> it goes through and tries to allocate resources for SR-IOV even though
>> it was disabled in the BIOS.
>>
>> It might make sense to add a kernel parameter something like a
>> "pci=3Dnosriov" that would allow for disabling SR-IOV and related
>> resource allocation if that is what we are talking about. That way you
>> could plug in these types of devices into a system with a legacy bios
>> or that doesn't wan to allocate addresses above 32b for MMIO, and this
>> parameter would be all that is needed to disable SR-IOV so you could
>> plug in a NIC that has SR-IOV associated with it.
>
> Hi,
>
> a) I think we're still ignoring the real bug that is something goes wrong=
 with the reallocation leaving the PF without resources.
>
> b) Why does an option to avoid re-allocation need to be sr-iov specific? =
 Shouldn't pci=3Drealloc=3Doff cover this?
>
> Thanks,
> Alex
>
>> >> I agree that failure to allocate VF resources should leave the device=
 in no worse condition than before it tried. I hope kernel could allocate P=
F device resource before allocating VF device resource, and keep PF device =
resource valid and functional if failed to allocate VF device resource.
>> >>
>> >> I will send out dmesg log lspci info tomorrow. Thanks.
>> >
>> > Thanks,
>> > Alex
>> >
>> >> -----Original Message-----
>> >> From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> >> Sent: Friday, May 12, 2017 10:43 PM
>> >> To: Cheng, Collins <Collins.Cheng@amd.com>
>> >> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
>> >> linux-kernel@vger.kernel.org; Deucher, Alexander
>> >> <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>> >> <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
>> >> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>> >> SR-IOV incapable platform
>> >>
>> >> On Fri, 12 May 2017 04:51:43 +0000
>> >> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>> >>
>> >> > Hi Williamson,
>> >> >
>> >> > I verified the patch is working for both AMD SR-IOV GPU and Intel S=
R-IOV NIC. I don't think it is redundant to check the VF BAR valid before c=
all sriov_init(), it is safe and saving boot time, also there is no a bette=
r method to know if system BIOS has correctly initialized the SR-IOV capabi=
lity or not.
>> >>
>> >> It also masks an underlying bug and creates a maintenance issue that =
we won't know when it's safe to remove this workaround.  I don't think fast=
er boot is valid rationale, in one case SR-IOV is completely disabled, the =
other we attempt to allocate the resources the BIOS failed to provide.  I e=
xpect this is also a corner case, the BIOS should typically support SR-IOV,=
 therefore this situation should be an exception.
>> >>
>> >> > I did not try to fix the issue from the kernel resource allocation =
perspective, it is because:
>> >> >     1. I am not very familiar with the PCI resource allocation sche=
me in kernel. For example, in sriov_init(), kernel will re-assign the PCI r=
esource for both VF and PF. I don't understand why kernel allocates resourc=
e for VF firstly, then PF. If it is PF firstly, then this issue could be av=
oided.
>> >> >     2. I am not sure if kernel has error handler if PCI resource al=
location failed. In this case, kernel cannot allocate enough resource to PF=
. It should trigger some error handler to either just keep original BAR val=
ues set by system BIOS, or disable this device and log errors.
>> >>
>> >> I think these are the issues we should be trying to solve and I'm
>> >> sure folks on the linux-pci list can help us identify the bug.
>> >> Minimally, failure to allocate VF resources should leave the device
>> >> in no worse condition than before it tried.  Perhaps you could post
>> >> more details about the issue, boot with pci=3Dearlydump, post dmesg
>> >> of a boot where the PF resources are incorrectly re-allocated, and
>> >> include lspci -vvv for the SR-IOV device.  Also, please test with
>> >> the latest upstream kernel, upstream only patches old kernels
>> >> through stable backports of commits to the latest kernel.  Adding
>> >> Yinghai as a resource allocation expert. Thanks,
>> >>
>> >> Alex
>> >>
>> >> > -----Original Message-----
>> >> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> >> > Sent: Friday, May 12, 2017 12:01 PM
>> >> > To: Cheng, Collins <Collins.Cheng@amd.com>
>> >> > Cc: Bjorn Helgaas <bhelgaas@google.com>;
>> >> > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher,
>> >> > Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>> >> > <Kelly.Zytaruk@amd.com>
>> >> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
>> >> > SR-IOV incapable platform
>> >> >
>> >> > On Fri, 12 May 2017 03:42:46 +0000 "Cheng, Collins"
>> >> > <Collins.Cheng@amd.com> wrote:
>> >> >
>> >> > > Hi Williamson,
>> >> > >
>> >> > > GPU card needs more BAR aperture resource than other PCI devices.=
 For example, Intel SR-IOV network card only require 512KB memory resource =
for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF =3D 4GB memory resource=
 for frame buffer BAR aperture.
>> >> > >
>> >> > > If the system BIOS supports SR-IOV, it will reserve enough resour=
ce for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot all=
ocate the enough resource for VF BARs, only PF BAR will be assigned and VF =
BARs are empty. Then system boots to Linux kernel and kernel doesn't check =
if the VF BARs are empty or valid. Kernel will re-assign the BAR resources =
for PF and all VFs. The problem I saw is that kernel will fail to allocate =
PF BAR resource because some resources are assigned to VF, this is not expe=
cted. So kernel might need to do some check before re-assign the PF/VF reso=
urce, so that PF device will be correctly assigned BAR resource and user ca=
n use PF device.
>> >> >
>> >> > So the problem is that something bad happens when the kernel is
>> >> > trying to reallocate resources in order to fulfill the
>> >> > requirements of the VFs, leaving the PF resources incorrectly
>> >> > programmed?  Why not just fix that bug rather than creating
>> >> > special handling for this vendor/class of device which disables
>> >> > any attempt to fixup resources for SR-IOV?  IOW, this patch just
>> >> > avoids the problem for your devices rather than fixing the bug.
>> >> > I'd suggest fixing the bug such that the PF is left in a
>> >> > functional state if the kernel is unable to allocate sufficient
>> >> > resources for the VFs.  Thanks,
>> >> >
>> >> > Alex
>> >> >
>> >> > > -----Original Message-----
>> >> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
>> >> > > Sent: Friday, May 12, 2017 11:21 AM
>> >> > > To: Cheng, Collins <Collins.Cheng@amd.com>
>> >> > > Cc: Bjorn Helgaas <bhelgaas@google.com>;
>> >> > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
>> >> > > Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly
>> >> > > <Kelly.Zytaruk@amd.com>
>> >> > > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on
>> >> > > the SR-IOV incapable platform
>> >> > >
>> >> > > On Fri, 12 May 2017 02:50:32 +0000 "Cheng, Collins"
>> >> > > <Collins.Cheng@amd.com> wrote:
>> >> > >
>> >> > > > Hi Helgaas,
>> >> > > >
>> >> > > > Some AMD GPUs have hardware support for graphics SR-IOV.
>> >> > > > If the SR-IOV capable GPU is plugged into the SR-IOV
>> >> > > > incapable platform. It would cause a problem on PCI resource
>> >> > > > allocation in current Linux kernel.
>> >> > > >
>> >> > > > Therefore in order to allow the PF (Physical Function) device
>> >> > > > of SR-IOV capable GPU to work on the SR-IOV incapable
>> >> > > > platform, it is required to verify conditions for
>> >> > > > initializing BAR resources on AMD SR-IOV capable GPUs.
>> >> > > >
>> >> > > > If the device is an AMD graphics device and it supports
>> >> > > > SR-IOV it will require a large amount of resources.
>> >> > > > Before calling sriov_init() must ensure that the system BIOS
>> >> > > > also supports SR-IOV and that system BIOS has been able to
>> >> > > > allocate enough resources.
>> >> > > > If the VF BARs are zero then the system BIOS does not support
>> >> > > > SR-IOV or it could not allocate the resources and this
>> >> > > > platform will not support AMD graphics SR-IOV.
>> >> > > > Therefore do not call sriov_init().
>> >> > > > If the system BIOS does support SR-IOV then the VF BARs will
>> >> > > > be properly initialized to non-zero values.
>> >> > > >
>> >> > > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
>> >> > > >
>> >> > > > I checked the drivers/pci/quirks.c, it looks the
>> >> > > > workarounds/fixes in quirks.c are for specific devices and
>> >> > > > one or more device ID are defined for the specific devices.
>> >> > > > However my patch is for all AMD SR-IOV capable GPUs, that inclu=
des all existing and future AMD server GPUs.
>> >> > > > So it doesn't seem like a good fit to put the fix in quirks.c.
>> >> > >
>> >> > >
>> >> > > Why is an AMD graphics card unique here?  Doesn't sriov_init()
>> >> > > always need to be able to deal with devices of any type where
>> >> > > the BIOS hasn't initialized the SR-IOV capability?  Some SR-IOV
>> >> > > devices can fit their VFs within a minimum bridge aperture,
>> >> > > most cannot.  I don't understand why the VF resource
>> >> > > requirements being exceptionally large dictates that they receive=
 special handling.
>> >> > > Thanks,
>> >> > >
>> >> > > Alex
>> >> > >
>> >> > > > Signed-off-by: Collins Cheng <collins.cheng@amd.com>
>> >> > > > ---
>> >> > > >  drivers/pci/iov.c | 63
>> >> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
>> >> > > >  1 file changed, 60 insertions(+), 3 deletions(-)
>> >> > > >
>> >> > > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
>> >> > > > e30f05c..e4f1405 100644
>> >> > > > --- a/drivers/pci/iov.c
>> >> > > > +++ b/drivers/pci/iov.c
>> >> > > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci=
_dev *dev)
>> >> > > >                 msleep(100);
>> >> > > >  }
>> >> > > >
>> >> > > > +/*
>> >> > > > + * pci_vf_bar_valid - check if VF BARs have resource
>> >> > > > +allocated
>> >> > > > + * @dev: the PCI device
>> >> > > > + * @pos: register offset of SR-IOV capability in PCI config
>> >> > > > +space
>> >> > > > + * Returns true any VF BAR has resource allocated, false
>> >> > > > + * if all VF BARs are empty.
>> >> > > > + */
>> >> > > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
>> >> > > > +       int i;
>> >> > > > +       u32 bar_value;
>> >> > > > +       u32 bar_size_mask =3D ~(PCI_BASE_ADDRESS_SPACE |
>> >> > > > +                       PCI_BASE_ADDRESS_MEM_TYPE_64 |
>> >> > > > +                       PCI_BASE_ADDRESS_MEM_PREFETCH);
>> >> > > > +
>> >> > > > +       for (i =3D 0; i < PCI_SRIOV_NUM_BARS; i++) {
>> >> > > > +               pci_read_config_dword(dev, pos + PCI_SRIOV_BAR =
+ i * 4, &bar_value);
>> >> > > > +               if (bar_value & bar_size_mask)
>> >> > > > +                       return true;
>> >> > > > +       }
>> >> > > > +
>> >> > > > +       return false;
>> >> > > > +}
>> >> > > > +
>> >> > > > +/*
>> >> > > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU
>> >> > > > +device
>> >> > > > + * @dev: the PCI device
>> >> > > > + *
>> >> > > > + * Returns true if device is an AMD/ATI display adapter,
>> >> > > > + * otherwise return false.
>> >> > > > + */
>> >> > > > +
>> >> > > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
>> >> > > > +       return (((dev->class >> 16) =3D=3D PCI_BASE_CLASS_DISPL=
AY) &&
>> >> > > > +               (dev->vendor =3D=3D PCI_VENDOR_ID_ATI ||
>> >> > > > +               dev->vendor =3D=3D PCI_VENDOR_ID_AMD)); }
>> >> > > > +
>> >> > > >  /**
>> >> > > >   * pci_iov_init - initialize the IOV capability
>> >> > > >   * @dev: the PCI device
>> >> > > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
>> >> > > >                 return -ENODEV;
>> >> > > >
>> >> > > >         pos =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRI=
OV);
>> >> > > > -       if (pos)
>> >> > > > -               return sriov_init(dev, pos);
>> >> > > > -
>> >> > > > +       if (pos) {
>> >> > > > +       /*
>> >> > > > +        * If the device is an AMD graphics device and it suppo=
rts
>> >> > > > +        * SR-IOV it will require a large amount of resources.
>> >> > > > +        * Before calling sriov_init() must ensure that the sys=
tem
>> >> > > > +        * BIOS also supports SR-IOV and that system BIOS has b=
een
>> >> > > > +        * able to allocate enough resources.
>> >> > > > +        * If the VF BARs are zero then the system BIOS does no=
t
>> >> > > > +        * support SR-IOV or it could not allocate the resource=
s
>> >> > > > +        * and this platform will not support AMD graphics SR-I=
OV.
>> >> > > > +        * Therefore do not call sriov_init().
>> >> > > > +        * If the system BIOS does support SR-IOV then the VF B=
ARs
>> >> > > > +        * will be properly initialized to non-zero values.
>> >> > > > +        */
>> >> > > > +               if (is_amd_display_adapter(dev)) {
>> >> > > > +                       if (pci_vf_bar_valid(dev, pos))
>> >> > > > +                               return sriov_init(dev, pos);
>> >> > > > +               } else {
>> >> > > > +                       return sriov_init(dev, pos);
>> >> > > > +               }
>> >> > > > +       }
>> >> > > >         return -ENODEV;
>> >> > > >  }
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alex Williamson]

On Tue, 23 May 2017 03:41:21 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex,
> 
> I owe you a dmesg log. Attachment are two log files. 1.txt is without "pci=earlydump", 2.txt is with "pci=earlydump". The platform is an ASUS Z170-A motherboard that doesn't support SR-IOV. The graphics card is AMD FirePro S7150 card which enabled SR-IOV. 
> 
> You could find the error info like below in both logs. From the log, kernel failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (256MB needed), but kernel reallocated resource for BAR9 which is for VF. You are right, the real bug that is something goes wrong with the reallocation leaving the PF without resources.
> 
> [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
> [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
> [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
> [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
> [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
> [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]

I've tried to extract more of the relevant resizing efforts below,
perhaps Yinghai or others can make more out of it.  In particular this
system offers no 64-bit MMIO and we'll never manage to allocate the
necessary SR-IOV resources without it.  AIUI, the PCI core won't try to
use anything outside the ACPI _CRS data without the option pci=nocrs.
This might present a second alternative in addition to the
pci=realloc=off, which is actually suggested by the kernel below.  So I
think we have at least two potential workarounds in the code as it
exists today, one leaving SR-IOV disabled, the other (hopefully)
enabling it using 64bit MMIO not described by the system BIOS.
Certainly an improvement would still be detecting the impossible
reallocation problem without nocrs and abandoning the process and
of course to revert the process before leaving more BARs unprogrammed
than we started with.  Thanks,

Alex

[    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
[    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
[    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
...
[    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.896590] pci 0000:01:00.0: supports D1 D2
[    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
[    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
[    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
[    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
[    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
...
[    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.943586] vgaarb: loaded
[    0.943586] vgaarb: bridge control possible 0000:01:00.0
...
[    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
[    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
...
[    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
[    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
[    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] released
[    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64bit pref] released
[    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adfffff]
[    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
[    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
[    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
[    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff]
...
[    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
[    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] released
[    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] released
[    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64bit pref] released
[    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 64bit pref] to [bus 01] add_size 48000000 add_align 8000000
[    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] to [bus 01] add_size 2200000 add_align 400000
[    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
[    0.997692] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0x5fffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0xa7ffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x022fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997694] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x044fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997695] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997696] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x004fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 64bit pref]
[    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000000 64bit pref]
[    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cffffff]
[    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88afffff]
[    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3fffff 64bit pref]
[    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abfffff]
[    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acfffff]
[    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aefffff 64bit pref]
[    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0fffff 64bit pref] (expanded by 0x200000)
[    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7fffff] (expanded by 0x2200000)
[    0.997719] pci 0000:01:00.0: res[7]=[mem size 0x00000000 64bit pref] res_to_dev_res add_size 40000000 min_align 0
[    0.997720] pci 0000:01:00.0: res[9]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 2000000 min_align 0
[    0.997721] pci 0000:01:00.0: res[12]=[mem size 0x00000000] res_to_dev_res add_size 2000000 min_align 0
[    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff 64bit pref]
[    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3fffff]
[    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff 64bit pref]
[    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff]
[    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff 64bit pref]
[    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff]
[    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7fffff]
[    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff]
...
[    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if
you have problem, try booting with pci=realloc=off

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

[-- Attachment #1: Type: text/plain, Size: 14274 bytes --]

Hi Alex W, Alex D,

I just tried two options, one is enable "Above 4G Decoding" in BIOS setup menu, the other is add "pci=realloc=off" in grub. Both can fix this issue. Please see the attached log files.

Previously I thought "Above 4G Decoding" is enabled, but it is off when I looked CMOS setup today.

For now I think we have a solution. For the system that supports "Above 4G Decoding", user should enable it when use a SR-IOV supported device. For the system that doesn't support "Above 4G Decoding", user needs to add "pci=realloc=off" in grub.

Potentially I think kernel still needs to find a way to avoid this issue happen, like keeps the resource as the BIOS assigned value if there is a failure on device's resource reallocation.


-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com] 
Sent: Wednesday, May 24, 2017 2:20 AM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Tue, 23 May 2017 03:41:21 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex,
> 
> I owe you a dmesg log. Attachment are two log files. 1.txt is without "pci=earlydump", 2.txt is with "pci=earlydump". The platform is an ASUS Z170-A motherboard that doesn't support SR-IOV. The graphics card is AMD FirePro S7150 card which enabled SR-IOV. 
> 
> You could find the error info like below in both logs. From the log, kernel failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (256MB needed), but kernel reallocated resource for BAR9 which is for VF. You are right, the real bug that is something goes wrong with the reallocation leaving the PF without resources.
> 
> [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
> [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
> [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
> [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
> [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
> [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]

I've tried to extract more of the relevant resizing efforts below, perhaps Yinghai or others can make more out of it.  In particular this system offers no 64-bit MMIO and we'll never manage to allocate the necessary SR-IOV resources without it.  AIUI, the PCI core won't try to use anything outside the ACPI _CRS data without the option pci=nocrs.
This might present a second alternative in addition to the pci=realloc=off, which is actually suggested by the kernel below.  So I think we have at least two potential workarounds in the code as it exists today, one leaving SR-IOV disabled, the other (hopefully) enabling it using 64bit MMIO not described by the system BIOS.
Certainly an improvement would still be detecting the impossible reallocation problem without nocrs and abandoning the process and of course to revert the process before leaving more BARs unprogrammed than we started with.  Thanks,

Alex

[    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
[    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
[    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
...
[    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.896590] pci 0000:01:00.0: supports D1 D2
[    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
[    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
[    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
[    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
[    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
...
[    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.943586] vgaarb: loaded
[    0.943586] vgaarb: bridge control possible 0000:01:00.0
...
[    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
[    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
...
[    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
[    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
[    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] released
[    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64bit pref] released
[    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adfffff]
[    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
[    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
[    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
[    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff]
...
[    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
[    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] released
[    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] released
[    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64bit pref] released
[    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 64bit pref] to [bus 01] add_size 48000000 add_align 8000000
[    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] to [bus 01] add_size 2200000 add_align 400000
[    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
[    0.997692] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0x5fffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0xa7ffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x022fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997694] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x044fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997695] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997696] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x004fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 64bit pref]
[    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000000 64bit pref]
[    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cffffff]
[    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88afffff]
[    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3fffff 64bit pref]
[    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abfffff]
[    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acfffff]
[    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aefffff 64bit pref]
[    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0fffff 64bit pref] (expanded by 0x200000)
[    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7fffff] (expanded by 0x2200000)
[    0.997719] pci 0000:01:00.0: res[7]=[mem size 0x00000000 64bit pref] res_to_dev_res add_size 40000000 min_align 0
[    0.997720] pci 0000:01:00.0: res[9]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 2000000 min_align 0
[    0.997721] pci 0000:01:00.0: res[12]=[mem size 0x00000000] res_to_dev_res add_size 2000000 min_align 0
[    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff 64bit pref]
[    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3fffff]
[    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff 64bit pref]
[    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff]
[    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff 64bit pref]
[    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff]
[    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7fffff]
[    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff]
...
[    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if
you have problem, try booting with pci=realloc=off

[-- Attachment #2: pci_realloc_off.txt --]
[-- Type: text/plain, Size: 74483 bytes --]

[    0.000000] Linux version 4.9.0-custom (root@rexzhu) (gcc version 4.8.2 (GCC) ) #2 SMP Fri Mar 31 12:52:40 CST 2017
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-custom root=UUID=900c4a92-9f8e-44be-86bc-9673dd3b0d78 ro console=ttyS0,115200,8N1 console=tty0 no_console_suspend pci=realloc=off quiet splash vt.handoff=7
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009c7ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009c800-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007d438fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007d439000-0x000000007d439fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000007d43a000-0x000000007d483fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007d484000-0x000000007d4c4fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007d4c5000-0x000000007db49fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007db4a000-0x00000000858a7fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000858a8000-0x0000000086db6fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000086db7000-0x0000000086deffff] ACPI data
[    0.000000] BIOS-e820: [mem 0x0000000086df0000-0x000000008774bfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000008774c000-0x0000000087ffefff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000087fff000-0x0000000087ffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000880fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000004767fffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 3.0 present.
[    0.000000] DMI: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x476800 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 00C0000000 mask 7FC0000000 uncachable
[    0.000000]   1 base 00A0000000 mask 7FE0000000 uncachable
[    0.000000]   2 base 0090000000 mask 7FF0000000 uncachable
[    0.000000]   3 base 008C000000 mask 7FFC000000 uncachable
[    0.000000]   4 base 008A000000 mask 7FFE000000 uncachable
[    0.000000]   5 base 0089000000 mask 7FFF000000 uncachable
[    0.000000]   6 base 0088800000 mask 7FFF800000 uncachable
[    0.000000]   7 disabled
[    0.000000]   8 disabled
[    0.000000]   9 disabled
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT  
[    0.000000] e820: last_pfn = 0x88000 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000fcc50-0x000fcc5f] mapped at [ffff8800000fcc50]
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff880000096000] 96000 size 24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] BRK [0x020c7000, 0x020c7fff] PGTABLE
[    0.000000] BRK [0x020c8000, 0x020c8fff] PGTABLE
[    0.000000] BRK [0x020c9000, 0x020c9fff] PGTABLE
[    0.000000] BRK [0x020ca000, 0x020cafff] PGTABLE
[    0.000000] BRK [0x020cb000, 0x020cbfff] PGTABLE
[    0.000000] BRK [0x020cc000, 0x020ccfff] PGTABLE
[    0.000000] RAMDISK: [mem 0x2f6de000-0x33b66fff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000F0580 000024 (v02 ALASKA)
[    0.000000] ACPI: XSDT 0x0000000086DC2098 0000AC (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FACP 0x0000000086DE5038 00010C (v05 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x0000000086DC21E0 022E52 (v02 ALASKA A M I    01072009 INTL 20120913)
[    0.000000] ACPI: FACS 0x000000008774BF80 000040
[    0.000000] ACPI: APIC 0x0000000086DE5148 0000BC (v03 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FPDT 0x0000000086DE5208 000044 (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FIDT 0x0000000086DE5250 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: MCFG 0x0000000086DE52F0 00003C (v01 ALASKA A M I    01072009 MSFT 00000097)
[    0.000000] ACPI: HPET 0x0000000086DE5330 000038 (v01 ALASKA A M I    01072009 AMI. 0005000B)
[    0.000000] ACPI: SSDT 0x0000000086DE5368 00036D (v01 SataRe SataTabl 00001000 INTL 20120913)
[    0.000000] ACPI: LPIT 0x0000000086DE56D8 000094 (v01 INTEL  SKL      00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000086DE5770 000248 (v02 INTEL  sensrhub 00000000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE59B8 002BAE (v02 INTEL  PtidDevc 00001000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE8568 000C45 (v02 INTEL  Ther_Rvp 00001000 INTL 20120913)
[    0.000000] ACPI: DBGP 0x0000000086DE91B0 000034 (v01 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: DBG2 0x0000000086DE91E8 000054 (v00 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000086DE9240 0006FD (v02 INTEL  xh_rvp08 00000000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE9940 005271 (v02 SaSsdt SaSsdt   00003000 INTL 20120913)
[    0.000000] ACPI: UEFI 0x0000000086DEEBB8 000042 (v01                 00000000      00000000)
[    0.000000] ACPI: SSDT 0x0000000086DEEC00 000E58 (v02 CpuRef CpuSsdt  00003000 INTL 20120913)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x00000004767fffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x4767f8000-0x4767fbfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000004767fffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009bfff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x000000007d438fff]
[    0.000000]   node   0: [mem 0x000000007d484000-0x000000007d4c4fff]
[    0.000000]   node   0: [mem 0x000000007db4a000-0x00000000858a7fff]
[    0.000000]   node   0: [mem 0x0000000087fff000-0x0000000087ffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x00000004767fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000004767fffff]
[    0.000000] On node 0 totalpages: 4176244
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3995 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 8456 pages used for memmap
[    0.000000]   DMA32 zone: 541145 pages, LIFO batch:31
[    0.000000]   Normal zone: 56736 pages used for memmap
[    0.000000]   Normal zone: 3631104 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x1808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009c000-0x0009cfff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d439000-0x7d439fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d43a000-0x7d483fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d4c5000-0x7db49fff]
[    0.000000] PM: Registered nosave memory: [mem 0x858a8000-0x86db6fff]
[    0.000000] PM: Registered nosave memory: [mem 0x86db7000-0x86deffff]
[    0.000000] PM: Registered nosave memory: [mem 0x86df0000-0x8774bfff]
[    0.000000] PM: Registered nosave memory: [mem 0x8774c000-0x87ffefff]
[    0.000000] PM: Registered nosave memory: [mem 0x88000000-0x880fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x88100000-0xdfffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xe0000000-0xefffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfdffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe000000-0xfe010fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe011000-0xfebfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfedfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfeffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff]
[    0.000000] e820: [mem 0x88100000-0xdfffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:8 nr_node_ids:1
[    0.000000] percpu: Embedded 36 pages/cpu @ffff880476400000 s108056 r8192 d31208 u262144
[    0.000000] pcpu-alloc: s108056 r8192 d31208 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 4110967
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-custom root=UUID=900c4a92-9f8e-44be-86bc-9673dd3b0d78 ro console=ttyS0,115200,8N1 console=tty0 no_console_suspend pci=realloc=off quiet splash vt.handoff=7
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 16288164K/16704976K available (8091K kernel code, 1445K rwdata, 3668K rodata, 2220K init, 1180K bss, 416812K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=8.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=8
[    0.000000] NR_IRQS:16640 nr_irqs:2048 16
[    0.000000] 	Offload RCU callbacks from all CPUs
[    0.000000] 	Offload RCU callbacks from CPUs: 0-7.
[    0.000000] spurious 8259A interrupt: IRQ7.
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [ttyS0] enabled
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Detected 3400.000 MHz processor
[    0.000014] Calibrating delay loop (skipped), value calculated using timer frequency.. 6816.00 BogoMIPS (lpj=13632000)
[    0.000016] pid_max: default: 32768 minimum: 301
[    0.000034] ACPI: Core revision 20160831
[    0.049168] ACPI Error: [\_SB_.PCI0.XHC_.RHUB.HS11] Namespace lookup failure, AE_NOT_FOUND (20160831/dswload-210)
[    0.059647] ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20160831/psobject-227)
[    0.068424] ACPI Exception: AE_NOT_FOUND, (SSDT:xh_rvp08) while loading table (20160831/tbxfload-228)
[    0.089854] ACPI Error: 1 table load failures, 7 successful (20160831/tbxfload-246)
[    0.097795] Security Framework initialized
[    0.097796] Yama: becoming mindful.
[    0.097806] AppArmor: AppArmor initialized
[    0.098382] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.100353] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.101205] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.101215] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.101583] CPU: Physical Processor ID: 0
[    0.101584] CPU: Processor Core ID: 0
[    0.101586] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.101587] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.101590] mce: CPU supports 10 MCE banks
[    0.101597] CPU0: Thermal monitoring enabled (TM1)
[    0.101611] process: using mwait in idle threads
[    0.101613] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.101613] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.101866] Freeing SMP alternatives memory: 32K (ffffffff81f96000 - ffffffff81f9e000)
[    0.131639] ftrace: allocating 32207 entries in 126 pages
[    0.140625] smpboot: APIC(0) Converting physical 0 to logical package 0
[    0.140626] smpboot: Max logical packages: 2
[    0.141825] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.144957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[    0.184645] TSC deadline timer enabled
[    0.184650] smpboot: CPU0: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz (family: 0x6, model: 0x5e, stepping: 0x3)
[    0.184652] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver.
[    0.184672] ... version:                4
[    0.184673] ... bit width:              48
[    0.184673] ... generic registers:      4
[    0.184673] ... value mask:             0000ffffffffffff
[    0.184674] ... max period:             00007fffffffffff
[    0.184674] ... fixed-purpose events:   3
[    0.184674] ... event mask:             000000070000000f
[    0.185381] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    0.185490] x86: Booting SMP configuration:
[    0.185490] .... node  #0, CPUs:      #1 #2 #3 #4 #5 #6 #7
[    0.746801] x86: Booted up 1 node, 8 CPUs
[    0.746803] smpboot: Total of 8 processors activated (54535.41 BogoMIPS)
[    0.753703] devtmpfs: initialized
[    0.753816] x86/mm: Memory block size: 128MB
[    0.758362] evm: security.selinux
[    0.758362] evm: security.SMACK64
[    0.758362] evm: security.ima
[    0.758363] evm: security.capability
[    0.758415] PM: Registering ACPI NVS region [mem 0x7d439000-0x7d439fff] (4096 bytes)
[    0.758416] PM: Registering ACPI NVS region [mem 0x86df0000-0x8774bfff] (9814016 bytes)
[    0.759206] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.759250] pinctrl core: initialized pinctrl subsystem
[    0.759485] RTC time:  8:19:19, date: 05/24/17
[    0.759595] NET: Registered protocol family 16
[    0.777397] cpuidle: using governor ladder
[    0.785193] cpuidle: using governor menu
[    0.785282] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.785283] ACPI: bus type PCI registered
[    0.785283] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.785404] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.785405] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[    0.785415] PCI: Using configuration type 1 for base access
[    0.809537] HugeTLB registered 1 GB page size, pre-allocated 0 pages
[    0.809537] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.809839] ACPI: Added _OSI(Module Device)
[    0.809839] ACPI: Added _OSI(Processor Device)
[    0.809840] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.809840] ACPI: Added _OSI(Processor Aggregator Device)
[    0.812137] ACPI: Executed 21 blocks of module-level executable AML code
[    0.831541] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    0.835143] ACPI: Dynamic OEM Table Load:
[    0.835147] ACPI: SSDT 0xFFFF8804625B8800 0006E4 (v02 PmRef  Cpu0Ist  00003000 INTL 20120913)
[    0.835659] ACPI: \_PR_.CPU0: _OSC native thermal LVT Acked
[    0.838349] ACPI: Dynamic OEM Table Load:
[    0.838353] ACPI: SSDT 0xFFFF880464BD0800 00037F (v02 PmRef  Cpu0Cst  00003001 INTL 20120913)
[    0.839508] ACPI: Dynamic OEM Table Load:
[    0.839511] ACPI: SSDT 0xFFFF880464042800 0005AA (v02 PmRef  ApIst    00003000 INTL 20120913)
[    0.840488] ACPI: Dynamic OEM Table Load:
[    0.840491] ACPI: SSDT 0xFFFF880464037600 000119 (v02 PmRef  ApCst    00003000 INTL 20120913)
[    0.848932] ACPI: Interpreter enabled
[    0.848958] ACPI: (supports S0 S3 S4 S5)
[    0.848958] ACPI: Using IOAPIC for interrupt routing
[    0.849002] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.852279] ACPI: Power Resource [PG00] (on)
[    0.853037] ACPI: Power Resource [PG01] (on)
[    0.853751] ACPI: Power Resource [PG02] (on)
[    0.858006] ACPI: Power Resource [WRST] (off)
[    0.858733] ACPI: Power Resource [WRST] (off)
[    0.859463] ACPI: Power Resource [WRST] (off)
[    0.860184] ACPI: Power Resource [WRST] (off)
[    0.860906] ACPI: Power Resource [WRST] (off)
[    0.861641] ACPI: Power Resource [WRST] (off)
[    0.862363] ACPI: Power Resource [WRST] (off)
[    0.863104] ACPI: Power Resource [WRST] (off)
[    0.863824] ACPI: Power Resource [WRST] (off)
[    0.864547] ACPI: Power Resource [WRST] (off)
[    0.865268] ACPI: Power Resource [WRST] (off)
[    0.865997] ACPI: Power Resource [WRST] (off)
[    0.866724] ACPI: Power Resource [WRST] (off)
[    0.867459] ACPI: Power Resource [WRST] (off)
[    0.868180] ACPI: Power Resource [WRST] (off)
[    0.868897] ACPI: Power Resource [WRST] (off)
[    0.869617] ACPI: Power Resource [WRST] (off)
[    0.870340] ACPI: Power Resource [WRST] (off)
[    0.871059] ACPI: Power Resource [WRST] (off)
[    0.871778] ACPI: Power Resource [WRST] (off)
[    0.886352] ACPI: Power Resource [FN00] (off)
[    0.886513] ACPI: Power Resource [FN01] (off)
[    0.886673] ACPI: Power Resource [FN02] (off)
[    0.886831] ACPI: Power Resource [FN03] (off)
[    0.886989] ACPI: Power Resource [FN04] (off)
[    0.888940] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe])
[    0.888944] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    0.889022] acpi PNP0A08:00: _OSC failed (AE_ERROR); disabling ASPM
[    0.890390] PCI host bridge to bus 0000:00
[    0.890391] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.890393] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.890394] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.890395] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
[    0.890396] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
[    0.890397] pci_bus 0000:00: root bus resource [bus 00-fe]
[    0.890403] pci 0000:00:00.0: [8086:191f] type 00 class 0x060000
[    0.890947] pci 0000:00:14.0: [8086:a12f] type 00 class 0x0c0330
[    0.890962] pci 0000:00:14.0: reg 0x10: [mem 0xdff30000-0xdff3ffff 64bit]
[    0.891019] pci 0000:00:14.0: PME# supported from D3hot D3cold
[    0.891239] pci 0000:00:14.0: System wakeup disabled by ACPI
[    0.891324] pci 0000:00:16.0: [8086:a13a] type 00 class 0x078000
[    0.891336] pci 0000:00:16.0: reg 0x10: [mem 0xdff4d000-0xdff4dfff 64bit]
[    0.891382] pci 0000:00:16.0: PME# supported from D3hot
[    0.891647] pci 0000:00:17.0: [8086:a102] type 00 class 0x010601
[    0.891657] pci 0000:00:17.0: reg 0x10: [mem 0xdff48000-0xdff49fff]
[    0.891663] pci 0000:00:17.0: reg 0x14: [mem 0xdff4c000-0xdff4c0ff]
[    0.891668] pci 0000:00:17.0: reg 0x18: [io  0xf050-0xf057]
[    0.891674] pci 0000:00:17.0: reg 0x1c: [io  0xf040-0xf043]
[    0.891679] pci 0000:00:17.0: reg 0x20: [io  0xf020-0xf03f]
[    0.891684] pci 0000:00:17.0: reg 0x24: [mem 0xdff4b000-0xdff4b7ff]
[    0.891717] pci 0000:00:17.0: PME# supported from D3hot
[    0.891979] pci 0000:00:1b.0: [8086:a167] type 01 class 0x060400
[    0.892028] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
[    0.892297] pci 0000:00:1b.0: System wakeup disabled by ACPI
[    0.892381] pci 0000:00:1c.0: [8086:a110] type 01 class 0x060400
[    0.892431] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    0.892700] pci 0000:00:1c.0: System wakeup disabled by ACPI
[    0.892764] pci 0000:00:1c.2: [8086:a112] type 01 class 0x060400
[    0.892813] pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold
[    0.893083] pci 0000:00:1c.2: System wakeup disabled by ACPI
[    0.893161] pci 0000:00:1d.0: [8086:a118] type 01 class 0x060400
[    0.893212] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[    0.893477] pci 0000:00:1d.0: System wakeup disabled by ACPI
[    0.893570] pci 0000:00:1f.0: [8086:a145] type 00 class 0x060100
[    0.893903] pci 0000:00:1f.2: [8086:a121] type 00 class 0x058000
[    0.893912] pci 0000:00:1f.2: reg 0x10: [mem 0xdff44000-0xdff47fff]
[    0.894197] pci 0000:00:1f.3: [8086:a170] type 00 class 0x040300
[    0.894217] pci 0000:00:1f.3: reg 0x10: [mem 0xdff40000-0xdff43fff 64bit]
[    0.894244] pci 0000:00:1f.3: reg 0x20: [mem 0xdff20000-0xdff2ffff 64bit]
[    0.894288] pci 0000:00:1f.3: PME# supported from D3hot D3cold
[    0.894561] pci 0000:00:1f.3: System wakeup disabled by ACPI
[    0.894628] pci 0000:00:1f.4: [8086:a123] type 00 class 0x0c0500
[    0.894674] pci 0000:00:1f.4: reg 0x10: [mem 0xdff4a000-0xdff4a0ff 64bit]
[    0.894743] pci 0000:00:1f.4: reg 0x20: [io  0xf000-0xf01f]
[    0.895059] pci 0000:00:1f.6: [8086:15b8] type 00 class 0x020000
[    0.895078] pci 0000:00:1f.6: reg 0x10: [mem 0xdff00000-0xdff1ffff]
[    0.895178] pci 0000:00:1f.6: PME# supported from D0 D3hot D3cold
[    0.895385] pci 0000:00:1f.6: System wakeup disabled by ACPI
[    0.895553] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.895568] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.895578] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.895584] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.895591] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.895597] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.895662] pci 0000:01:00.0: supports D1 D2
[    0.895662] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.895697] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
[    0.895698] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.895707] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
[    0.895707] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
[    0.895718] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.895719] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
[    0.895772] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.905564] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.905569] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.905574] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.905582] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.905585] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
[    0.905722] pci 0000:02:00.0: [1b21:1242] type 00 class 0x0c0330
[    0.905741] pci 0000:02:00.0: reg 0x10: [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.905847] pci 0000:02:00.0: PME# supported from D3hot D3cold
[    0.905901] pci 0000:02:00.0: System wakeup disabled by ACPI
[    0.917558] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.917561] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.917693] pci 0000:03:00.0: [1b21:1080] type 01 class 0x060400
[    0.917809] pci 0000:03:00.0: supports D1 D2
[    0.917810] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.917854] pci 0000:03:00.0: System wakeup disabled by ACPI
[    0.929581] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.929732] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.929879] acpiphp: Slot [1] registered
[    0.929882] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.932715] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.932829] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.932941] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 10 11 12 14 *15)
[    0.933051] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.933161] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.933271] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.933381] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.933491] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
[    0.934345] ACPI: Enabled 6 GPEs in block 00 to 7F
[    0.934627] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.934628] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.934629] vgaarb: loaded
[    0.934629] vgaarb: bridge control possible 0000:01:00.0
[    0.934900] SCSI subsystem initialized
[    0.934953] libata version 3.00 loaded.
[    0.934984] ACPI: bus type USB registered
[    0.935011] usbcore: registered new interface driver usbfs
[    0.935026] usbcore: registered new interface driver hub
[    0.935048] usbcore: registered new device driver usb
[    0.935177] PCI: Using ACPI for IRQ routing
[    0.962999] PCI: pci_cache_line_size set to 64 bytes
[    0.963036] e820: reserve RAM buffer [mem 0x0009c800-0x0009ffff]
[    0.963038] e820: reserve RAM buffer [mem 0x7d439000-0x7fffffff]
[    0.963039] e820: reserve RAM buffer [mem 0x7d4c5000-0x7fffffff]
[    0.963040] e820: reserve RAM buffer [mem 0x858a8000-0x87ffffff]
[    0.963041] e820: reserve RAM buffer [mem 0x476800000-0x477ffffff]
[    0.963197] NetLabel: Initializing
[    0.963197] NetLabel:  domain hash size = 128
[    0.963198] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.963216] NetLabel:  unlabeled traffic allowed by default
[    0.963343] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    0.963346] hpet0: 8 comparators, 64-bit 24.000000 MHz counter
[    0.965386] clocksource: Switched to clocksource hpet
[    0.975388] VFS: Disk quotas dquot_6.6.0
[    0.975425] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.975556] AppArmor: AppArmor Filesystem Enabled
[    0.975611] pnp: PnP ACPI init
[    0.975896] system 00:00: [io  0x0290-0x029f] has been reserved
[    0.975898] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.976370] pnp 00:01: [dma 0 disabled]
[    0.976436] pnp 00:01: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.976677] system 00:02: [io  0x0680-0x069f] has been reserved
[    0.976679] system 00:02: [io  0xffff] has been reserved
[    0.976680] system 00:02: [io  0xffff] has been reserved
[    0.976681] system 00:02: [io  0xffff] has been reserved
[    0.976683] system 00:02: [io  0x1800-0x18fe] has been reserved
[    0.976684] system 00:02: [io  0x164e-0x164f] has been reserved
[    0.976685] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.976795] system 00:03: [io  0x0800-0x087f] has been reserved
[    0.976796] system 00:03: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.976838] pnp 00:04: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.976907] system 00:05: [io  0x1854-0x1857] has been reserved
[    0.976908] system 00:05: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
[    0.977298] system 00:06: [mem 0xfed10000-0xfed17fff] has been reserved
[    0.977300] system 00:06: [mem 0xfed18000-0xfed18fff] has been reserved
[    0.977301] system 00:06: [mem 0xfed19000-0xfed19fff] has been reserved
[    0.977303] system 00:06: [mem 0xe0000000-0xefffffff] has been reserved
[    0.977304] system 00:06: [mem 0xfed20000-0xfed3ffff] has been reserved
[    0.977306] system 00:06: [mem 0xfed90000-0xfed93fff] has been reserved
[    0.977307] system 00:06: [mem 0xfed45000-0xfed8ffff] has been reserved
[    0.977309] system 00:06: [mem 0xff000000-0xffffffff] has been reserved
[    0.977311] system 00:06: [mem 0xfee00000-0xfeefffff] could not be reserved
[    0.977313] system 00:06: [mem 0xdffc0000-0xdffdffff] has been reserved
[    0.977314] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.977407] system 00:07: [mem 0xfd000000-0xfdabffff] has been reserved
[    0.977408] system 00:07: [mem 0xfdad0000-0xfdadffff] has been reserved
[    0.977410] system 00:07: [mem 0xfdb00000-0xfdffffff] has been reserved
[    0.977412] system 00:07: [mem 0xfe000000-0xfe01ffff] could not be reserved
[    0.977413] system 00:07: [mem 0xfe036000-0xfe03bfff] has been reserved
[    0.977415] system 00:07: [mem 0xfe03d000-0xfe3fffff] has been reserved
[    0.977416] system 00:07: [mem 0xfe410000-0xfe7fffff] has been reserved
[    0.977417] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.979157] system 00:08: [mem 0xfdaf0000-0xfdafffff] has been reserved
[    0.979159] system 00:08: [mem 0xfdae0000-0xfdaeffff] has been reserved
[    0.979161] system 00:08: [mem 0xfdac0000-0xfdacffff] has been reserved
[    0.979162] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.980528] pnp: PnP ACPI: found 9 devices
[    0.988514] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.988542] pci 0000:00:1d.0: bridge window [io  0x1000-0x0fff] to [bus 05] add_size 1000
[    0.988543] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
[    0.988544] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x000fffff] to [bus 05] add_size 200000 add_align 100000
[    0.988548] pci 0000:00:1d.0: res[14]=[mem 0x00100000-0x000fffff] res_to_dev_res add_size 200000 min_align 100000
[    0.988549] pci 0000:00:1d.0: res[14]=[mem 0x00100000-0x002fffff] res_to_dev_res add_size 200000 min_align 100000
[    0.988550] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.988551] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.988551] pci 0000:00:1d.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[    0.988552] pci 0000:00:1d.0: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[    0.988554] pci 0000:00:1d.0: BAR 14: assigned [mem 0x88800000-0x889fffff]
[    0.988557] pci 0000:00:1d.0: BAR 15: assigned [mem 0x88a00000-0x88bfffff 64bit pref]
[    0.988558] pci 0000:00:1d.0: BAR 13: assigned [io  0x2000-0x2fff]
[    0.988564] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.988564] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.988566] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
[    0.988567] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
[    0.988568] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.988569] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.988570] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.988572] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.988574] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.988576] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.988579] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.988581] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.988585] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.988598] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.988604] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.988605] pci 0000:00:1d.0:   bridge window [io  0x2000-0x2fff]
[    0.988608] pci 0000:00:1d.0:   bridge window [mem 0x88800000-0x889fffff]
[    0.988609] pci 0000:00:1d.0:   bridge window [mem 0x88a00000-0x88bfffff 64bit pref]
[    0.988614] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.988615] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.988615] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.988616] pci_bus 0000:00: resource 7 [mem 0x88800000-0xdfffffff window]
[    0.988616] pci_bus 0000:00: resource 8 [mem 0xfd000000-0xfe7fffff window]
[    0.988617] pci_bus 0000:01: resource 0 [io  0xe000-0xefff]
[    0.988618] pci_bus 0000:01: resource 1 [mem 0xdfe00000-0xdfefffff]
[    0.988618] pci_bus 0000:01: resource 2 [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.988619] pci_bus 0000:02: resource 1 [mem 0xdfd00000-0xdfdfffff]
[    0.988620] pci_bus 0000:05: resource 0 [io  0x2000-0x2fff]
[    0.988620] pci_bus 0000:05: resource 1 [mem 0x88800000-0x889fffff]
[    0.988621] pci_bus 0000:05: resource 2 [mem 0x88a00000-0x88bfffff 64bit pref]
[    0.988812] NET: Registered protocol family 2
[    0.989011] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.989130] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.989207] TCP: Hash tables configured (established 131072 bind 65536)
[    0.989229] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    0.989258] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    0.989349] NET: Registered protocol family 1
[    0.990037] pci 0000:01:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.990715] PCI: CLS 0 bytes, default 64
[    0.990759] Trying to unpack rootfs image as initramfs...
[    1.718028] Freeing initrd memory: 70180K (ffff88002f6de000 - ffff880033b67000)
[    1.718030] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.718032] software IO TLB [mem 0x818a8000-0x858a8000] (64MB) mapped at [ffff8800818a8000-ffff8800858a7fff]
[    1.718247] RAPL PMU: API unit is 2^-32 Joules, 5 fixed counters, 655360 ms ovfl timer
[    1.718247] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
[    1.718248] RAPL PMU: hw unit of domain package 2^-14 Joules
[    1.718248] RAPL PMU: hw unit of domain dram 2^-14 Joules
[    1.718248] RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules
[    1.718249] RAPL PMU: hw unit of domain psys 2^-14 Joules
[    1.718830] Scanning for low memory corruption every 60 seconds
[    1.719405] futex hash table entries: 2048 (order: 5, 131072 bytes)
[    1.719433] audit: initializing netlink subsys (disabled)
[    1.719449] audit: type=2000 audit(1495613959.684:1): initialized
[    1.720201] Initialise system trusted keyrings
[    1.720332] workingset: timestamp_bits=40 max_order=22 bucket_order=0
[    1.724845] fuse init (API version 7.26)
[    1.726239] Key type asymmetric registered
[    1.726239] Asymmetric key parser 'x509' registered
[    1.726302] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    1.726356] io scheduler noop registered
[    1.726356] io scheduler deadline registered (default)
[    1.726364] io scheduler cfq registered
[    1.726855] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    1.726862] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    1.726903] efifb: probing for efifb
[    1.726905] efifb: framebuffer at 0xa0000, using 64k, total 64k
[    1.726906] efifb: mode is 640x480x1, linelength=80, pages=1
[    1.726906] efifb: scrolling: redraw
[    1.726907] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.727650] Console: switching to colour frame buffer device 80x30
[    1.728271] fb0: EFI VGA frame buffer device
[    1.728279] intel_idle: MWAIT substates: 0x142120
[    1.728280] intel_idle: v0.4.1 model 0x5E
[    1.728997] intel_idle: lapic_timer_reliable_states 0xffffffff
[    1.729007] ipmi message handler version 39.2
[    1.729113] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input0
[    1.729115] ACPI: Sleep Button [SLPB]
[    1.729175] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[    1.729177] ACPI: Power Button [PWRB]
[    1.729238] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[    1.729240] ACPI: Power Button [PWRF]
[    1.731017] thermal LNXTHERM:00: registered as thermal_zone0
[    1.731017] ACPI: Thermal Zone [TZ00] (28 C)
[    1.731359] thermal LNXTHERM:01: registered as thermal_zone1
[    1.731360] ACPI: Thermal Zone [TZ01] (30 C)
[    1.731392] GHES: HEST is not enabled!
[    1.731528] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    1.752068] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.755190] Linux agpgart interface v0.103
[    1.761140] brd: module loaded
[    1.764940] loop: module loaded
[    1.765379] libphy: Fixed MDIO Bus: probed
[    1.765795] tun: Universal TUN/TAP device driver, 1.6
[    1.765795] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    1.765974] PPP generic driver version 2.4.2
[    1.766055] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.766058] ehci-pci: EHCI PCI platform driver
[    1.766071] ehci-platform: EHCI generic platform driver
[    1.766081] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.766082] ohci-pci: OHCI PCI platform driver
[    1.766095] ohci-platform: OHCI generic platform driver
[    1.766103] uhci_hcd: USB Universal Host Controller Interface driver
[    1.766291] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.766297] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[    1.767379] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x00109810
[    1.767382] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
[    1.767494] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    1.767494] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.767495] usb usb1: Product: xHCI Host Controller
[    1.767496] usb usb1: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.767496] usb usb1: SerialNumber: 0000:00:14.0
[    1.767690] hub 1-0:1.0: USB hub found
[    1.767707] hub 1-0:1.0: 16 ports detected
[    1.782526] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.782530] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[    1.782572] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[    1.782573] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.782574] usb usb2: Product: xHCI Host Controller
[    1.782574] usb usb2: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.782575] usb usb2: SerialNumber: 0000:00:14.0
[    1.782783] hub 2-0:1.0: USB hub found
[    1.782796] hub 2-0:1.0: 10 ports detected
[    1.792243] xhci_hcd 0000:02:00.0: xHCI Host Controller
[    1.792248] xhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 3
[    1.851013] xhci_hcd 0000:02:00.0: hcc params 0x0200eec1 hci version 0x110 quirks 0x00000010
[    1.851354] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
[    1.851355] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.851355] usb usb3: Product: xHCI Host Controller
[    1.851356] usb usb3: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.851357] usb usb3: SerialNumber: 0000:02:00.0
[    1.851565] hub 3-0:1.0: USB hub found
[    1.851573] hub 3-0:1.0: 2 ports detected
[    1.851699] xhci_hcd 0000:02:00.0: xHCI Host Controller
[    1.851703] xhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 4
[    1.851730] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    1.851756] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003
[    1.851757] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.851757] usb usb4: Product: xHCI Host Controller
[    1.851758] usb usb4: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.851758] usb usb4: SerialNumber: 0000:02:00.0
[    1.851966] hub 4-0:1.0: USB hub found
[    1.851974] hub 4-0:1.0: 2 ports detected
[    1.852143] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.855497] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.855500] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.855785] mousedev: PS/2 mouse device common for all mice
[    1.856191] rtc_cmos 00:04: RTC can wake from S4
[    1.856638] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
[    1.856718] rtc_cmos 00:04: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
[    1.856791] device-mapper: uevent: version 1.0.3
[    1.856943] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
[    1.856945] intel_pstate: Intel P-state driver initializing
[    1.857758] intel_pstate: HWP enabled
[    1.858161] ledtrig-cpu: registered to indicate activity on CPUs
[    1.858452] NET: Registered protocol family 10
[    1.858947] NET: Registered protocol family 17
[    1.858954] Key type dns_resolver registered
[    1.859438] microcode: sig=0x506e3, pf=0x2, revision=0x33
[    1.859844] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    1.860185] registered taskstats version 1
[    1.860186] Loading compiled-in X.509 certificates
[    1.862555] Loaded X.509 cert 'Build time autogenerated kernel key: 425e568c95731e69b073bb62214af700479a67a7'
[    1.862699] kmemleak: Kernel memory leak detector initialized
[    1.862708] kmemleak: Automatic memory scanning thread started
[    1.864477] Key type trusted registered
[    1.868227] Key type encrypted registered
[    1.868229] AppArmor: AppArmor sha1 policy hashing enabled
[    1.868231] ima: No TPM chip found, activating TPM-bypass!
[    1.868248] evm: HMAC attrs: 0x1
[    1.869640]   Magic number: 1:914:321
[    1.869674] tty tty46: hash matches
[    1.869877] rtc_cmos 00:04: setting system clock to 2017-05-24 08:19:20 UTC (1495613960)
[    1.870098] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    1.870099] EDD information not available.
[    1.870231] PM: Hibernation image not present or could not be loaded.
[    1.871058] Freeing unused kernel memory: 2220K (ffffffff81d6b000 - ffffffff81f96000)
[    1.871059] Write protecting the kernel read-only data: 12288k
[    1.871237] Freeing unused kernel memory: 84K (ffff8800017eb000 - ffff880001800000)
[    1.871845] Freeing unused kernel memory: 428K (ffff880001b95000 - ffff880001c00000)
[    1.880495] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.880558] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.880565] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.880885] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.880919] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.881965] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.882004] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.882510] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.882548] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.882584] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.915314] hid: module verification failed: signature and/or required key missing - tainting kernel
[    1.915699] hidraw: raw HID events driver (C) Jiri Kosina
[    1.919698] wmi: Mapper loaded
[    1.934870] ahci 0000:00:17.0: version 3.0
[    1.936657] pps_core: LinuxPPS API ver. 1 registered
[    1.936657] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.937027] PTP clock support registered
[    1.937893] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.937893] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.945531] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[    1.945532] ahci 0000:00:17.0: flags: 64bit ncq sntf led clo only pio slum part ems deso sadm sds apst 
[    2.007163] scsi host0: ahci
[    2.007580] scsi host1: ahci
[    2.007992] scsi host2: ahci
[    2.008366] scsi host3: ahci
[    2.008610] scsi host4: ahci
[    2.008900] scsi host5: ahci
[    2.008999] ata1: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b100 irq 129
[    2.009003] ata2: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b180 irq 129
[    2.009007] ata3: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b200 irq 129
[    2.009012] ata4: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b280 irq 129
[    2.009016] ata5: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b300 irq 129
[    2.009021] ata6: SATA max UDMA/133 abar m2048@0xdff4b000 port 0xdff4b380 irq 129
[    2.009531] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    2.109549] usb 1-10: new full-speed USB device number 2 using xhci_hcd
[    2.252600] usb 1-10: New USB device found, idVendor=046d, idProduct=c534
[    2.252601] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.252601] usb 1-10: Product: USB Receiver
[    2.252602] usb 1-10: Manufacturer: Logitech
[    2.257549] usbcore: registered new interface driver usbhid
[    2.257550] usbhid: USB HID core driver
[    2.258442] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C534.0001/input/input6
[    2.289284] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    2.318000] hid-generic 0003:046D:C534.0001: input,hidraw0: USB HID v1.11 Keyboard [Logitech USB Receiver] on usb-0000:00:14.0-10/input0
[    2.318199] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.1/0003:046D:C534.0002/input/input7
[    2.325110] CPU: 1 PID: 214 Comm: scsi_eh_2 Tainted: G            E   4.9.0-custom #2
[    2.325113] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.325115]  ffffc90001f23ae8 ffffffff813c2d3f ffff8804520a1e80 ffff8804520a22b0
[    2.325122]  ffffc90001f23b10 ffffffff815a7226 ffffc90001f23b10 0000000000000286
[    2.325127]  ffffc90000029200 ffffc90001f23b30 ffffffffa0055634 ffff8804520a1e80
[    2.325132] Call Trace:
[    2.325143]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.325161]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.325163]  [<ffffffffa0055634>] ahci_postreset+0x34/0x60 [libahci]
[    2.325165]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.325166]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325167]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325168]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325169]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325170]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325171]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325172]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.325173]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325174]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325175]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325176]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.325177]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.325179]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.325180]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.325181]  [<ffffffffa0055de8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.325183]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.325183]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.325185]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325186]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.325187]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325188]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325189]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.325189]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.325191]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.325194] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.325549] CPU: 2 PID: 218 Comm: scsi_eh_4 Tainted: G            E   4.9.0-custom #2
[    2.325550] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.325550]  ffffc90001f43ae8 ffffffff813c2d3f ffff8804520a9e80 ffff8804520aa2b0
[    2.325552]  ffffc90001f43b10 ffffffff815a7226 ffffc90001f43b10 0000000000000286
[    2.325553]  ffffc90000029300 ffffc90001f43b30 ffffffffa0055634 ffff8804520a9e80
[    2.325554] Call Trace:
[    2.325556]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.325558]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.325559]  [<ffffffffa0055634>] ahci_postreset+0x34/0x60 [libahci]
[    2.325561]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.325562]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325563]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325565]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325566]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325568]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325569]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325570]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.325571]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325572]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325574]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325574]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.325576]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.325577]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.325578]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.325580]  [<ffffffffa0055de8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.325581]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.325582]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.325583]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325584]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.325585]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325586]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325587]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.325588]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.325589]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.325593] ata5: SATA link down (SStatus 4 SControl 300)
[    2.325613] CPU: 6 PID: 220 Comm: scsi_eh_5 Tainted: G            E   4.9.0-custom #2
[    2.325613] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.325614]  ffffc90001f53ae8 ffffffff813c2d3f ffff8804520ade80 ffff8804520ae2b0
[    2.325615]  ffffc90001f53b10 ffffffff815a7226 ffffc90001f53b10 0000000000000286
[    2.325616]  ffffc90000029380 ffffc90001f53b30 ffffffffa0055634 ffff8804520ade80
[    2.325617] Call Trace:
[    2.325618]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.325619]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.325620]  [<ffffffffa0055634>] ahci_postreset+0x34/0x60 [libahci]
[    2.325621]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.325622]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325623]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325624]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325625]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325626]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325627]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325628]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.325629]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325630]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325631]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325632]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.325633]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.325634]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.325635]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.325636]  [<ffffffffa0055de8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.325637]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.325638]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.325639]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325640]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.325641]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325642]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325643]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.325643]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.325644]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.325647] ata6: SATA link down (SStatus 4 SControl 300)
[    2.325740] CPU: 7 PID: 212 Comm: scsi_eh_1 Tainted: G            E   4.9.0-custom #2
[    2.325740] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.325741]  ffffc90001f03ae8 ffffffff813c2d3f ffff88045209de80 ffff88045209e2b0
[    2.325754]  ffffc90001f03b10 ffffffff815a7226 ffffc90001f03b10 0000000000000286
[    2.325756]  ffffc90000029180 ffffc90001f03b30 ffffffffa0055634 ffff88045209de80
[    2.325757] Call Trace:
[    2.325759]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.325760]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.325762]  [<ffffffffa0055634>] ahci_postreset+0x34/0x60 [libahci]
[    2.325763]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.325764]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325765]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325766]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325767]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325768]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325769]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325770]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.325771]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325772]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325773]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325774]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.325775]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.325777]  [<ffffffff810c6fd1>] ? mutex_optimistic_spin+0x111/0x1a0
[    2.325778]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.325779]  [<ffffffffa0055de8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.325780]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.325781]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.325782]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325783]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.325784]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325785]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325786]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.325787]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.325788]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.325792] ata2: SATA link down (SStatus 4 SControl 300)
[    2.325812] CPU: 4 PID: 210 Comm: scsi_eh_0 Tainted: G            E   4.9.0-custom #2
[    2.325813] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.325813]  ffffc90001ef3ae8 ffffffff813c2d3f ffff88045399de80 ffff88045399e2b0
[    2.325815]  ffffc90001ef3b10 ffffffff815a7226 ffffc90001ef3b10 0000000000000286
[    2.325816]  ffffc90000029100 ffffc90001ef3b30 ffffffffa0055634 ffff88045399de80
[    2.325817] Call Trace:
[    2.325820]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.325821]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.325822]  [<ffffffffa0055634>] ahci_postreset+0x34/0x60 [libahci]
[    2.325824]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.325837]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325839]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325840]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325841]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325842]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325843]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325844]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.325845]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325847]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325848]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325848]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.325850]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.325863]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.325865]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.325866]  [<ffffffffa0055de8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.325867]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.325868]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.325869]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325870]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.325871]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325873]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325873]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.325874]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.325876]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.325879] ata1: SATA link down (SStatus 4 SControl 300)
[    2.325899] CPU: 0 PID: 216 Comm: scsi_eh_3 Tainted: G            E   4.9.0-custom #2
[    2.325899] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.325900]  ffffc90001f33ae8 ffffffff813c2d3f ffff8804520a5e80 ffff8804520a62b0
[    2.325901]  ffffc90001f33b10 ffffffff815a7226 ffffc90001f33b10 0000000000000286
[    2.325902]  ffffc90000029280 ffffc90001f33b30 ffffffffa0055634 ffff8804520a5e80
[    2.325903] Call Trace:
[    2.325904]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.325905]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.325906]  [<ffffffffa0055634>] ahci_postreset+0x34/0x60 [libahci]
[    2.325907]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.325908]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325909]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325910]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325911]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325912]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325913]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325914]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.325915]  [<ffffffffa0055600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.325916]  [<ffffffffa0057350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.325917]  [<ffffffffa0056e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.325917]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.325918]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.325919]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.325921]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.325922]  [<ffffffffa0055de8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.325923]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.325924]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.325925]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325926]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.325926]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325927]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.325928]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.325929]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.325930]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.325933] ata4: SATA link down (SStatus 4 SControl 300)
[    2.326373] ata3.00: ATA-8: ST3500413AS, JC45, max UDMA/133
[    2.326374] ata3.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32)
[    2.327911] ata3.00: configured for UDMA/133
[    2.328381] scsi 2:0:0:0: Direct-Access     ATA      ST3500413AS      JC45 PQ: 0 ANSI: 5
[    2.377744] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 30:5a:3a:e0:7f:4d
[    2.377745] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    2.377818] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: FFFFFF-0FF
[    2.377883] hid-generic 0003:046D:C534.0002: input,hiddev0,hidraw1: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-0000:00:14.0-10/input1
[    2.378786] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
[    2.385815] sd 2:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/466 GiB)
[    2.385886] sd 2:0:0:0: Attached scsi generic sg0 type 0
[    2.386123] sd 2:0:0:0: [sda] Write Protect is off
[    2.386125] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.386270] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.418509]  sda: sda1 sda2 < sda5 >
[    2.420063] sd 2:0:0:0: [sda] Attached SCSI disk
[    2.749441] tsc: Refined TSC clocksource calibration: 3407.999 MHz
[    2.749456] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x311fd336761, max_idle_ns: 440795243819 ns
[    2.764787] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[    2.817394] random: fast init done
[    3.657801] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[    3.657924] systemd[1]: Detected architecture x86-64.
[    3.665736] systemd[1]: Set hostname to <jenkins-MS-7984>.
[    3.773854] clocksource: Switched to clocksource tsc
[    4.330579] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    4.330628] systemd[1]: Listening on Journal Socket.
[    4.330655] systemd[1]: Listening on udev Control Socket.
[    4.330698] systemd[1]: Listening on Journal Audit Socket.
[    4.330717] systemd[1]: Listening on Journal Socket (/dev/log).
[    4.330742] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
[    4.330831] systemd[1]: Created slice System Slice.
[    4.978280] random: crng init done
[    5.119195] lp: driver loaded but no devices found
[    5.126984] ppdev: user-space parallel port driver
[    5.194790] RPC: Registered named UNIX socket transport module.
[    5.194790] RPC: Registered udp transport module.
[    5.194791] RPC: Registered tcp transport module.
[    5.194791] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    5.428972] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    5.788951] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro
[    6.085037] systemd-journald[286]: Received request to flush runtime journal from PID 1
[    6.340341] mei_me 0000:00:16.0: enabling device (0000 -> 0002)
[    6.407237] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    6.598269] AVX2 version of gcm_enc/dec engaged.
[    6.598269] AES CTR mode by8 optimization enabled
[    7.262871] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC892: line_outs=3 (0x14/0x15/0x16/0x0/0x0) type:line
[    7.262873] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    7.262874] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    7.262874] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    7.262875] snd_hda_codec_realtek hdaudioC0D0:    dig-out=0x11/0x1e
[    7.262875] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    7.262876] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[    7.262877] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[    7.262877] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a
[    7.274944] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input8
[    7.275046] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9
[    7.275160] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
[    7.275266] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
[    7.275397] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1f.3/sound/card0/input12
[    7.275499] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13
[    7.275602] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input14
[    8.008061] intel_rapl: Found RAPL domain package
[    8.008063] intel_rapl: Found RAPL domain core
[    8.008065] intel_rapl: Found RAPL domain dram
[    8.662199] asus_wmi: ASUS WMI generic driver loaded
[    8.666571] asus_wmi: Initialization: 0x0
[    8.666640] asus_wmi: BIOS WMI version: 0.9
[    8.666771] asus_wmi: SFUN value: 0x0
[    8.667845] input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input15
[    8.668069] asus_wmi: Number of fans: 1
[    8.863966] audit: type=1400 audit(1495613967.488:2): apparmor="STATUS" operation="profile_load" name="/usr/sbin/ippusbxd" pid=761 comm="apparmor_parser"
[    8.866778] audit: type=1400 audit(1495613967.492:3): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine" pid=758 comm="apparmor_parser"
[    8.866783] audit: type=1400 audit(1495613967.492:4): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=758 comm="apparmor_parser"
[    8.992068] audit: type=1400 audit(1495613967.620:5): apparmor="STATUS" operation="profile_load" name="/usr/lib/lightdm/lightdm-guest-session" pid=753 comm="apparmor_parser"
[    8.992073] audit: type=1400 audit(1495613967.620:6): apparmor="STATUS" operation="profile_load" name="/usr/lib/lightdm/lightdm-guest-session//chromium" pid=753 comm="apparmor_parser"
[    9.001482] audit: type=1400 audit(1495613967.628:7): apparmor="STATUS" operation="profile_load" name="/usr/sbin/cups-browsed" pid=759 comm="apparmor_parser"
[    9.002052] audit: type=1400 audit(1495613967.628:8): apparmor="STATUS" operation="profile_load" name="/usr/lib/cups/backend/cups-pdf" pid=760 comm="apparmor_parser"
[    9.002070] audit: type=1400 audit(1495613967.628:9): apparmor="STATUS" operation="profile_load" name="/usr/sbin/cupsd" pid=760 comm="apparmor_parser"
[    9.002074] audit: type=1400 audit(1495613967.628:10): apparmor="STATUS" operation="profile_load" name="/usr/sbin/cupsd//third_party" pid=760 comm="apparmor_parser"
[    9.018830] audit: type=1400 audit(1495613967.644:11): apparmor="STATUS" operation="profile_load" name="/usr/sbin/tcpdump" pid=768 comm="apparmor_parser"
[    9.546141] Adding 8348668k swap on /dev/sda5.  Priority:-1 extents:1 across:8348668k FS
[   13.779112] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[   13.788162] NFSD: starting 90-second grace period (net ffffffff81d06400)
[   14.722383] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   14.937673] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   17.921145] e1000e: enp0s31f6 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx
[   17.921147] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[   17.921193] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready

===============================================================================================
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at d0000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at e000 [size=256]
	Region 5: Memory at dfe00000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [200 v1] #15
	Capabilities: [270 v1] #19
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] #13
	Capabilities: [2d0 v1] #1b
	Capabilities: [328 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [330 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 256, stride: 1, Device ID: 692f
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
		Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [400 v1] Vendor Specific Information: ID=0002 Rev=1 Len=070 <?>
	Kernel modules: amdgpu


[-- Attachment #3: enable_above_4G_decoding.txt --]
[-- Type: text/plain, Size: 76255 bytes --]

[    0.000000] Linux version 4.9.0-custom (root@rexzhu) (gcc version 4.8.2 (GCC) ) #2 SMP Fri Mar 31 12:52:40 CST 2017
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-custom root=UUID=900c4a92-9f8e-44be-86bc-9673dd3b0d78 ro console=ttyS0,115200,8N1 console=tty0 no_console_suspend quiet splash vt.handoff=7
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009c7ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009c800-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007d438fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007d439000-0x000000007d439fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000007d43a000-0x000000007d483fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007d484000-0x000000007d4c4fff] usable
[    0.000000] BIOS-e820: [mem 0x000000007d4c5000-0x000000007db49fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000007db4a000-0x00000000858a7fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000858a8000-0x0000000086db6fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000086db7000-0x0000000086deffff] ACPI data
[    0.000000] BIOS-e820: [mem 0x0000000086df0000-0x000000008774bfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000008774c000-0x0000000087ffefff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000087fff000-0x0000000087ffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000880fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000004767fffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 3.0 present.
[    0.000000] DMI: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x476800 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 00C0000000 mask 7FC0000000 uncachable
[    0.000000]   1 base 00A0000000 mask 7FE0000000 uncachable
[    0.000000]   2 base 0090000000 mask 7FF0000000 uncachable
[    0.000000]   3 base 008C000000 mask 7FFC000000 uncachable
[    0.000000]   4 base 008A000000 mask 7FFE000000 uncachable
[    0.000000]   5 base 0089000000 mask 7FFF000000 uncachable
[    0.000000]   6 base 0088800000 mask 7FFF800000 uncachable
[    0.000000]   7 base 0800000000 mask 7800000000 uncachable
[    0.000000]   8 base 0FC0000000 mask 7FC0000000 uncachable
[    0.000000]   9 disabled
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT  
[    0.000000] e820: last_pfn = 0x88000 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000fcc60-0x000fcc6f] mapped at [ffff8800000fcc60]
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff880000096000] 96000 size 24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] BRK [0x020c7000, 0x020c7fff] PGTABLE
[    0.000000] BRK [0x020c8000, 0x020c8fff] PGTABLE
[    0.000000] BRK [0x020c9000, 0x020c9fff] PGTABLE
[    0.000000] BRK [0x020ca000, 0x020cafff] PGTABLE
[    0.000000] BRK [0x020cb000, 0x020cbfff] PGTABLE
[    0.000000] BRK [0x020cc000, 0x020ccfff] PGTABLE
[    0.000000] RAMDISK: [mem 0x2f6de000-0x33b66fff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000F0580 000024 (v02 ALASKA)
[    0.000000] ACPI: XSDT 0x0000000086DC2098 0000AC (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FACP 0x0000000086DE5038 00010C (v05 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x0000000086DC21E0 022E52 (v02 ALASKA A M I    01072009 INTL 20120913)
[    0.000000] ACPI: FACS 0x000000008774BF80 000040
[    0.000000] ACPI: APIC 0x0000000086DE5148 0000BC (v03 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FPDT 0x0000000086DE5208 000044 (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: FIDT 0x0000000086DE5250 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: MCFG 0x0000000086DE52F0 00003C (v01 ALASKA A M I    01072009 MSFT 00000097)
[    0.000000] ACPI: HPET 0x0000000086DE5330 000038 (v01 ALASKA A M I    01072009 AMI. 0005000B)
[    0.000000] ACPI: SSDT 0x0000000086DE5368 00036D (v01 SataRe SataTabl 00001000 INTL 20120913)
[    0.000000] ACPI: LPIT 0x0000000086DE56D8 000094 (v01 INTEL  SKL      00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000086DE5770 000248 (v02 INTEL  sensrhub 00000000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE59B8 002BAE (v02 INTEL  PtidDevc 00001000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE8568 000C45 (v02 INTEL  Ther_Rvp 00001000 INTL 20120913)
[    0.000000] ACPI: DBGP 0x0000000086DE91B0 000034 (v01 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: DBG2 0x0000000086DE91E8 000054 (v00 INTEL           00000000 MSFT 0000005F)
[    0.000000] ACPI: SSDT 0x0000000086DE9240 0006FD (v02 INTEL  xh_rvp08 00000000 INTL 20120913)
[    0.000000] ACPI: SSDT 0x0000000086DE9940 005271 (v02 SaSsdt SaSsdt   00003000 INTL 20120913)
[    0.000000] ACPI: UEFI 0x0000000086DEEBB8 000042 (v01                 00000000      00000000)
[    0.000000] ACPI: SSDT 0x0000000086DEEC00 000E58 (v02 CpuRef CpuSsdt  00003000 INTL 20120913)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x00000004767fffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x4767f8000-0x4767fbfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000004767fffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009bfff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x000000007d438fff]
[    0.000000]   node   0: [mem 0x000000007d484000-0x000000007d4c4fff]
[    0.000000]   node   0: [mem 0x000000007db4a000-0x00000000858a7fff]
[    0.000000]   node   0: [mem 0x0000000087fff000-0x0000000087ffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x00000004767fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000004767fffff]
[    0.000000] On node 0 totalpages: 4176244
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3995 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 8456 pages used for memmap
[    0.000000]   DMA32 zone: 541145 pages, LIFO batch:31
[    0.000000]   Normal zone: 56736 pages used for memmap
[    0.000000]   Normal zone: 3631104 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x1808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009c000-0x0009cfff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d439000-0x7d439fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d43a000-0x7d483fff]
[    0.000000] PM: Registered nosave memory: [mem 0x7d4c5000-0x7db49fff]
[    0.000000] PM: Registered nosave memory: [mem 0x858a8000-0x86db6fff]
[    0.000000] PM: Registered nosave memory: [mem 0x86db7000-0x86deffff]
[    0.000000] PM: Registered nosave memory: [mem 0x86df0000-0x8774bfff]
[    0.000000] PM: Registered nosave memory: [mem 0x8774c000-0x87ffefff]
[    0.000000] PM: Registered nosave memory: [mem 0x88000000-0x880fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x88100000-0xdfffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xe0000000-0xefffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfdffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe000000-0xfe010fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe011000-0xfebfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfedfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfeffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff]
[    0.000000] e820: [mem 0x88100000-0xdfffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:8 nr_node_ids:1
[    0.000000] percpu: Embedded 36 pages/cpu @ffff880476400000 s108056 r8192 d31208 u262144
[    0.000000] pcpu-alloc: s108056 r8192 d31208 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 4110967
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-custom root=UUID=900c4a92-9f8e-44be-86bc-9673dd3b0d78 ro console=ttyS0,115200,8N1 console=tty0 no_console_suspend quiet splash vt.handoff=7
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 16288164K/16704976K available (8091K kernel code, 1445K rwdata, 3668K rodata, 2220K init, 1180K bss, 416812K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=8.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=8
[    0.000000] NR_IRQS:16640 nr_irqs:2048 16
[    0.000000] 	Offload RCU callbacks from all CPUs
[    0.000000] 	Offload RCU callbacks from CPUs: 0-7.
[    0.000000] spurious 8259A interrupt: IRQ7.
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [ttyS0] enabled
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Detected 3400.000 MHz processor
[    0.000015] Calibrating delay loop (skipped), value calculated using timer frequency.. 6816.00 BogoMIPS (lpj=13632000)
[    0.000016] pid_max: default: 32768 minimum: 301
[    0.000035] ACPI: Core revision 20160831
[    0.049186] ACPI Error: [\_SB_.PCI0.XHC_.RHUB.HS11] Namespace lookup failure, AE_NOT_FOUND (20160831/dswload-210)
[    0.059664] ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20160831/psobject-227)
[    0.068398] ACPI Exception: AE_NOT_FOUND, (SSDT:xh_rvp08) while loading table (20160831/tbxfload-228)
[    0.089683] ACPI Error: 1 table load failures, 7 successful (20160831/tbxfload-246)
[    0.097611] Security Framework initialized
[    0.097612] Yama: becoming mindful.
[    0.097622] AppArmor: AppArmor initialized
[    0.098204] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.100184] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.101038] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.101049] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.101417] CPU: Physical Processor ID: 0
[    0.101417] CPU: Processor Core ID: 0
[    0.101420] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.101420] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.101424] mce: CPU supports 10 MCE banks
[    0.101431] CPU0: Thermal monitoring enabled (TM1)
[    0.101445] process: using mwait in idle threads
[    0.101447] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.101447] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.101701] Freeing SMP alternatives memory: 32K (ffffffff81f96000 - ffffffff81f9e000)
[    0.132993] ftrace: allocating 32207 entries in 126 pages
[    0.141962] smpboot: APIC(0) Converting physical 0 to logical package 0
[    0.141963] smpboot: Max logical packages: 2
[    0.143163] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.146364] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[    0.186052] TSC deadline timer enabled
[    0.186057] smpboot: CPU0: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz (family: 0x6, model: 0x5e, stepping: 0x3)
[    0.186059] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver.
[    0.186082] ... version:                4
[    0.186083] ... bit width:              48
[    0.186083] ... generic registers:      4
[    0.186083] ... value mask:             0000ffffffffffff
[    0.186084] ... max period:             00007fffffffffff
[    0.186084] ... fixed-purpose events:   3
[    0.186084] ... event mask:             000000070000000f
[    0.186786] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    0.186895] x86: Booting SMP configuration:
[    0.186895] .... node  #0, CPUs:      #1 #2 #3 #4 #5 #6 #7
[    0.748208] x86: Booted up 1 node, 8 CPUs
[    0.748209] smpboot: Total of 8 processors activated (54535.47 BogoMIPS)
[    0.755115] devtmpfs: initialized
[    0.755228] x86/mm: Memory block size: 128MB
[    0.759773] evm: security.selinux
[    0.759773] evm: security.SMACK64
[    0.759774] evm: security.ima
[    0.759774] evm: security.capability
[    0.759825] PM: Registering ACPI NVS region [mem 0x7d439000-0x7d439fff] (4096 bytes)
[    0.759826] PM: Registering ACPI NVS region [mem 0x86df0000-0x8774bfff] (9814016 bytes)
[    0.760617] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.760662] pinctrl core: initialized pinctrl subsystem
[    0.760895] RTC time:  8:08:46, date: 05/24/17
[    0.761005] NET: Registered protocol family 16
[    0.786816] cpuidle: using governor ladder
[    0.794610] cpuidle: using governor menu
[    0.794698] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.794698] ACPI: bus type PCI registered
[    0.794699] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.794820] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[    0.794821] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[    0.794832] PCI: Using configuration type 1 for base access
[    0.818955] HugeTLB registered 1 GB page size, pre-allocated 0 pages
[    0.818955] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.819255] ACPI: Added _OSI(Module Device)
[    0.819255] ACPI: Added _OSI(Processor Device)
[    0.819256] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.819256] ACPI: Added _OSI(Processor Aggregator Device)
[    0.821558] ACPI: Executed 21 blocks of module-level executable AML code
[    0.840956] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    0.844557] ACPI: Dynamic OEM Table Load:
[    0.844562] ACPI: SSDT 0xFFFF8804625B9000 0006E4 (v02 PmRef  Cpu0Ist  00003000 INTL 20120913)
[    0.845071] ACPI: \_PR_.CPU0: _OSC native thermal LVT Acked
[    0.847755] ACPI: Dynamic OEM Table Load:
[    0.847759] ACPI: SSDT 0xFFFF880464BC8800 00037F (v02 PmRef  Cpu0Cst  00003001 INTL 20120913)
[    0.848912] ACPI: Dynamic OEM Table Load:
[    0.848916] ACPI: SSDT 0xFFFF880464042800 0005AA (v02 PmRef  ApIst    00003000 INTL 20120913)
[    0.849903] ACPI: Dynamic OEM Table Load:
[    0.849906] ACPI: SSDT 0xFFFF880462178000 000119 (v02 PmRef  ApCst    00003000 INTL 20120913)
[    0.858413] ACPI: Interpreter enabled
[    0.858438] ACPI: (supports S0 S3 S4 S5)
[    0.858439] ACPI: Using IOAPIC for interrupt routing
[    0.858482] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.861749] ACPI: Power Resource [PG00] (on)
[    0.862513] ACPI: Power Resource [PG01] (on)
[    0.863229] ACPI: Power Resource [PG02] (on)
[    0.867462] ACPI: Power Resource [WRST] (off)
[    0.868183] ACPI: Power Resource [WRST] (off)
[    0.868898] ACPI: Power Resource [WRST] (off)
[    0.869617] ACPI: Power Resource [WRST] (off)
[    0.870337] ACPI: Power Resource [WRST] (off)
[    0.871065] ACPI: Power Resource [WRST] (off)
[    0.871782] ACPI: Power Resource [WRST] (off)
[    0.872516] ACPI: Power Resource [WRST] (off)
[    0.873231] ACPI: Power Resource [WRST] (off)
[    0.873948] ACPI: Power Resource [WRST] (off)
[    0.874665] ACPI: Power Resource [WRST] (off)
[    0.875386] ACPI: Power Resource [WRST] (off)
[    0.876103] ACPI: Power Resource [WRST] (off)
[    0.876826] ACPI: Power Resource [WRST] (off)
[    0.877549] ACPI: Power Resource [WRST] (off)
[    0.878268] ACPI: Power Resource [WRST] (off)
[    0.878985] ACPI: Power Resource [WRST] (off)
[    0.879700] ACPI: Power Resource [WRST] (off)
[    0.880414] ACPI: Power Resource [WRST] (off)
[    0.881126] ACPI: Power Resource [WRST] (off)
[    0.895671] ACPI: Power Resource [FN00] (off)
[    0.895832] ACPI: Power Resource [FN01] (off)
[    0.895993] ACPI: Power Resource [FN02] (off)
[    0.896151] ACPI: Power Resource [FN03] (off)
[    0.896308] ACPI: Power Resource [FN04] (off)
[    0.898327] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe])
[    0.898332] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    0.898411] acpi PNP0A08:00: _OSC failed (AE_ERROR); disabling ASPM
[    0.899829] PCI host bridge to bus 0000:00
[    0.899831] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.899832] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.899833] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.899834] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
[    0.899835] pci_bus 0000:00: root bus resource [mem 0x800000000-0xfbfffffff window]
[    0.899836] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
[    0.899837] pci_bus 0000:00: root bus resource [bus 00-fe]
[    0.899843] pci 0000:00:00.0: [8086:191f] type 00 class 0x060000
[    0.900389] pci 0000:00:14.0: [8086:a12f] type 00 class 0x0c0330
[    0.900404] pci 0000:00:14.0: reg 0x10: [mem 0xfbff10000-0xfbff1ffff 64bit]
[    0.900460] pci 0000:00:14.0: PME# supported from D3hot D3cold
[    0.900679] pci 0000:00:14.0: System wakeup disabled by ACPI
[    0.900764] pci 0000:00:16.0: [8086:a13a] type 00 class 0x078000
[    0.900776] pci 0000:00:16.0: reg 0x10: [mem 0xfbff25000-0xfbff25fff 64bit]
[    0.900823] pci 0000:00:16.0: PME# supported from D3hot
[    0.901084] pci 0000:00:17.0: [8086:a102] type 00 class 0x010601
[    0.901095] pci 0000:00:17.0: reg 0x10: [mem 0xdff24000-0xdff25fff]
[    0.901100] pci 0000:00:17.0: reg 0x14: [mem 0xdff27000-0xdff270ff]
[    0.901106] pci 0000:00:17.0: reg 0x18: [io  0xf050-0xf057]
[    0.901111] pci 0000:00:17.0: reg 0x1c: [io  0xf040-0xf043]
[    0.901116] pci 0000:00:17.0: reg 0x20: [io  0xf020-0xf03f]
[    0.901122] pci 0000:00:17.0: reg 0x24: [mem 0xdff26000-0xdff267ff]
[    0.901155] pci 0000:00:17.0: PME# supported from D3hot
[    0.901412] pci 0000:00:1b.0: [8086:a167] type 01 class 0x060400
[    0.901461] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
[    0.901730] pci 0000:00:1b.0: System wakeup disabled by ACPI
[    0.901812] pci 0000:00:1c.0: [8086:a110] type 01 class 0x060400
[    0.901861] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    0.902127] pci 0000:00:1c.0: System wakeup disabled by ACPI
[    0.902189] pci 0000:00:1c.2: [8086:a112] type 01 class 0x060400
[    0.902238] pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold
[    0.902503] pci 0000:00:1c.2: System wakeup disabled by ACPI
[    0.902579] pci 0000:00:1d.0: [8086:a118] type 01 class 0x060400
[    0.902630] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
[    0.902891] pci 0000:00:1d.0: System wakeup disabled by ACPI
[    0.902979] pci 0000:00:1f.0: [8086:a145] type 00 class 0x060100
[    0.903307] pci 0000:00:1f.2: [8086:a121] type 00 class 0x058000
[    0.903317] pci 0000:00:1f.2: reg 0x10: [mem 0xdff20000-0xdff23fff]
[    0.903599] pci 0000:00:1f.3: [8086:a170] type 00 class 0x040300
[    0.903618] pci 0000:00:1f.3: reg 0x10: [mem 0xfbff20000-0xfbff23fff 64bit]
[    0.903646] pci 0000:00:1f.3: reg 0x20: [mem 0xfbff00000-0xfbff0ffff 64bit]
[    0.903690] pci 0000:00:1f.3: PME# supported from D3hot D3cold
[    0.903961] pci 0000:00:1f.3: System wakeup disabled by ACPI
[    0.904027] pci 0000:00:1f.4: [8086:a123] type 00 class 0x0c0500
[    0.904073] pci 0000:00:1f.4: reg 0x10: [mem 0xfbff24000-0xfbff240ff 64bit]
[    0.904142] pci 0000:00:1f.4: reg 0x20: [io  0xf000-0xf01f]
[    0.904456] pci 0000:00:1f.6: [8086:15b8] type 00 class 0x020000
[    0.904475] pci 0000:00:1f.6: reg 0x10: [mem 0xdff00000-0xdff1ffff]
[    0.904576] pci 0000:00:1f.6: PME# supported from D0 D3hot D3cold
[    0.904781] pci 0000:00:1f.6: System wakeup disabled by ACPI
[    0.904949] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.904964] pci 0000:01:00.0: reg 0x10: [mem 0xfa0000000-0xfafffffff 64bit pref]
[    0.904974] pci 0000:01:00.0: reg 0x18: [mem 0xfb0000000-0xfb01fffff 64bit pref]
[    0.904981] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.904987] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.904994] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.905057] pci 0000:01:00.0: supports D1 D2
[    0.905058] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.905092] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
[    0.905093] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.905101] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
[    0.905102] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
[    0.905114] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.905114] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
[    0.905166] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.915006] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.915011] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.915017] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.915024] pci 0000:00:1b.0:   bridge window [mem 0xfa0000000-0xfb01fffff 64bit pref]
[    0.915042] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
[    0.915164] pci 0000:02:00.0: [1b21:1242] type 00 class 0x0c0330
[    0.915183] pci 0000:02:00.0: reg 0x10: [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.915289] pci 0000:02:00.0: PME# supported from D3hot D3cold
[    0.915342] pci 0000:02:00.0: System wakeup disabled by ACPI
[    0.927027] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.927035] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.927180] pci 0000:03:00.0: [1b21:1080] type 01 class 0x060400
[    0.927296] pci 0000:03:00.0: supports D1 D2
[    0.927296] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.927341] pci 0000:03:00.0: System wakeup disabled by ACPI
[    0.939025] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.939172] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.939320] acpiphp: Slot [1] registered
[    0.939323] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.942135] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
[    0.942249] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.942361] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 10 11 12 14 *15)
[    0.942472] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.942584] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.942695] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.942806] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 11 12 14 15) *0
[    0.942917] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15)
[    0.943744] ACPI: Enabled 6 GPEs in block 00 to 7F
[    0.944022] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.944023] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.944023] vgaarb: loaded
[    0.944024] vgaarb: bridge control possible 0000:01:00.0
[    0.944291] SCSI subsystem initialized
[    0.944343] libata version 3.00 loaded.
[    0.944374] ACPI: bus type USB registered
[    0.944403] usbcore: registered new interface driver usbfs
[    0.944418] usbcore: registered new interface driver hub
[    0.944439] usbcore: registered new device driver usb
[    0.944570] PCI: Using ACPI for IRQ routing
[    0.972385] PCI: pci_cache_line_size set to 64 bytes
[    0.972422] e820: reserve RAM buffer [mem 0x0009c800-0x0009ffff]
[    0.972424] e820: reserve RAM buffer [mem 0x7d439000-0x7fffffff]
[    0.972425] e820: reserve RAM buffer [mem 0x7d4c5000-0x7fffffff]
[    0.972426] e820: reserve RAM buffer [mem 0x858a8000-0x87ffffff]
[    0.972427] e820: reserve RAM buffer [mem 0x476800000-0x477ffffff]
[    0.972586] NetLabel: Initializing
[    0.972587] NetLabel:  domain hash size = 128
[    0.972587] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.972605] NetLabel:  unlabeled traffic allowed by default
[    0.972732] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    0.972735] hpet0: 8 comparators, 64-bit 24.000000 MHz counter
[    0.974777] clocksource: Switched to clocksource hpet
[    0.984798] VFS: Disk quotas dquot_6.6.0
[    0.984834] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.984964] AppArmor: AppArmor Filesystem Enabled
[    0.985017] pnp: PnP ACPI init
[    0.985303] system 00:00: [io  0x0290-0x029f] has been reserved
[    0.985305] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.985776] pnp 00:01: [dma 0 disabled]
[    0.985840] pnp 00:01: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.986081] system 00:02: [io  0x0680-0x069f] has been reserved
[    0.986083] system 00:02: [io  0xffff] has been reserved
[    0.986084] system 00:02: [io  0xffff] has been reserved
[    0.986086] system 00:02: [io  0xffff] has been reserved
[    0.986087] system 00:02: [io  0x1800-0x18fe] has been reserved
[    0.986089] system 00:02: [io  0x164e-0x164f] has been reserved
[    0.986090] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.986198] system 00:03: [io  0x0800-0x087f] has been reserved
[    0.986199] system 00:03: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.986238] pnp 00:04: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.986308] system 00:05: [io  0x1854-0x1857] has been reserved
[    0.986309] system 00:05: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
[    0.986698] system 00:06: [mem 0xfed10000-0xfed17fff] has been reserved
[    0.986699] system 00:06: [mem 0xfed18000-0xfed18fff] has been reserved
[    0.986701] system 00:06: [mem 0xfed19000-0xfed19fff] has been reserved
[    0.986703] system 00:06: [mem 0xe0000000-0xefffffff] has been reserved
[    0.986704] system 00:06: [mem 0xfed20000-0xfed3ffff] has been reserved
[    0.986705] system 00:06: [mem 0xfed90000-0xfed93fff] has been reserved
[    0.986707] system 00:06: [mem 0xfed45000-0xfed8ffff] has been reserved
[    0.986708] system 00:06: [mem 0xff000000-0xffffffff] has been reserved
[    0.986710] system 00:06: [mem 0xfee00000-0xfeefffff] could not be reserved
[    0.986712] system 00:06: [mem 0xdffc0000-0xdffdffff] has been reserved
[    0.986713] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.986814] system 00:07: [mem 0xfd000000-0xfdabffff] has been reserved
[    0.986816] system 00:07: [mem 0xfdad0000-0xfdadffff] has been reserved
[    0.986817] system 00:07: [mem 0xfdb00000-0xfdffffff] has been reserved
[    0.986820] system 00:07: [mem 0xfe000000-0xfe01ffff] could not be reserved
[    0.986821] system 00:07: [mem 0xfe036000-0xfe03bfff] has been reserved
[    0.986824] system 00:07: [mem 0xfe03d000-0xfe3fffff] has been reserved
[    0.986825] system 00:07: [mem 0xfe410000-0xfe7fffff] has been reserved
[    0.986827] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.988557] system 00:08: [mem 0xfdaf0000-0xfdafffff] has been reserved
[    0.988559] system 00:08: [mem 0xfdae0000-0xfdaeffff] has been reserved
[    0.988560] system 00:08: [mem 0xfdac0000-0xfdacffff] has been reserved
[    0.988562] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.989928] pnp: PnP ACPI: found 9 devices
[    0.997868] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.997871] pci_bus 0000:00: max bus depth: 2 pci_try_num: 3
[    0.997900] pci 0000:00:1d.0: BAR 14: assigned [mem 0x88800000-0x889fffff]
[    0.997902] pci 0000:00:1d.0: BAR 15: assigned [mem 0x800000000-0x8001fffff 64bit pref]
[    0.997903] pci 0000:00:1d.0: BAR 13: assigned [io  0x2000-0x2fff]
[    0.997907] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997908] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997910] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
[    0.997910] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
[    0.997911] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997912] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997914] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997915] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997917] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.997919] pci 0000:00:1b.0:   bridge window [mem 0xfa0000000-0xfb01fffff 64bit pref]
[    0.997922] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997924] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.997928] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.997942] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.997947] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997948] pci 0000:00:1d.0:   bridge window [io  0x2000-0x2fff]
[    0.997951] pci 0000:00:1d.0:   bridge window [mem 0x88800000-0x889fffff]
[    0.997953] pci 0000:00:1d.0:   bridge window [mem 0x800000000-0x8001fffff 64bit pref]
[    0.997956] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.997957] release child resource [mem 0xdfe00000-0xdfe3ffff]
[    0.997957] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
[    0.997958] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] released
[    0.997958] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997960] release child resource [mem 0xfa0000000-0xfafffffff 64bit pref]
[    0.997960] release child resource [mem 0xfb0000000-0xfb01fffff 64bit pref]
[    0.997961] pci 0000:00:1b.0: resource 15 [mem 0xfa0000000-0xfb01fffff 64bit pref] released
[    0.997961] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997992] pci 0000:00:1b.0: BAR 15: assigned [mem 0x808000000-0x85fffffff 64bit pref]
[    0.997993] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abfffff]
[    0.997997] pci 0000:01:00.0: BAR 0: assigned [mem 0x810000000-0x81fffffff 64bit pref]
[    0.998004] pci 0000:01:00.0: BAR 7: assigned [mem 0x820000000-0x85fffffff 64bit pref]
[    0.998011] pci 0000:01:00.0: BAR 9: assigned [mem 0x808000000-0x809ffffff 64bit pref]
[    0.998017] pci 0000:01:00.0: BAR 12: assigned [mem 0x88c00000-0x8abfffff]
[    0.998019] pci 0000:01:00.0: BAR 2: assigned [mem 0x80a000000-0x80a1fffff 64bit pref]
[    0.998026] pci 0000:01:00.0: BAR 5: assigned [mem 0x88a00000-0x88a3ffff]
[    0.998028] pci 0000:01:00.0: BAR 6: assigned [mem 0x88a40000-0x88a5ffff pref]
[    0.998030] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.998031] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.998033] pci 0000:00:1b.0:   bridge window [mem 0x88a00000-0x8abfffff]
[    0.998035] pci 0000:00:1b.0:   bridge window [mem 0x808000000-0x85fffffff 64bit pref]
[    0.998038] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.998040] pci 0000:00:1c.0:   bridge window [mem 0xdfd00000-0xdfdfffff]
[    0.998044] pci 0000:03:00.0: PCI bridge to [bus 04]
[    0.998058] pci 0000:00:1c.2: PCI bridge to [bus 03-04]
[    0.998063] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.998064] pci 0000:00:1d.0:   bridge window [io  0x2000-0x2fff]
[    0.998067] pci 0000:00:1d.0:   bridge window [mem 0x88800000-0x889fffff]
[    0.998069] pci 0000:00:1d.0:   bridge window [mem 0x800000000-0x8001fffff 64bit pref]
[    0.998072] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.998073] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.998074] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.998074] pci_bus 0000:00: resource 7 [mem 0x88800000-0xdfffffff window]
[    0.998075] pci_bus 0000:00: resource 8 [mem 0x800000000-0xfbfffffff window]
[    0.998075] pci_bus 0000:00: resource 9 [mem 0xfd000000-0xfe7fffff window]
[    0.998076] pci_bus 0000:01: resource 0 [io  0xe000-0xefff]
[    0.998077] pci_bus 0000:01: resource 1 [mem 0x88a00000-0x8abfffff]
[    0.998077] pci_bus 0000:01: resource 2 [mem 0x808000000-0x85fffffff 64bit pref]
[    0.998078] pci_bus 0000:02: resource 1 [mem 0xdfd00000-0xdfdfffff]
[    0.998079] pci_bus 0000:05: resource 0 [io  0x2000-0x2fff]
[    0.998079] pci_bus 0000:05: resource 1 [mem 0x88800000-0x889fffff]
[    0.998080] pci_bus 0000:05: resource 2 [mem 0x800000000-0x8001fffff 64bit pref]
[    0.998267] NET: Registered protocol family 2
[    0.998466] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.998584] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.998662] TCP: Hash tables configured (established 131072 bind 65536)
[    0.998683] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    0.998712] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    0.998818] NET: Registered protocol family 1
[    0.999492] pci 0000:01:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    1.000166] PCI: CLS 0 bytes, default 64
[    1.000212] Trying to unpack rootfs image as initramfs...
[    1.727856] Freeing initrd memory: 70180K (ffff88002f6de000 - ffff880033b67000)
[    1.727858] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.727859] software IO TLB [mem 0x818a8000-0x858a8000] (64MB) mapped at [ffff8800818a8000-ffff8800858a7fff]
[    1.728068] RAPL PMU: API unit is 2^-32 Joules, 5 fixed counters, 655360 ms ovfl timer
[    1.728069] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
[    1.728069] RAPL PMU: hw unit of domain package 2^-14 Joules
[    1.728069] RAPL PMU: hw unit of domain dram 2^-14 Joules
[    1.728069] RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules
[    1.728070] RAPL PMU: hw unit of domain psys 2^-14 Joules
[    1.728647] Scanning for low memory corruption every 60 seconds
[    1.729225] futex hash table entries: 2048 (order: 5, 131072 bytes)
[    1.729251] audit: initializing netlink subsys (disabled)
[    1.729267] audit: type=2000 audit(1495613326.692:1): initialized
[    1.730018] Initialise system trusted keyrings
[    1.730150] workingset: timestamp_bits=40 max_order=22 bucket_order=0
[    1.734481] fuse init (API version 7.26)
[    1.735712] Key type asymmetric registered
[    1.735713] Asymmetric key parser 'x509' registered
[    1.735773] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    1.735831] io scheduler noop registered
[    1.735832] io scheduler deadline registered (default)
[    1.735844] io scheduler cfq registered
[    1.736331] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    1.736338] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    1.736377] efifb: probing for efifb
[    1.736380] efifb: framebuffer at 0xa0000, using 64k, total 64k
[    1.736381] efifb: mode is 640x480x1, linelength=80, pages=1
[    1.736381] efifb: scrolling: redraw
[    1.736382] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.737123] Console: switching to colour frame buffer device 80x30
[    1.737737] fb0: EFI VGA frame buffer device
[    1.737744] intel_idle: MWAIT substates: 0x142120
[    1.737745] intel_idle: v0.4.1 model 0x5E
[    1.738473] intel_idle: lapic_timer_reliable_states 0xffffffff
[    1.738485] ipmi message handler version 39.2
[    1.738588] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input0
[    1.738590] ACPI: Sleep Button [SLPB]
[    1.738650] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[    1.738651] ACPI: Power Button [PWRB]
[    1.738710] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[    1.738712] ACPI: Power Button [PWRF]
[    1.740515] thermal LNXTHERM:00: registered as thermal_zone0
[    1.740516] ACPI: Thermal Zone [TZ00] (28 C)
[    1.740858] thermal LNXTHERM:01: registered as thermal_zone1
[    1.740859] ACPI: Thermal Zone [TZ01] (30 C)
[    1.740889] GHES: HEST is not enabled!
[    1.741021] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    1.761567] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.764348] Linux agpgart interface v0.103
[    1.770590] brd: module loaded
[    1.773931] loop: module loaded
[    1.774360] libphy: Fixed MDIO Bus: probed
[    1.774751] tun: Universal TUN/TAP device driver, 1.6
[    1.774751] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    1.774837] PPP generic driver version 2.4.2
[    1.774916] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.774919] ehci-pci: EHCI PCI platform driver
[    1.774932] ehci-platform: EHCI generic platform driver
[    1.774942] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.774944] ohci-pci: OHCI PCI platform driver
[    1.774955] ohci-platform: OHCI generic platform driver
[    1.774963] uhci_hcd: USB Universal Host Controller Interface driver
[    1.775149] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.775155] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[    1.776238] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x00109810
[    1.776241] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
[    1.776351] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    1.776352] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.776352] usb usb1: Product: xHCI Host Controller
[    1.776353] usb usb1: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.776354] usb usb1: SerialNumber: 0000:00:14.0
[    1.776543] hub 1-0:1.0: USB hub found
[    1.776560] hub 1-0:1.0: 16 ports detected
[    1.791307] xhci_hcd 0000:00:14.0: xHCI Host Controller
[    1.791310] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[    1.791353] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[    1.791353] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.791354] usb usb2: Product: xHCI Host Controller
[    1.791354] usb usb2: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.791355] usb usb2: SerialNumber: 0000:00:14.0
[    1.791548] hub 2-0:1.0: USB hub found
[    1.791561] hub 2-0:1.0: 10 ports detected
[    1.800922] xhci_hcd 0000:02:00.0: xHCI Host Controller
[    1.800927] xhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 3
[    1.859691] xhci_hcd 0000:02:00.0: hcc params 0x0200eec1 hci version 0x110 quirks 0x00000010
[    1.860008] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
[    1.860009] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.860010] usb usb3: Product: xHCI Host Controller
[    1.860010] usb usb3: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.860011] usb usb3: SerialNumber: 0000:02:00.0
[    1.860195] hub 3-0:1.0: USB hub found
[    1.860203] hub 3-0:1.0: 2 ports detected
[    1.860324] xhci_hcd 0000:02:00.0: xHCI Host Controller
[    1.860327] xhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 4
[    1.860355] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    1.860379] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003
[    1.860380] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.860380] usb usb4: Product: xHCI Host Controller
[    1.860381] usb usb4: Manufacturer: Linux 4.9.0-custom xhci-hcd
[    1.860381] usb usb4: SerialNumber: 0000:02:00.0
[    1.860561] hub 4-0:1.0: USB hub found
[    1.860568] hub 4-0:1.0: 2 ports detected
[    1.860726] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.864072] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.864076] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.864333] mousedev: PS/2 mouse device common for all mice
[    1.864684] rtc_cmos 00:04: RTC can wake from S4
[    1.865126] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
[    1.865206] rtc_cmos 00:04: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
[    1.865278] device-mapper: uevent: version 1.0.3
[    1.865415] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
[    1.865417] intel_pstate: Intel P-state driver initializing
[    1.866281] intel_pstate: HWP enabled
[    1.866704] ledtrig-cpu: registered to indicate activity on CPUs
[    1.867033] NET: Registered protocol family 10
[    1.867547] NET: Registered protocol family 17
[    1.867554] Key type dns_resolver registered
[    1.868060] microcode: sig=0x506e3, pf=0x2, revision=0x33
[    1.868463] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    1.868853] registered taskstats version 1
[    1.868854] Loading compiled-in X.509 certificates
[    1.871212] Loaded X.509 cert 'Build time autogenerated kernel key: 425e568c95731e69b073bb62214af700479a67a7'
[    1.871304] kmemleak: Kernel memory leak detector initialized
[    1.871372] kmemleak: Automatic memory scanning thread started
[    1.873205] Key type trusted registered
[    1.877582] Key type encrypted registered
[    1.877584] AppArmor: AppArmor sha1 policy hashing enabled
[    1.877585] ima: No TPM chip found, activating TPM-bypass!
[    1.877601] evm: HMAC attrs: 0x1
[    1.878948]   Magic number: 1:708:119
[    1.879197] rtc_cmos 00:04: setting system clock to 2017-05-24 08:08:47 UTC (1495613327)
[    1.879439] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    1.879439] EDD information not available.
[    1.879588] PM: Hibernation image not present or could not be loaded.
[    1.880425] Freeing unused kernel memory: 2220K (ffffffff81d6b000 - ffffffff81f96000)
[    1.880425] Write protecting the kernel read-only data: 12288k
[    1.880640] Freeing unused kernel memory: 84K (ffff8800017eb000 - ffff880001800000)
[    1.881267] Freeing unused kernel memory: 428K (ffff880001b95000 - ffff880001c00000)
[    1.890528] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.890590] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.890597] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.890928] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.890952] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.892018] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.892059] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.892575] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.892612] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.892648] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.921193] hid: module verification failed: signature and/or required key missing - tainting kernel
[    1.921599] hidraw: raw HID events driver (C) Jiri Kosina
[    1.926810] wmi: Mapper loaded
[    1.941711] ahci 0000:00:17.0: version 3.0
[    1.942212] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[    1.942214] ahci 0000:00:17.0: flags: 64bit ncq sntf led clo only pio slum part ems deso sadm sds apst 
[    1.943366] pps_core: LinuxPPS API ver. 1 registered
[    1.943366] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.943776] PTP clock support registered
[    1.944426] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.944426] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.000668] scsi host0: ahci
[    2.001171] scsi host1: ahci
[    2.001663] scsi host2: ahci
[    2.002052] scsi host3: ahci
[    2.002447] scsi host4: ahci
[    2.002844] scsi host5: ahci
[    2.002971] ata1: SATA max UDMA/133 abar m2048@0xdff26000 port 0xdff26100 irq 129
[    2.002972] ata2: SATA max UDMA/133 abar m2048@0xdff26000 port 0xdff26180 irq 129
[    2.002973] ata3: SATA max UDMA/133 abar m2048@0xdff26000 port 0xdff26200 irq 129
[    2.002974] ata4: SATA max UDMA/133 abar m2048@0xdff26000 port 0xdff26280 irq 129
[    2.002975] ata5: SATA max UDMA/133 abar m2048@0xdff26000 port 0xdff26300 irq 129
[    2.002976] ata6: SATA max UDMA/133 abar m2048@0xdff26000 port 0xdff26380 irq 129
[    2.003589] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    2.118908] usb 1-10: new full-speed USB device number 2 using xhci_hcd
[    2.262007] usb 1-10: New USB device found, idVendor=046d, idProduct=c534
[    2.262008] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    2.262009] usb 1-10: Product: USB Receiver
[    2.262009] usb 1-10: Manufacturer: Logitech
[    2.267030] usbcore: registered new interface driver usbhid
[    2.267031] usbhid: USB HID core driver
[    2.267845] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/0003:046D:C534.0001/input/input6
[    2.278741] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    2.319778] CPU: 4 PID: 207 Comm: scsi_eh_2 Tainted: G            E   4.9.0-custom #2
[    2.319779] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.319779]  ffffc90002033ae8 ffffffff813c2d3f ffff880452935e80 ffff8804529362b0
[    2.319781]  ffffc90002033b10 ffffffff815a7226 ffffc90002033b10 0000000000000286
[    2.319782]  ffffc90000029200 ffffc90002033b30 ffffffffa003f634 ffff880452935e80
[    2.319783] Call Trace:
[    2.319786]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.319788]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.319790]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.319791]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.319793]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.319794]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.319795]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319796]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319797]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319798]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.319799]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.319800]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.319801]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319802]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.319803]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.319804]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.319806]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.319808]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.319809]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.319810]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.319811]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.319813]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.319814]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.319815]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.319816]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.319817]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.319818]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.319820]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.319823] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.319839] CPU: 5 PID: 213 Comm: scsi_eh_5 Tainted: G            E   4.9.0-custom #2
[    2.319840] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.319841]  ffffc900020f7ae8 ffffffff813c2d3f ffff880452941e80 ffff8804529422b0
[    2.319842]  ffffc900020f7b10 ffffffff815a7226 ffffc900020f7b10 0000000000000286
[    2.319843]  ffffc90000029380 ffffc900020f7b30 ffffffffa003f634 ffff880452941e80
[    2.319844] Call Trace:
[    2.319846]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.319847]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.319849]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.319850]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.319851]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.319852]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.319853]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319854]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319855]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319856]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.319857]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.319858]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.319859]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319860]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.319861]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.319862]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.319864]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.319865]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.319866]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.319867]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.319868]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.319870]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.319871]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.319872]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.319873]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.319873]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.319874]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.319875]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.319879] ata6: SATA link down (SStatus 4 SControl 300)
[    2.319911] CPU: 6 PID: 211 Comm: scsi_eh_4 Tainted: G            E   4.9.0-custom #2
[    2.319912] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.319913]  ffffc9000205fae8 ffffffff813c2d3f ffff88045293de80 ffff88045293e2b0
[    2.319914]  ffffc9000205fb10 ffffffff815a7226 ffffc9000205fb10 0000000000000286
[    2.319915]  ffffc90000029300 ffffc9000205fb30 ffffffffa003f634 ffff88045293de80
[    2.319917] Call Trace:
[    2.319919]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.319920]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.319921]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.319922]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.319924]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.319926]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.319927]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319928]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319930]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319931]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.319932]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.319934]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.319935]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.319936]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.319937]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.319938]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.319939]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.319941]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.319942]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.319943]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.319944]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.319957]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.319958]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.319959]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.319960]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.319961]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.319962]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.319963]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.319966] ata5: SATA link down (SStatus 4 SControl 300)
[    2.319984] CPU: 3 PID: 205 Comm: scsi_eh_1 Tainted: G            E   4.9.0-custom #2
[    2.319985] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.319986]  ffffc900020efae8 ffffffff813c2d3f ffff880452931e80 ffff8804529322b0
[    2.320001]  ffffc900020efb10 ffffffff815a7226 ffffc900020efb10 0000000000000286
[    2.320002]  ffffc90000029180 ffffc900020efb30 ffffffffa003f634 ffff880452931e80
[    2.320004] Call Trace:
[    2.320006]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.320007]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.320008]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.320010]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.320012]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.320013]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.320015]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320016]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320017]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320018]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.320019]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.320021]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.320022]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320024]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.320024]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.320026]  [<ffffffff813cc65e>] ? number+0x2ce/0x300
[    2.320027]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.320028]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.320030]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.320031]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.320032]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.320033]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.320034]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.320035]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.320036]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.320037]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.320038]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.320039]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.320040]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.320054]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.320057] ata2: SATA link down (SStatus 4 SControl 300)
[    2.320082] CPU: 7 PID: 203 Comm: scsi_eh_0 Tainted: G            E   4.9.0-custom #2
[    2.320082] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.320082]  ffffc90002023ae8 ffffffff813c2d3f ffff88045292de80 ffff88045292e2b0
[    2.320083]  ffffc90002023b10 ffffffff815a7226 ffffc90002023b10 0000000000000286
[    2.320085]  ffffc90000029100 ffffc90002023b30 ffffffffa003f634 ffff88045292de80
[    2.320086] Call Trace:
[    2.320087]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.320088]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.320089]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.320090]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.320091]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.320092]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.320093]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320094]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320095]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320096]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.320097]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.320098]  [<ffffffff8152c910>] ? device_create_release+0x60/0x60
[    2.320099]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.320100]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320101]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.320102]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.320103]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.320104]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.320105]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.320106]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.320107]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.320108]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.320109]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.320110]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.320111]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.320112]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.320113]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.320114]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.320115]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.320118] ata1: SATA link down (SStatus 4 SControl 300)
[    2.320134] CPU: 2 PID: 209 Comm: scsi_eh_3 Tainted: G            E   4.9.0-custom #2
[    2.320135] Hardware name: System manufacturer System Product Name/Z170-A, BIOS 0901 08/31/2015
[    2.320135]  ffffc90002043ae8 ffffffff813c2d3f ffff880452939e80 ffff88045293a2b0
[    2.320150]  ffffc90002043b10 ffffffff815a7226 ffffc90002043b10 0000000000000286
[    2.320151]  ffffc90000029280 ffffc90002043b30 ffffffffa003f634 ffff880452939e80
[    2.320152] Call Trace:
[    2.320154]  [<ffffffff813c2d3f>] dump_stack+0x63/0x84
[    2.320154]  [<ffffffff815a7226>] ata_std_postreset+0x16/0xe0
[    2.320156]  [<ffffffffa003f634>] ahci_postreset+0x34/0x60 [libahci]
[    2.320157]  [<ffffffff815b1964>] ata_eh_reset+0x484/0xda0
[    2.320158]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.320171]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.320172]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320173]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320174]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320175]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.320176]  [<ffffffff815b2652>] ata_eh_recover+0x282/0x12d0
[    2.320177]  [<ffffffffa003f600>] ? ahci_pmp_attach+0x70/0x70 [libahci]
[    2.320178]  [<ffffffffa0041350>] ? ahci_sw_activity_blink+0xd0/0xd0 [libahci]
[    2.320179]  [<ffffffffa0040e90>] ? ahci_do_softreset+0x220/0x220 [libahci]
[    2.320180]  [<ffffffff815a7380>] ? ata_phys_link_offline+0x30/0x30
[    2.320181]  [<ffffffff815b9921>] sata_pmp_error_handler+0xe1/0xaa0
[    2.320183]  [<ffffffff81096090>] ? try_to_grab_pending+0xb0/0x150
[    2.320184]  [<ffffffff810ea2ed>] ? lock_timer_base+0x7d/0xa0
[    2.320185]  [<ffffffffa003fde8>] ahci_error_handler+0x28/0x70 [libahci]
[    2.320186]  [<ffffffff815b3c7c>] ata_scsi_port_error_handler+0x50c/0x8d0
[    2.320188]  [<ffffffff815b40d8>] ata_scsi_error+0x98/0xc0
[    2.320189]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.320190]  [<ffffffff81582a07>] scsi_error_handler+0xd7/0x8b0
[    2.320191]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.320207]  [<ffffffff81582930>] ? scsi_eh_get_sense+0x240/0x240
[    2.320207]  [<ffffffff8109b423>] kthread+0xd3/0xf0
[    2.320208]  [<ffffffff8109b350>] ? kthread_park+0x60/0x60
[    2.320209]  [<ffffffff817e1ee5>] ret_from_fork+0x25/0x30
[    2.320232] ata4: SATA link down (SStatus 4 SControl 300)
[    2.321116] ata3.00: ATA-8: ST3500413AS, JC45, max UDMA/133
[    2.321117] ata3.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32)
[    2.322647] ata3.00: configured for UDMA/133
[    2.323221] scsi 2:0:0:0: Direct-Access     ATA      ST3500413AS      JC45 PQ: 0 ANSI: 5
[    2.327221] hid-generic 0003:046D:C534.0001: input,hidraw0: USB HID v1.11 Keyboard [Logitech USB Receiver] on usb-0000:00:14.0-10/input0
[    2.327399] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.1/0003:046D:C534.0002/input/input7
[    2.367060] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 30:5a:3a:e0:7f:4d
[    2.367061] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    2.367144] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: FFFFFF-0FF
[    2.367819] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
[    2.375374] sd 2:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/466 GiB)
[    2.375458] sd 2:0:0:0: Attached scsi generic sg0 type 0
[    2.375714] sd 2:0:0:0: [sda] Write Protect is off
[    2.375715] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.375889] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.387540] hid-generic 0003:046D:C534.0002: input,hiddev0,hidraw1: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-0000:00:14.0-10/input1
[    2.401973]  sda: sda1 sda2 < sda5 >
[    2.403492] sd 2:0:0:0: [sda] Attached SCSI disk
[    2.731874] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[    2.750887] tsc: Refined TSC clocksource calibration: 3407.999 MHz
[    2.750893] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x311fd336761, max_idle_ns: 440795243819 ns
[    2.802667] random: fast init done
[    3.625199] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[    3.625331] systemd[1]: Detected architecture x86-64.
[    3.632623] systemd[1]: Set hostname to <jenkins-MS-7984>.
[    3.775192] clocksource: Switched to clocksource tsc
[    4.272154] systemd[1]: Listening on udev Kernel Socket.
[    4.272233] systemd[1]: Listening on Journal Audit Socket.
[    4.272259] systemd[1]: Listening on Journal Socket (/dev/log).
[    4.272288] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    4.272303] systemd[1]: Listening on fsck to fsckd communication Socket.
[    4.272453] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    4.272477] systemd[1]: Listening on udev Control Socket.
[    4.842891] random: crng init done
[    4.873008] lp: driver loaded but no devices found
[    4.874333] ppdev: user-space parallel port driver
[    5.086789] RPC: Registered named UNIX socket transport module.
[    5.086789] RPC: Registered udp transport module.
[    5.086789] RPC: Registered tcp transport module.
[    5.086790] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    5.420022] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    5.764199] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro
[    6.016854] systemd-journald[298]: Received request to flush runtime journal from PID 1
[    6.498910] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    6.673532] AVX2 version of gcm_enc/dec engaged.
[    6.673533] AES CTR mode by8 optimization enabled
[    6.755847] mei_me 0000:00:16.0: enabling device (0000 -> 0002)
[    7.615057] intel_rapl: Found RAPL domain package
[    7.615059] intel_rapl: Found RAPL domain core
[    7.615061] intel_rapl: Found RAPL domain dram
[    7.684934] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC892: line_outs=3 (0x14/0x15/0x16/0x0/0x0) type:line
[    7.684935] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    7.684936] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    7.684937] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    7.684937] snd_hda_codec_realtek hdaudioC0D0:    dig-out=0x11/0x1e
[    7.684938] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    7.684938] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[    7.684939] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[    7.684940] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a
[    7.697072] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input8
[    7.697178] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9
[    7.697284] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
[    7.697391] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
[    7.697499] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1f.3/sound/card0/input12
[    7.697604] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13
[    7.697711] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input14
[    8.643643] audit: type=1400 audit(1495613334.260:2): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine" pid=701 comm="apparmor_parser"
[    8.643649] audit: type=1400 audit(1495613334.260:3): apparmor="STATUS" operation="profile_load" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=701 comm="apparmor_parser"
[    8.664449] audit: type=1400 audit(1495613334.280:4): apparmor="STATUS" operation="profile_load" name="/usr/sbin/ippusbxd" pid=704 comm="apparmor_parser"
[    8.767842] audit: type=1400 audit(1495613334.384:5): apparmor="STATUS" operation="profile_load" name="/usr/sbin/cups-browsed" pid=702 comm="apparmor_parser"
[    8.792412] audit: type=1400 audit(1495613334.412:6): apparmor="STATUS" operation="profile_load" name="/sbin/dhclient" pid=697 comm="apparmor_parser"
[    8.792419] audit: type=1400 audit(1495613334.412:7): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=697 comm="apparmor_parser"
[    8.792426] audit: type=1400 audit(1495613334.412:8): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=697 comm="apparmor_parser"
[    8.792432] audit: type=1400 audit(1495613334.412:9): apparmor="STATUS" operation="profile_load" name="/usr/lib/connman/scripts/dhclient-script" pid=697 comm="apparmor_parser"
[    8.793197] audit: type=1400 audit(1495613334.412:10): apparmor="STATUS" operation="profile_load" name="/usr/lib/cups/backend/cups-pdf" pid=703 comm="apparmor_parser"
[    8.793203] audit: type=1400 audit(1495613334.412:11): apparmor="STATUS" operation="profile_load" name="/usr/sbin/cupsd" pid=703 comm="apparmor_parser"
[    8.920486] asus_wmi: ASUS WMI generic driver loaded
[    8.933316] asus_wmi: Initialization: 0x0
[    8.933378] asus_wmi: BIOS WMI version: 0.9
[    8.933506] asus_wmi: SFUN value: 0x0
[    8.934587] input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input15
[    8.934890] asus_wmi: Number of fans: 1
[    9.519567] Adding 8348668k swap on /dev/sda5.  Priority:-1 extents:1 across:8348668k FS
[   14.493113] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[   14.712021] NFSD: starting 90-second grace period (net ffffffff81d06400)
[   15.068671] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   15.283067] IPv6: ADDRCONF(NETDEV_UP): enp0s31f6: link is not ready
[   18.070482] e1000e: enp0s31f6 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx
[   18.070484] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[   18.070528] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready
======================================================================================================
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at 810000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at 80a000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at e000 [size=256]
	Region 5: Memory at 88a00000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [200 v1] #15
	Capabilities: [270 v1] #19
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] #13
	Capabilities: [2d0 v1] #1b
	Capabilities: [328 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [330 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 256, stride: 1, Device ID: 692f
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000000820000000 (64-bit, prefetchable)
		Region 2: Memory at 0000000808000000 (64-bit, prefetchable)
		Region 5: Memory at 88c00000 (32-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [400 v1] Vendor Specific Information: ID=0002 Rev=1 Len=070 <?>
	Kernel modules: amdgpu

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Alex,

How do you know "particular this system offers no 64-bit MMIO", from dmesg log?

-Collins Cheng


-----Original Message-----
From: Cheng, Collins 
Sent: Wednesday, May 24, 2017 4:56 PM
To: 'Alex Williamson' <alex.williamson@redhat.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

Hi Alex W, Alex D,

I just tried two options, one is enable "Above 4G Decoding" in BIOS setup menu, the other is add "pci=realloc=off" in grub. Both can fix this issue. Please see the attached log files.

Previously I thought "Above 4G Decoding" is enabled, but it is off when I looked CMOS setup today.

For now I think we have a solution. For the system that supports "Above 4G Decoding", user should enable it when use a SR-IOV supported device. For the system that doesn't support "Above 4G Decoding", user needs to add "pci=realloc=off" in grub.

Potentially I think kernel still needs to find a way to avoid this issue happen, like keeps the resource as the BIOS assigned value if there is a failure on device's resource reallocation.


-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com] 
Sent: Wednesday, May 24, 2017 2:20 AM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Tue, 23 May 2017 03:41:21 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex,
> 
> I owe you a dmesg log. Attachment are two log files. 1.txt is without "pci=earlydump", 2.txt is with "pci=earlydump". The platform is an ASUS Z170-A motherboard that doesn't support SR-IOV. The graphics card is AMD FirePro S7150 card which enabled SR-IOV. 
> 
> You could find the error info like below in both logs. From the log, kernel failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (256MB needed), but kernel reallocated resource for BAR9 which is for VF. You are right, the real bug that is something goes wrong with the reallocation leaving the PF without resources.
> 
> [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
> [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
> [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
> [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
> [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
> [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]

I've tried to extract more of the relevant resizing efforts below, perhaps Yinghai or others can make more out of it.  In particular this system offers no 64-bit MMIO and we'll never manage to allocate the necessary SR-IOV resources without it.  AIUI, the PCI core won't try to use anything outside the ACPI _CRS data without the option pci=nocrs.
This might present a second alternative in addition to the pci=realloc=off, which is actually suggested by the kernel below.  So I think we have at least two potential workarounds in the code as it exists today, one leaving SR-IOV disabled, the other (hopefully) enabling it using 64bit MMIO not described by the system BIOS.
Certainly an improvement would still be detecting the impossible reallocation problem without nocrs and abandoning the process and of course to revert the process before leaving more BARs unprogrammed than we started with.  Thanks,

Alex

[    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
[    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
[    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
...
[    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.896590] pci 0000:01:00.0: supports D1 D2
[    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
[    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
[    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
[    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
[    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
...
[    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.943586] vgaarb: loaded
[    0.943586] vgaarb: bridge control possible 0000:01:00.0
...
[    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
[    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
...
[    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
[    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
[    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] released
[    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64bit pref] released
[    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adfffff]
[    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
[    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
[    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
[    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff]
...
[    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
[    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] released
[    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] released
[    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64bit pref] released
[    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 64bit pref] to [bus 01] add_size 48000000 add_align 8000000
[    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] to [bus 01] add_size 2200000 add_align 400000
[    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
[    0.997692] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0x5fffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0xa7ffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x022fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997694] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x044fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997695] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997696] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x004fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 64bit pref]
[    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000000 64bit pref]
[    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cffffff]
[    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88afffff]
[    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3fffff 64bit pref]
[    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abfffff]
[    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acfffff]
[    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aefffff 64bit pref]
[    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0fffff 64bit pref] (expanded by 0x200000)
[    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7fffff] (expanded by 0x2200000)
[    0.997719] pci 0000:01:00.0: res[7]=[mem size 0x00000000 64bit pref] res_to_dev_res add_size 40000000 min_align 0
[    0.997720] pci 0000:01:00.0: res[9]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 2000000 min_align 0
[    0.997721] pci 0000:01:00.0: res[12]=[mem size 0x00000000] res_to_dev_res add_size 2000000 min_align 0
[    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff 64bit pref]
[    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3fffff]
[    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff 64bit pref]
[    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff]
[    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff 64bit pref]
[    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff]
[    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7fffff]
[    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff]
...
[    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if
you have problem, try booting with pci=realloc=off

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Alex,

How do you know "particular this system offers no 64-bit MMIO", from dmesg =
log?

-Collins Cheng


-----Original Message-----
From: Cheng, Collins=20
Sent: Wednesday, May 24, 2017 4:56 PM
To: 'Alex Williamson' <alex.williamson@redhat.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@go=
ogle.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher=
, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.=
com>; Yinghai Lu <yinghai@kernel.org>
Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV inc=
apable platform

Hi Alex W, Alex D,

I just tried two options, one is enable "Above 4G Decoding" in BIOS setup m=
enu, the other is add "pci=3Drealloc=3Doff" in grub. Both can fix this issu=
e. Please see the attached log files.

Previously I thought "Above 4G Decoding" is enabled, but it is off when I l=
ooked CMOS setup today.

For now I think we have a solution. For the system that supports "Above 4G =
Decoding", user should enable it when use a SR-IOV supported device. For th=
e system that doesn't support "Above 4G Decoding", user needs to add "pci=
=3Drealloc=3Doff" in grub.

Potentially I think kernel still needs to find a way to avoid this issue ha=
ppen, like keeps the resource as the BIOS assigned value if there is a fail=
ure on device's resource reallocation.


-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com]=20
Sent: Wednesday, May 24, 2017 2:20 AM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@go=
ogle.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher=
, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.=
com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV inc=
apable platform

On Tue, 23 May 2017 03:41:21 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex,
>=20
> I owe you a dmesg log. Attachment are two log files. 1.txt is without "pc=
i=3Dearlydump", 2.txt is with "pci=3Dearlydump". The platform is an ASUS Z1=
70-A motherboard that doesn't support SR-IOV. The graphics card is AMD Fire=
Pro S7150 card which enabled SR-IOV.=20
>=20
> You could find the error info like below in both logs. From the log, kern=
el failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (2=
56MB needed), but kernel reallocated resource for BAR9 which is for VF. You=
 are right, the real bug that is something goes wrong with the reallocation=
 leaving the PF without resources.
>=20
> [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000=
 64bit pref]
> [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x1000=
0000 64bit pref]
> [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000=
 64bit pref]
> [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x4000=
0000 64bit pref]
> [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfff=
ff 64bit pref]
> [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x0200000=
0]
> [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x020=
00000]
> [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfff=
ff 64bit pref]
> [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000=
]
> [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x0004=
0000]
> [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000=
 pref]
> [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x0002=
0000 pref]

I've tried to extract more of the relevant resizing efforts below, perhaps =
Yinghai or others can make more out of it.  In particular this system offer=
s no 64-bit MMIO and we'll never manage to allocate the necessary SR-IOV re=
sources without it.  AIUI, the PCI core won't try to use anything outside t=
he ACPI _CRS data without the option pci=3Dnocrs.
This might present a second alternative in addition to the pci=3Drealloc=3D=
off, which is actually suggested by the kernel below.  So I think we have a=
t least two potential workarounds in the code as it exists today, one leavi=
ng SR-IOV disabled, the other (hopefully) enabling it using 64bit MMIO not =
described by the system BIOS.
Certainly an improvement would still be detecting the impossible reallocati=
on problem without nocrs and abandoning the process and of course to revert=
 the process before leaving more BARs unprogrammed than we started with.  T=
hanks,

Alex

[    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window=
]
[    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window=
]
[    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bfff=
f window]
[    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdffffff=
f window]
[    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7ffff=
f window]
[    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
...
[    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit=
 pref]
[    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit=
 pref]
[    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.896590] pci 0000:01:00.0: supports D1 D2
[    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bi=
t pref]
[    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffff=
ff 64bit pref] (contains BAR0 for 8 VFs)
[    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bi=
t pref]
[    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffff=
ff 64bit pref] (contains BAR2 for 8 VFs)
[    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffff=
ff] (contains BAR5 for 8 VFs)
[    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff=
]
[    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff=
 64bit pref]
[    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
...
[    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=3Dio+mem,owns=
=3Dio+mem,locks=3Dnone
[    0.943586] vgaarb: loaded
[    0.943586] vgaarb: bridge control possible 0000:01:00.0
...
[    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 6=
4bit pref]
[    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x400000=
00 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 6=
4bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x020000=
00 64bit pref]
[    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000=
000]
[    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff=
]
[    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff=
 64bit pref]
...
[    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
[    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
[    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] re=
leased
[    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pref=
]
[    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pref=
]
[    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64b=
it pref] released
[    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 =
64bit pref]
[    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000=
000 64bit pref]
[    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adffff=
f]
[    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 6=
4bit pref]
[    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x100000=
00 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 6=
4bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x400000=
00 64bit pref]
[    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff=
 64bit pref]
[    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000=
000]
[    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff=
 64bit pref]
[    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x000400=
00]
[    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 p=
ref]
[    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x000200=
00 pref]
[    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff=
]
...
[    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
[    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pref=
]
[    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref=
]
[    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] re=
leased
[    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] re=
leased
[    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64b=
it pref] released
[    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 6=
4bit pref] to [bus 01] add_size 48000000 add_align 8000000
[    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] =
to [bus 01] add_size 2200000 add_align 400000
[    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 6=
4bit pref] to [bus 05] add_size 200000 add_align 100000
[    0.997692] pci 0000:00:1b.0: res[15]=3D[mem 0x08000000-0x5fffffff 64bit=
 pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[15]=3D[mem 0x08000000-0xa7ffffff 64bit=
 pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[14]=3D[mem 0x00100000-0x022fffff] res_=
to_dev_res add_size 2200000 min_align 400000
[    0.997694] pci 0000:00:1b.0: res[14]=3D[mem 0x00100000-0x044fffff] res_=
to_dev_res add_size 2200000 min_align 400000
[    0.997695] pci 0000:00:1d.0: res[15]=3D[mem 0x00100000-0x002fffff 64bit=
 pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997696] pci 0000:00:1d.0: res[15]=3D[mem 0x00100000-0x004fffff 64bit=
 pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 =
64bit pref]
[    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000=
000 64bit pref]
[    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cfffff=
f]
[    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88affff=
f]
[    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3ffff=
f 64bit pref]
[    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 =
64bit pref]
[    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000=
000 64bit pref]
[    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abffff=
f]
[    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acffff=
f]
[    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aeffff=
f 64bit pref]
[    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0ff=
fff 64bit pref] (expanded by 0x200000)
[    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7ff=
fff] (expanded by 0x2200000)
[    0.997719] pci 0000:01:00.0: res[7]=3D[mem size 0x00000000 64bit pref] =
res_to_dev_res add_size 40000000 min_align 0
[    0.997720] pci 0000:01:00.0: res[9]=3D[mem 0x00000000-0xfffffffffffffff=
f 64bit pref] res_to_dev_res add_size 2000000 min_align 0
[    0.997721] pci 0000:01:00.0: res[12]=3D[mem size 0x00000000] res_to_dev=
_res add_size 2000000 min_align 0
[    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 6=
4bit pref]
[    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x100000=
00 64bit pref]
[    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 6=
4bit pref]
[    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x400000=
00 64bit pref]
[    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff=
 64bit pref]
[    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3ffff=
f]
[    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff=
 64bit pref]
[    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff=
]
[    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 6=
4bit pref]
[    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x100000=
00 64bit pref]
[    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff=
 64bit pref]
[    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff=
]
[    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7ffff=
f]
[    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff=
 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 6=
4bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x400000=
00 64bit pref]
[    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff=
]
...
[    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if
you have problem, try booting with pci=3Drealloc=3Doff

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alex Williamson]

On Wed, 24 May 2017 08:57:53 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex,
> 
> How do you know "particular this system offers no 64-bit MMIO", from dmesg log?

>From this:

> [    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
> [    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
> [    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
> [    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
> [    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
> [    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]

There are only 32-bit ranges listed.  Thanks,

Alex

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Alex W,

I don't need the kernel patch anymore. However it looks the kernel could be improved to handle this more gracefully when PCI resource allocation fail. Do you have a plan to improve it in kernel PCI code?

-Collins Cheng


-----Original Message-----
From: Cheng, Collins 
Sent: Wednesday, May 24, 2017 4:56 PM
To: 'Alex Williamson' <alex.williamson@redhat.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

Hi Alex W, Alex D,

I just tried two options, one is enable "Above 4G Decoding" in BIOS setup menu, the other is add "pci=realloc=off" in grub. Both can fix this issue. Please see the attached log files.

Previously I thought "Above 4G Decoding" is enabled, but it is off when I looked CMOS setup today.

For now I think we have a solution. For the system that supports "Above 4G Decoding", user should enable it when use a SR-IOV supported device. For the system that doesn't support "Above 4G Decoding", user needs to add "pci=realloc=off" in grub.

Potentially I think kernel still needs to find a way to avoid this issue happen, like keeps the resource as the BIOS assigned value if there is a failure on device's resource reallocation.


-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com] 
Sent: Wednesday, May 24, 2017 2:20 AM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Tue, 23 May 2017 03:41:21 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex,
> 
> I owe you a dmesg log. Attachment are two log files. 1.txt is without "pci=earlydump", 2.txt is with "pci=earlydump". The platform is an ASUS Z170-A motherboard that doesn't support SR-IOV. The graphics card is AMD FirePro S7150 card which enabled SR-IOV. 
> 
> You could find the error info like below in both logs. From the log, kernel failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (256MB needed), but kernel reallocated resource for BAR9 which is for VF. You are right, the real bug that is something goes wrong with the reallocation leaving the PF without resources.
> 
> [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
> [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
> [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
> [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
> [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
> [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]

I've tried to extract more of the relevant resizing efforts below, perhaps Yinghai or others can make more out of it.  In particular this system offers no 64-bit MMIO and we'll never manage to allocate the necessary SR-IOV resources without it.  AIUI, the PCI core won't try to use anything outside the ACPI _CRS data without the option pci=nocrs.
This might present a second alternative in addition to the pci=realloc=off, which is actually suggested by the kernel below.  So I think we have at least two potential workarounds in the code as it exists today, one leaving SR-IOV disabled, the other (hopefully) enabling it using 64bit MMIO not described by the system BIOS.
Certainly an improvement would still be detecting the impossible reallocation problem without nocrs and abandoning the process and of course to revert the process before leaving more BARs unprogrammed than we started with.  Thanks,

Alex

[    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
[    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
[    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
...
[    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.896590] pci 0000:01:00.0: supports D1 D2
[    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
[    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
[    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
[    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
[    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
[    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
[    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
...
[    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.943586] vgaarb: loaded
[    0.943586] vgaarb: bridge control possible 0000:01:00.0
...
[    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
[    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
[    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
...
[    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
[    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
[    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] released
[    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pref]
[    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64bit pref] released
[    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adfffff]
[    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
[    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
[    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
[    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
[    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff]
...
[    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
[    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pref]
[    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref]
[    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] released
[    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] released
[    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64bit pref] released
[    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 64bit pref] to [bus 01] add_size 48000000 add_align 8000000
[    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] to [bus 01] add_size 2200000 add_align 400000
[    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
[    0.997692] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0x5fffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0xa7ffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x022fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997694] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x044fffff] res_to_dev_res add_size 2200000 min_align 400000
[    0.997695] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997696] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x004fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 64bit pref]
[    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000000 64bit pref]
[    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cffffff]
[    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88afffff]
[    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3fffff 64bit pref]
[    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
[    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
[    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abfffff]
[    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acfffff]
[    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aefffff 64bit pref]
[    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0fffff 64bit pref] (expanded by 0x200000)
[    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7fffff] (expanded by 0x2200000)
[    0.997719] pci 0000:01:00.0: res[7]=[mem size 0x00000000 64bit pref] res_to_dev_res add_size 40000000 min_align 0
[    0.997720] pci 0000:01:00.0: res[9]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 2000000 min_align 0
[    0.997721] pci 0000:01:00.0: res[12]=[mem size 0x00000000] res_to_dev_res add_size 2000000 min_align 0
[    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff 64bit pref]
[    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3fffff]
[    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff 64bit pref]
[    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff]
[    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff 64bit pref]
[    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff]
[    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7fffff]
[    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
[    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff]
...
[    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if
you have problem, try booting with pci=realloc=off

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Hi Alex W,

I don't need the kernel patch anymore. However it looks the kernel could be=
 improved to handle this more gracefully when PCI resource allocation fail.=
 Do you have a plan to improve it in kernel PCI code?

-Collins Cheng


-----Original Message-----
From: Cheng, Collins=20
Sent: Wednesday, May 24, 2017 4:56 PM
To: 'Alex Williamson' <alex.williamson@redhat.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@go=
ogle.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher=
, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.=
com>; Yinghai Lu <yinghai@kernel.org>
Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV inc=
apable platform

Hi Alex W, Alex D,

I just tried two options, one is enable "Above 4G Decoding" in BIOS setup m=
enu, the other is add "pci=3Drealloc=3Doff" in grub. Both can fix this issu=
e. Please see the attached log files.

Previously I thought "Above 4G Decoding" is enabled, but it is off when I l=
ooked CMOS setup today.

For now I think we have a solution. For the system that supports "Above 4G =
Decoding", user should enable it when use a SR-IOV supported device. For th=
e system that doesn't support "Above 4G Decoding", user needs to add "pci=
=3Drealloc=3Doff" in grub.

Potentially I think kernel still needs to find a way to avoid this issue ha=
ppen, like keeps the resource as the BIOS assigned value if there is a fail=
ure on device's resource reallocation.


-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com]=20
Sent: Wednesday, May 24, 2017 2:20 AM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@go=
ogle.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher=
, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.=
com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV inc=
apable platform

On Tue, 23 May 2017 03:41:21 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex,
>=20
> I owe you a dmesg log. Attachment are two log files. 1.txt is without "pc=
i=3Dearlydump", 2.txt is with "pci=3Dearlydump". The platform is an ASUS Z1=
70-A motherboard that doesn't support SR-IOV. The graphics card is AMD Fire=
Pro S7150 card which enabled SR-IOV.=20
>=20
> You could find the error info like below in both logs. From the log, kern=
el failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (2=
56MB needed), but kernel reallocated resource for BAR9 which is for VF. You=
 are right, the real bug that is something goes wrong with the reallocation=
 leaving the PF without resources.
>=20
> [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000=
 64bit pref]
> [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x1000=
0000 64bit pref]
> [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000=
 64bit pref]
> [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x4000=
0000 64bit pref]
> [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfff=
ff 64bit pref]
> [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x0200000=
0]
> [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x020=
00000]
> [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfff=
ff 64bit pref]
> [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000=
]
> [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x0004=
0000]
> [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000=
 pref]
> [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x0002=
0000 pref]

I've tried to extract more of the relevant resizing efforts below, perhaps =
Yinghai or others can make more out of it.  In particular this system offer=
s no 64-bit MMIO and we'll never manage to allocate the necessary SR-IOV re=
sources without it.  AIUI, the PCI core won't try to use anything outside t=
he ACPI _CRS data without the option pci=3Dnocrs.
This might present a second alternative in addition to the pci=3Drealloc=3D=
off, which is actually suggested by the kernel below.  So I think we have a=
t least two potential workarounds in the code as it exists today, one leavi=
ng SR-IOV disabled, the other (hopefully) enabling it using 64bit MMIO not =
described by the system BIOS.
Certainly an improvement would still be detecting the impossible reallocati=
on problem without nocrs and abandoning the process and of course to revert=
 the process before leaving more BARs unprogrammed than we started with.  T=
hanks,

Alex

[    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window=
]
[    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window=
]
[    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bfff=
f window]
[    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdffffff=
f window]
[    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7ffff=
f window]
[    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
...
[    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
[    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit=
 pref]
[    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit=
 pref]
[    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
[    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
[    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
[    0.896590] pci 0000:01:00.0: supports D1 D2
[    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bi=
t pref]
[    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffff=
ff 64bit pref] (contains BAR0 for 8 VFs)
[    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bi=
t pref]
[    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffff=
ff 64bit pref] (contains BAR2 for 8 VFs)
[    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
[    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffff=
ff] (contains BAR5 for 8 VFs)
[    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
[    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff=
]
[    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff=
 64bit pref]
[    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
...
[    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
[    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=3Dio+mem,owns=
=3Dio+mem,locks=3Dnone
[    0.943586] vgaarb: loaded
[    0.943586] vgaarb: bridge control possible 0000:01:00.0
...
[    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 6=
4bit pref]
[    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x400000=
00 64bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 6=
4bit pref]
[    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x020000=
00 64bit pref]
[    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000=
000]
[    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff=
]
[    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff=
 64bit pref]
...
[    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
[    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
[    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
[    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] re=
leased
[    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pref=
]
[    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pref=
]
[    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64b=
it pref] released
[    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 =
64bit pref]
[    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000=
000 64bit pref]
[    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adffff=
f]
[    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 6=
4bit pref]
[    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x100000=
00 64bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 6=
4bit pref]
[    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x400000=
00 64bit pref]
[    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff=
 64bit pref]
[    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
[    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000=
000]
[    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff=
 64bit pref]
[    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
[    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x000400=
00]
[    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 p=
ref]
[    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x000200=
00 pref]
[    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff=
]
...
[    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
[    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pref=
]
[    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref=
]
[    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] re=
leased
[    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
[    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] re=
leased
[    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
[    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64b=
it pref] released
[    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
[    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 6=
4bit pref] to [bus 01] add_size 48000000 add_align 8000000
[    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] =
to [bus 01] add_size 2200000 add_align 400000
[    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 6=
4bit pref] to [bus 05] add_size 200000 add_align 100000
[    0.997692] pci 0000:00:1b.0: res[15]=3D[mem 0x08000000-0x5fffffff 64bit=
 pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[15]=3D[mem 0x08000000-0xa7ffffff 64bit=
 pref] res_to_dev_res add_size 48000000 min_align 8000000
[    0.997693] pci 0000:00:1b.0: res[14]=3D[mem 0x00100000-0x022fffff] res_=
to_dev_res add_size 2200000 min_align 400000
[    0.997694] pci 0000:00:1b.0: res[14]=3D[mem 0x00100000-0x044fffff] res_=
to_dev_res add_size 2200000 min_align 400000
[    0.997695] pci 0000:00:1d.0: res[15]=3D[mem 0x00100000-0x002fffff 64bit=
 pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997696] pci 0000:00:1d.0: res[15]=3D[mem 0x00100000-0x004fffff 64bit=
 pref] res_to_dev_res add_size 200000 min_align 100000
[    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 =
64bit pref]
[    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000=
000 64bit pref]
[    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cfffff=
f]
[    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88affff=
f]
[    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3ffff=
f 64bit pref]
[    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 =
64bit pref]
[    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000=
000 64bit pref]
[    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abffff=
f]
[    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acffff=
f]
[    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aeffff=
f 64bit pref]
[    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0ff=
fff 64bit pref] (expanded by 0x200000)
[    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7ff=
fff] (expanded by 0x2200000)
[    0.997719] pci 0000:01:00.0: res[7]=3D[mem size 0x00000000 64bit pref] =
res_to_dev_res add_size 40000000 min_align 0
[    0.997720] pci 0000:01:00.0: res[9]=3D[mem 0x00000000-0xfffffffffffffff=
f 64bit pref] res_to_dev_res add_size 2000000 min_align 0
[    0.997721] pci 0000:01:00.0: res[12]=3D[mem size 0x00000000] res_to_dev=
_res add_size 2000000 min_align 0
[    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 6=
4bit pref]
[    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x100000=
00 64bit pref]
[    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 6=
4bit pref]
[    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x400000=
00 64bit pref]
[    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff=
 64bit pref]
[    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3ffff=
f]
[    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff=
 64bit pref]
[    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff=
]
[    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 6=
4bit pref]
[    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x100000=
00 64bit pref]
[    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff=
 64bit pref]
[    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff=
]
[    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7ffff=
f]
[    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff=
 64bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 6=
4bit pref]
[    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x400000=
00 64bit pref]
[    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
[    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
[    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff=
]
...
[    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if
you have problem, try booting with pci=3Drealloc=3Doff

Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Alex Williamson]

On Fri, 26 May 2017 01:52:35 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex W,
> 
> I don't need the kernel patch anymore. However it looks the kernel could be improved to handle this more gracefully when PCI resource allocation fail. Do you have a plan to improve it in kernel PCI code?

I don't have a device capable of reproducing and I'm currently working
on issues elsewhere.  If you don't plan to continue working on it, I'd
suggest filing a bug at bugzilla.kernel.org so that we can at least
track the problem.  Thanks,

Alex

> -----Original Message-----
> From: Cheng, Collins 
> Sent: Wednesday, May 24, 2017 4:56 PM
> To: 'Alex Williamson' <alex.williamson@redhat.com>
> Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
> Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform
> 
> Hi Alex W, Alex D,
> 
> I just tried two options, one is enable "Above 4G Decoding" in BIOS setup menu, the other is add "pci=realloc=off" in grub. Both can fix this issue. Please see the attached log files.
> 
> Previously I thought "Above 4G Decoding" is enabled, but it is off when I looked CMOS setup today.
> 
> For now I think we have a solution. For the system that supports "Above 4G Decoding", user should enable it when use a SR-IOV supported device. For the system that doesn't support "Above 4G Decoding", user needs to add "pci=realloc=off" in grub.
> 
> Potentially I think kernel still needs to find a way to avoid this issue happen, like keeps the resource as the BIOS assigned value if there is a failure on device's resource reallocation.
> 
> 
> -Collins Cheng
> 
> 
> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com] 
> Sent: Wednesday, May 24, 2017 2:20 AM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform
> 
> On Tue, 23 May 2017 03:41:21 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> 
> > Hi Alex,
> > 
> > I owe you a dmesg log. Attachment are two log files. 1.txt is without "pci=earlydump", 2.txt is with "pci=earlydump". The platform is an ASUS Z170-A motherboard that doesn't support SR-IOV. The graphics card is AMD FirePro S7150 card which enabled SR-IOV. 
> > 
> > You could find the error info like below in both logs. From the log, kernel failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (256MB needed), but kernel reallocated resource for BAR9 which is for VF. You are right, the real bug that is something goes wrong with the reallocation leaving the PF without resources.
> > 
> > [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> > [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> > [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> > [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> > [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
> > [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> > [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> > [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
> > [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
> > [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
> > [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
> > [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]  
> 
> I've tried to extract more of the relevant resizing efforts below, perhaps Yinghai or others can make more out of it.  In particular this system offers no 64-bit MMIO and we'll never manage to allocate the necessary SR-IOV resources without it.  AIUI, the PCI core won't try to use anything outside the ACPI _CRS data without the option pci=nocrs.
> This might present a second alternative in addition to the pci=realloc=off, which is actually suggested by the kernel below.  So I think we have at least two potential workarounds in the code as it exists today, one leaving SR-IOV disabled, the other (hopefully) enabling it using 64bit MMIO not described by the system BIOS.
> Certainly an improvement would still be detecting the impossible reallocation problem without nocrs and abandoning the process and of course to revert the process before leaving more BARs unprogrammed than we started with.  Thanks,
> 
> Alex
> 
> [    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
> [    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
> [    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
> [    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
> [    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
> [    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
> ...
> [    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
> [    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
> [    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit pref]
> [    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
> [    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
> [    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
> [    0.896590] pci 0000:01:00.0: supports D1 D2
> [    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
> [    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
> [    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
> [    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
> [    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
> [    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
> [    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
> [    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
> [    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
> [    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
> [    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
> ...
> [    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
> [    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
> [    0.943586] vgaarb: loaded
> [    0.943586] vgaarb: bridge control possible 0000:01:00.0
> ...
> [    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
> [    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
> [    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> [    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> [    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
> [    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
> ...
> [    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
> [    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
> [    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
> [    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] released
> [    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pref]
> [    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pref]
> [    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64bit pref] released
> [    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
> [    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
> [    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adfffff]
> [    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
> [    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> [    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> [    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
> [    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
> [    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
> [    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
> [    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
> [    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff]
> ...
> [    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
> [    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pref]
> [    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref]
> [    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] released
> [    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
> [    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] released
> [    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
> [    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64bit pref] released
> [    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
> [    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 64bit pref] to [bus 01] add_size 48000000 add_align 8000000
> [    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] to [bus 01] add_size 2200000 add_align 400000
> [    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
> [    0.997692] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0x5fffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
> [    0.997693] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0xa7ffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
> [    0.997693] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x022fffff] res_to_dev_res add_size 2200000 min_align 400000
> [    0.997694] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x044fffff] res_to_dev_res add_size 2200000 min_align 400000
> [    0.997695] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
> [    0.997696] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x004fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
> [    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 64bit pref]
> [    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000000 64bit pref]
> [    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cffffff]
> [    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88afffff]
> [    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3fffff 64bit pref]
> [    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
> [    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
> [    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abfffff]
> [    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acfffff]
> [    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aefffff 64bit pref]
> [    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0fffff 64bit pref] (expanded by 0x200000)
> [    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7fffff] (expanded by 0x2200000)
> [    0.997719] pci 0000:01:00.0: res[7]=[mem size 0x00000000 64bit pref] res_to_dev_res add_size 40000000 min_align 0
> [    0.997720] pci 0000:01:00.0: res[9]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 2000000 min_align 0
> [    0.997721] pci 0000:01:00.0: res[12]=[mem size 0x00000000] res_to_dev_res add_size 2000000 min_align 0
> [    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff 64bit pref]
> [    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3fffff]
> [    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff 64bit pref]
> [    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff]
> [    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff 64bit pref]
> [    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff]
> [    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7fffff]
> [    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff 64bit pref]
> [    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff]
> ...
> [    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if
> you have problem, try booting with pci=realloc=off

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Thanks Alex. I know it is difficult to reproduce this issue on your side.

I just created a bug to track it at https://bugzilla.kernel.org/show_bug.cgi?id=195891


-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com] 
Sent: Friday, May 26, 2017 11:52 PM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

On Fri, 26 May 2017 01:52:35 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex W,
> 
> I don't need the kernel patch anymore. However it looks the kernel could be improved to handle this more gracefully when PCI resource allocation fail. Do you have a plan to improve it in kernel PCI code?

I don't have a device capable of reproducing and I'm currently working on issues elsewhere.  If you don't plan to continue working on it, I'd suggest filing a bug at bugzilla.kernel.org so that we can at least track the problem.  Thanks,

Alex

> -----Original Message-----
> From: Cheng, Collins
> Sent: Wednesday, May 24, 2017 4:56 PM
> To: 'Alex Williamson' <alex.williamson@redhat.com>
> Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas 
> <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> linux-kernel@vger.kernel.org; Deucher, Alexander 
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; 
> Yinghai Lu <yinghai@kernel.org>
> Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> SR-IOV incapable platform
> 
> Hi Alex W, Alex D,
> 
> I just tried two options, one is enable "Above 4G Decoding" in BIOS setup menu, the other is add "pci=realloc=off" in grub. Both can fix this issue. Please see the attached log files.
> 
> Previously I thought "Above 4G Decoding" is enabled, but it is off when I looked CMOS setup today.
> 
> For now I think we have a solution. For the system that supports "Above 4G Decoding", user should enable it when use a SR-IOV supported device. For the system that doesn't support "Above 4G Decoding", user needs to add "pci=realloc=off" in grub.
> 
> Potentially I think kernel still needs to find a way to avoid this issue happen, like keeps the resource as the BIOS assigned value if there is a failure on device's resource reallocation.
> 
> 
> -Collins Cheng
> 
> 
> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Wednesday, May 24, 2017 2:20 AM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas 
> <bhelgaas@google.com>; linux-pci@vger.kernel.org; 
> linux-kernel@vger.kernel.org; Deucher, Alexander 
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>; 
> Yinghai Lu <yinghai@kernel.org>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the 
> SR-IOV incapable platform
> 
> On Tue, 23 May 2017 03:41:21 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
> 
> > Hi Alex,
> > 
> > I owe you a dmesg log. Attachment are two log files. 1.txt is without "pci=earlydump", 2.txt is with "pci=earlydump". The platform is an ASUS Z170-A motherboard that doesn't support SR-IOV. The graphics card is AMD FirePro S7150 card which enabled SR-IOV. 
> > 
> > You could find the error info like below in both logs. From the log, kernel failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR (256MB needed), but kernel reallocated resource for BAR9 which is for VF. You are right, the real bug that is something goes wrong with the reallocation leaving the PF without resources.
> > 
> > [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> > [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> > [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> > [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> > [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
> > [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> > [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> > [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
> > [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
> > [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
> > [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
> > [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]  
> 
> I've tried to extract more of the relevant resizing efforts below, perhaps Yinghai or others can make more out of it.  In particular this system offers no 64-bit MMIO and we'll never manage to allocate the necessary SR-IOV resources without it.  AIUI, the PCI core won't try to use anything outside the ACPI _CRS data without the option pci=nocrs.
> This might present a second alternative in addition to the pci=realloc=off, which is actually suggested by the kernel below.  So I think we have at least two potential workarounds in the code as it exists today, one leaving SR-IOV disabled, the other (hopefully) enabling it using 64bit MMIO not described by the system BIOS.
> Certainly an improvement would still be detecting the impossible 
> reallocation problem without nocrs and abandoning the process and of 
> course to revert the process before leaving more BARs unprogrammed 
> than we started with.  Thanks,
> 
> Alex
> 
> [    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
> [    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
> [    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
> [    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdfffffff window]
> [    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
> [    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
> ...
> [    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
> [    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
> [    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64bit pref]
> [    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
> [    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
> [    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pref]
> [    0.896590] pci 0000:01:00.0: supports D1 D2
> [    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
> [    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64bit pref]
> [    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fffffff 64bit pref] (contains BAR0 for 8 VFs)
> [    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64bit pref]
> [    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ffffff 64bit pref] (contains BAR2 for 8 VFs)
> [    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
> [    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ffffff] (contains BAR5 for 8 VFs)
> [    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
> [    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
> [    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
> [    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 02
> ...
> [    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
> [    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
> [    0.943586] vgaarb: loaded
> [    0.943586] vgaarb: bridge control possible 0000:01:00.0
> ...
> [    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000 64bit pref]
> [    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x02000000 64bit pref]
> [    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> [    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> [    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefffff]
> [    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fffff 64bit pref]
> ...
> [    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
> [    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
> [    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
> [    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] released
> [    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pref]
> [    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pref]
> [    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 64bit pref] released
> [    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
> [    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
> [    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adfffff]
> [    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfffff 64bit pref]
> [    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000000]
> [    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x02000000]
> [    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfffff 64bit pref]
> [    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000]
> [    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00040000]
> [    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]
> [    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]
> [    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfffff]
> ...
> [    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
> [    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pref]
> [    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pref]
> [    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] released
> [    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
> [    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] released
> [    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
> [    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 64bit pref] released
> [    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
> [    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff 64bit pref] to [bus 01] add_size 48000000 add_align 8000000
> [    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff] to [bus 01] add_size 2200000 add_align 400000
> [    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
> [    0.997692] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0x5fffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
> [    0.997693] pci 0000:00:1b.0: res[15]=[mem 0x08000000-0xa7ffffff 64bit pref] res_to_dev_res add_size 48000000 min_align 8000000
> [    0.997693] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x022fffff] res_to_dev_res add_size 2200000 min_align 400000
> [    0.997694] pci 0000:00:1b.0: res[14]=[mem 0x00100000-0x044fffff] res_to_dev_res add_size 2200000 min_align 400000
> [    0.997695] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
> [    0.997696] pci 0000:00:1d.0: res[15]=[mem 0x00100000-0x004fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
> [    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa0000000 64bit pref]
> [    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa0000000 64bit pref]
> [    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cffffff]
> [    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88afffff]
> [    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3fffff 64bit pref]
> [    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x58000000 64bit pref]
> [    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x58000000 64bit pref]
> [    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abfffff]
> [    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acfffff]
> [    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aefffff 64bit pref]
> [    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0fffff 64bit pref] (expanded by 0x200000)
> [    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7fffff] (expanded by 0x2200000)
> [    0.997719] pci 0000:01:00.0: res[7]=[mem size 0x00000000 64bit pref] res_to_dev_res add_size 40000000 min_align 0
> [    0.997720] pci 0000:01:00.0: res[9]=[mem 0x00000000-0xffffffffffffffff 64bit pref] res_to_dev_res add_size 2000000 min_align 0
> [    0.997721] pci 0000:01:00.0: res[12]=[mem size 0x00000000] res_to_dev_res add_size 2000000 min_align 0
> [    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fffff 64bit pref]
> [    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3fffff]
> [    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fffff 64bit pref]
> [    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ffff]
> [    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
> [    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
> [    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fffff 64bit pref]
> [    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ffff]
> [    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7fffff]
> [    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fffff 64bit pref]
> [    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000 64bit pref]
> [    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40000000 64bit pref]
> [    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fffff]
> ...
> [    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if
> you have problem, try booting with pci=realloc=off

RE: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

[Cheng, Collins]

Thanks Alex. I know it is difficult to reproduce this issue on your side.

I just created a bug to track it at https://bugzilla.kernel.org/show_bug.cg=
i?id=3D195891


-Collins Cheng


-----Original Message-----
From: Alex Williamson [mailto:alex.williamson@redhat.com]=20
Sent: Friday, May 26, 2017 11:52 PM
To: Cheng, Collins <Collins.Cheng@amd.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas <bhelgaas@go=
ogle.com>; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Deucher=
, Alexander <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.=
com>; Yinghai Lu <yinghai@kernel.org>
Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV inc=
apable platform

On Fri, 26 May 2017 01:52:35 +0000
"Cheng, Collins" <Collins.Cheng@amd.com> wrote:

> Hi Alex W,
>=20
> I don't need the kernel patch anymore. However it looks the kernel could =
be improved to handle this more gracefully when PCI resource allocation fai=
l. Do you have a plan to improve it in kernel PCI code?

I don't have a device capable of reproducing and I'm currently working on i=
ssues elsewhere.  If you don't plan to continue working on it, I'd suggest =
filing a bug at bugzilla.kernel.org so that we can at least track the probl=
em.  Thanks,

Alex

> -----Original Message-----
> From: Cheng, Collins
> Sent: Wednesday, May 24, 2017 4:56 PM
> To: 'Alex Williamson' <alex.williamson@redhat.com>
> Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas=20
> <bhelgaas@google.com>; linux-pci@vger.kernel.org;=20
> linux-kernel@vger.kernel.org; Deucher, Alexander=20
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>;=20
> Yinghai Lu <yinghai@kernel.org>
> Subject: RE: [PATCH] PCI: Make SR-IOV capable GPU working on the=20
> SR-IOV incapable platform
>=20
> Hi Alex W, Alex D,
>=20
> I just tried two options, one is enable "Above 4G Decoding" in BIOS setup=
 menu, the other is add "pci=3Drealloc=3Doff" in grub. Both can fix this is=
sue. Please see the attached log files.
>=20
> Previously I thought "Above 4G Decoding" is enabled, but it is off when I=
 looked CMOS setup today.
>=20
> For now I think we have a solution. For the system that supports "Above 4=
G Decoding", user should enable it when use a SR-IOV supported device. For =
the system that doesn't support "Above 4G Decoding", user needs to add "pci=
=3Drealloc=3Doff" in grub.
>=20
> Potentially I think kernel still needs to find a way to avoid this issue =
happen, like keeps the resource as the BIOS assigned value if there is a fa=
ilure on device's resource reallocation.
>=20
>=20
> -Collins Cheng
>=20
>=20
> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Wednesday, May 24, 2017 2:20 AM
> To: Cheng, Collins <Collins.Cheng@amd.com>
> Cc: Alexander Duyck <alexander.duyck@gmail.com>; Bjorn Helgaas=20
> <bhelgaas@google.com>; linux-pci@vger.kernel.org;=20
> linux-kernel@vger.kernel.org; Deucher, Alexander=20
> <Alexander.Deucher@amd.com>; Zytaruk, Kelly <Kelly.Zytaruk@amd.com>;=20
> Yinghai Lu <yinghai@kernel.org>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the=20
> SR-IOV incapable platform
>=20
> On Tue, 23 May 2017 03:41:21 +0000
> "Cheng, Collins" <Collins.Cheng@amd.com> wrote:
>=20
> > Hi Alex,
> >=20
> > I owe you a dmesg log. Attachment are two log files. 1.txt is without "=
pci=3Dearlydump", 2.txt is with "pci=3Dearlydump". The platform is an ASUS =
Z170-A motherboard that doesn't support SR-IOV. The graphics card is AMD Fi=
rePro S7150 card which enabled SR-IOV.=20
> >=20
> > You could find the error info like below in both logs. From the log, ke=
rnel failed to reallocate resource for BAR0 which is PF's Frame Buffer BAR =
(256MB needed), but kernel reallocated resource for BAR9 which is for VF. Y=
ou are right, the real bug that is something goes wrong with the reallocati=
on leaving the PF without resources.
> >=20
> > [    0.992976] pci 0000:01:00.0: BAR 0: no space for [mem size 0x100000=
00 64bit pref]
> > [    0.992976] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10=
000000 64bit pref]
> > [    0.992977] pci 0000:01:00.0: BAR 7: no space for [mem size 0x400000=
00 64bit pref]
> > [    0.992978] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x40=
000000 64bit pref]
> > [    0.992979] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abf=
ffff 64bit pref]
> > [    0.992986] pci 0000:01:00.0: BAR 12: no space for [mem size 0x02000=
000]
> > [    0.992986] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x0=
2000000]
> > [    0.992988] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adf=
ffff 64bit pref]
> > [    0.992994] pci 0000:01:00.0: BAR 5: no space for [mem size 0x000400=
00]
> > [    0.992995] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x00=
040000]
> > [    0.992996] pci 0000:01:00.0: BAR 6: no space for [mem size 0x000200=
00 pref]
> > [    0.992997] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00=
020000 pref] =20
>=20
> I've tried to extract more of the relevant resizing efforts below, perhap=
s Yinghai or others can make more out of it.  In particular this system off=
ers no 64-bit MMIO and we'll never manage to allocate the necessary SR-IOV =
resources without it.  AIUI, the PCI core won't try to use anything outside=
 the ACPI _CRS data without the option pci=3Dnocrs.
> This might present a second alternative in addition to the pci=3Drealloc=
=3Doff, which is actually suggested by the kernel below.  So I think we hav=
e at least two potential workarounds in the code as it exists today, one le=
aving SR-IOV disabled, the other (hopefully) enabling it using 64bit MMIO n=
ot described by the system BIOS.
> Certainly an improvement would still be detecting the impossible=20
> reallocation problem without nocrs and abandoning the process and of=20
> course to revert the process before leaving more BARs unprogrammed=20
> than we started with.  Thanks,
>=20
> Alex
>=20
> [    0.891319] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 wind=
ow]
> [    0.891321] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff wind=
ow]
> [    0.891322] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bf=
fff window]
> [    0.891323] pci_bus 0000:00: root bus resource [mem 0x88800000-0xdffff=
fff window]
> [    0.891324] pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7ff=
fff window]
> [    0.891325] pci_bus 0000:00: root bus resource [bus 00-fe]
> ...
> [    0.896481] pci 0000:01:00.0: [1002:6929] type 00 class 0x030000
> [    0.896496] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64b=
it pref]
> [    0.896506] pci 0000:01:00.0: reg 0x18: [mem 0xd0000000-0xd01fffff 64b=
it pref]
> [    0.896513] pci 0000:01:00.0: reg 0x20: [io  0xe000-0xe0ff]
> [    0.896519] pci 0000:01:00.0: reg 0x24: [mem 0xdfe00000-0xdfe3ffff]
> [    0.896526] pci 0000:01:00.0: reg 0x30: [mem 0xdfe40000-0xdfe5ffff pre=
f]
> [    0.896590] pci 0000:01:00.0: supports D1 D2
> [    0.896590] pci 0000:01:00.0: PME# supported from D1 D2 D3hot D3cold
> [    0.896625] pci 0000:01:00.0: reg 0x354: [mem 0x00000000-0x07ffffff 64=
bit pref]
> [    0.896626] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x3fff=
ffff 64bit pref] (contains BAR0 for 8 VFs)
> [    0.896634] pci 0000:01:00.0: reg 0x35c: [mem 0x00000000-0x003fffff 64=
bit pref]
> [    0.896635] pci 0000:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x01ff=
ffff 64bit pref] (contains BAR2 for 8 VFs)
> [    0.896646] pci 0000:01:00.0: reg 0x368: [mem 0x00000000-0x003fffff]
> [    0.896647] pci 0000:01:00.0: VF(n) BAR5 space: [mem 0x00000000-0x01ff=
ffff] (contains BAR5 for 8 VFs)
> [    0.896700] pci 0000:01:00.0: System wakeup disabled by ACPI
> [    0.906527] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.906544] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.906546] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefff=
ff]
> [    0.906549] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fff=
ff 64bit pref]
> [    0.906550] pci 0000:00:1b.0: bridge has subordinate 01 but max busn 0=
2
> ...
> [    0.943584] vgaarb: setting as boot device: PCI:0000:01:00.0
> [    0.943585] vgaarb: device added: PCI:0000:01:00.0,decodes=3Dio+mem,ow=
ns=3Dio+mem,locks=3Dnone
> [    0.943586] vgaarb: loaded
> [    0.943586] vgaarb: bridge control possible 0000:01:00.0
> ...
> [    0.997491] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000=
 64bit pref]
> [    0.997491] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x4000=
0000 64bit pref]
> [    0.997493] pci 0000:01:00.0: BAR 9: no space for [mem size 0x02000000=
 64bit pref]
> [    0.997493] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x0200=
0000 64bit pref]
> [    0.997495] pci 0000:01:00.0: BAR 12: no space for [mem size 0x0200000=
0]
> [    0.997495] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x020=
00000]
> [    0.997497] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997498] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.997501] pci 0000:00:1b.0:   bridge window [mem 0xdfe00000-0xdfefff=
ff]
> [    0.997502] pci 0000:00:1b.0:   bridge window [mem 0xc0000000-0xd01fff=
ff 64bit pref]
> ...
> [    0.997540] pci_bus 0000:00: No. 2 try to assign unassigned res
> [    0.997540] release child resource [mem 0xdfe00000-0xdfe3ffff]
> [    0.997540] release child resource [mem 0xdfe40000-0xdfe5ffff pref]
> [    0.997541] pci 0000:00:1b.0: resource 14 [mem 0xdfe00000-0xdfefffff] =
released
> [    0.997542] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997543] release child resource [mem 0xc0000000-0xcfffffff 64bit pr=
ef]
> [    0.997544] release child resource [mem 0xd0000000-0xd01fffff 64bit pr=
ef]
> [    0.997544] pci 0000:00:1b.0: resource 15 [mem 0xc0000000-0xd01fffff 6=
4bit pref] released
> [    0.997545] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997576] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x5800000=
0 64bit pref]
> [    0.997577] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x580=
00000 64bit pref]
> [    0.997578] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8adff=
fff]
> [    0.997583] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000=
 64bit pref]
> [    0.997583] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x1000=
0000 64bit pref]
> [    0.997585] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000=
 64bit pref]
> [    0.997585] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x4000=
0000 64bit pref]
> [    0.997587] pci 0000:01:00.0: BAR 9: assigned [mem 0x88c00000-0x8abfff=
ff 64bit pref]
> [    0.997593] pci 0000:01:00.0: BAR 12: no space for [mem size 0x0200000=
0]
> [    0.997594] pci 0000:01:00.0: BAR 12: failed to assign [mem size 0x020=
00000]
> [    0.997595] pci 0000:01:00.0: BAR 2: assigned [mem 0x8ac00000-0x8adfff=
ff 64bit pref]
> [    0.997602] pci 0000:01:00.0: BAR 5: no space for [mem size 0x00040000=
]
> [    0.997602] pci 0000:01:00.0: BAR 5: failed to assign [mem size 0x0004=
0000]
> [    0.997603] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000=
 pref]
> [    0.997604] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x0002=
0000 pref]
> [    0.997606] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997607] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.997609] pci 0000:00:1b.0:   bridge window [mem 0x88c00000-0x8adfff=
ff]
> ...
> [    0.997647] pci_bus 0000:00: No. 3 try to assign unassigned res
> [    0.997648] release child resource [mem 0x88c00000-0x8abfffff 64bit pr=
ef]
> [    0.997648] release child resource [mem 0x8ac00000-0x8adfffff 64bit pr=
ef]
> [    0.997649] pci 0000:00:1b.0: resource 14 [mem 0x88c00000-0x8adfffff] =
released
> [    0.997649] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997651] release child resource [mem 0xdfd00000-0xdfd07fff 64bit]
> [    0.997651] pci 0000:00:1c.0: resource 14 [mem 0xdfd00000-0xdfdfffff] =
released
> [    0.997652] pci 0000:00:1c.0: PCI bridge to [bus 02]
> [    0.997654] pci 0000:00:1d.0: resource 15 [mem 0x88a00000-0x88bfffff 6=
4bit pref] released
> [    0.997654] pci 0000:00:1d.0: PCI bridge to [bus 05]
> [    0.997664] pci 0000:00:1b.0: bridge window [mem 0x08000000-0x5fffffff=
 64bit pref] to [bus 01] add_size 48000000 add_align 8000000
> [    0.997666] pci 0000:00:1b.0: bridge window [mem 0x00100000-0x022fffff=
] to [bus 01] add_size 2200000 add_align 400000
> [    0.997687] pci 0000:00:1d.0: bridge window [mem 0x00100000-0x002fffff=
 64bit pref] to [bus 05] add_size 200000 add_align 100000
> [    0.997692] pci 0000:00:1b.0: res[15]=3D[mem 0x08000000-0x5fffffff 64b=
it pref] res_to_dev_res add_size 48000000 min_align 8000000
> [    0.997693] pci 0000:00:1b.0: res[15]=3D[mem 0x08000000-0xa7ffffff 64b=
it pref] res_to_dev_res add_size 48000000 min_align 8000000
> [    0.997693] pci 0000:00:1b.0: res[14]=3D[mem 0x00100000-0x022fffff] re=
s_to_dev_res add_size 2200000 min_align 400000
> [    0.997694] pci 0000:00:1b.0: res[14]=3D[mem 0x00100000-0x044fffff] re=
s_to_dev_res add_size 2200000 min_align 400000
> [    0.997695] pci 0000:00:1d.0: res[15]=3D[mem 0x00100000-0x002fffff 64b=
it pref] res_to_dev_res add_size 200000 min_align 100000
> [    0.997696] pci 0000:00:1d.0: res[15]=3D[mem 0x00100000-0x004fffff 64b=
it pref] res_to_dev_res add_size 200000 min_align 100000
> [    0.997698] pci 0000:00:1b.0: BAR 15: no space for [mem size 0xa000000=
0 64bit pref]
> [    0.997699] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0xa00=
00000 64bit pref]
> [    0.997700] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88c00000-0x8cfff=
fff]
> [    0.997701] pci 0000:00:1c.0: BAR 14: assigned [mem 0x88a00000-0x88aff=
fff]
> [    0.997702] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8d000000-0x8d3ff=
fff 64bit pref]
> [    0.997705] pci 0000:00:1b.0: BAR 15: no space for [mem size 0x5800000=
0 64bit pref]
> [    0.997706] pci 0000:00:1b.0: BAR 15: failed to assign [mem size 0x580=
00000 64bit pref]
> [    0.997707] pci 0000:00:1b.0: BAR 14: assigned [mem 0x88a00000-0x8abff=
fff]
> [    0.997708] pci 0000:00:1c.0: BAR 14: assigned [mem 0x8ac00000-0x8acff=
fff]
> [    0.997709] pci 0000:00:1d.0: BAR 15: assigned [mem 0x8ad00000-0x8aeff=
fff 64bit pref]
> [    0.997711] pci 0000:00:1d.0: BAR 15: reassigned [mem 0x8ad00000-0x8b0=
fffff 64bit pref] (expanded by 0x200000)
> [    0.997713] pci 0000:00:1b.0: BAR 14: reassigned [mem 0x8b400000-0x8f7=
fffff] (expanded by 0x2200000)
> [    0.997719] pci 0000:01:00.0: res[7]=3D[mem size 0x00000000 64bit pref=
] res_to_dev_res add_size 40000000 min_align 0
> [    0.997720] pci 0000:01:00.0: res[9]=3D[mem 0x00000000-0xfffffffffffff=
fff 64bit pref] res_to_dev_res add_size 2000000 min_align 0
> [    0.997721] pci 0000:01:00.0: res[12]=3D[mem size 0x00000000] res_to_d=
ev_res add_size 2000000 min_align 0
> [    0.997722] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000=
 64bit pref]
> [    0.997722] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x1000=
0000 64bit pref]
> [    0.997723] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000=
 64bit pref]
> [    0.997724] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x4000=
0000 64bit pref]
> [    0.997725] pci 0000:01:00.0: BAR 9: assigned [mem 0x8b400000-0x8d3fff=
ff 64bit pref]
> [    0.997731] pci 0000:01:00.0: BAR 12: assigned [mem 0x8d400000-0x8f3ff=
fff]
> [    0.997734] pci 0000:01:00.0: BAR 2: assigned [mem 0x8f400000-0x8f5fff=
ff 64bit pref]
> [    0.997740] pci 0000:01:00.0: BAR 5: assigned [mem 0x8f600000-0x8f63ff=
ff]
> [    0.997744] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000=
 64bit pref]
> [    0.997745] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x1000=
0000 64bit pref]
> [    0.997746] pci 0000:01:00.0: BAR 2: assigned [mem 0x8b400000-0x8b5fff=
ff 64bit pref]
> [    0.997753] pci 0000:01:00.0: BAR 5: assigned [mem 0x8b600000-0x8b63ff=
ff]
> [    0.997756] pci 0000:01:00.0: BAR 12: assigned [mem 0x8b800000-0x8d7ff=
fff]
> [    0.997758] pci 0000:01:00.0: BAR 9: assigned [mem 0x8d800000-0x8f7fff=
ff 64bit pref]
> [    0.997765] pci 0000:01:00.0: BAR 7: no space for [mem size 0x40000000=
 64bit pref]
> [    0.997765] pci 0000:01:00.0: BAR 7: failed to assign [mem size 0x4000=
0000 64bit pref]
> [    0.997767] pci 0000:00:1b.0: PCI bridge to [bus 01]
> [    0.997768] pci 0000:00:1b.0:   bridge window [io  0xe000-0xefff]
> [    0.997770] pci 0000:00:1b.0:   bridge window [mem 0x8b400000-0x8f7fff=
ff]
> ...
> [    0.997818] pci_bus 0000:00: Automatically enabled pci realloc, if
> you have problem, try booting with pci=3Drealloc=3Doff