[PATCH 0/2] Augment device matching its upstream Root Port's MPS logic

[PATCH 0/2] Augment device matching its upstream Root Port's MPS logic

[Myron Stowe]

In commit 27d868b5e6cf ("PCI: Set MPS to match upstream bridge") Keith made
sure every device's MPS setting matched its upstream bridge, making it more
likely that a hot-added devices would work in a system with an optimized MPS
configuration.

This series augments Keith's approach to include tuning down a Root Port's
MPS setting in the case where a hot-added device is not capable of matching
it (see: [1]).

Testing by Dongdong exposed a bug with the logic including Virtual Functions
(VFs).  VFs should not be included so a pre-cursor patch, 1/2, was added to
cover such.

[1] https://bugzilla.kernel.org/show_bug.cgi?id=200527

Myron Stowe (2):
      PCI: Skip MPS logic for Virtual Functions (VFs)
      PCI: Match Root Port's MPS to endpoint's MPSS as necessary


 drivers/pci/probe.c |   16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

-- 

[PATCH 1/2] PCI: Skip MPS logic for Virtual Functions (VFs)

[Myron Stowe]

Section 9.3.5.4 "Device Control Register" specifically shows both
Max_Payload_Size (MPS) and Max_Read_request_Size (MRRS) to be 'RsvdP' for
VFs in Table 9-16 [1].  Just prior to the table it states:
  "PF and VF functionality is defined in Section 7.5.3.4 except where
   noted in Table 9-16.  For VF fields marked 'RsvdP', the PF setting
   applies to the VF."

All of which implies that with respect to Max_Payload_Size Supported
(MPSS), MPS, and MRRS values, we should not be paying any attention to the
VF's fields, but rather only to the PF's.  Only looking at the PF's fields
also logically makes sense as it's the sole physical interface to the PCIe
bus.

[1] PCI Express Base Specification, Revision 4.0 Version 1.0 (2017).

Link: https://bugzilla.kernel.org/show_bug.cgi?id=200527
Cc: Keith Busch <keith.busch@intel.com>
Cc: Sinan Kaya <okaya@kernel.org>
Cc: Dongdong Liu <liudongdong3@huawei.com>
Cc: Jon Mason <jdmason@kudzu.us>
Fixes: 27d868b5e6cf ("PCI: Set MPS to match upstream bridge")
Cc: stable@vger.kernel.org # 4.3+
Signed-off-by: Myron Stowe <myron.stowe@redhat.com>
---
 drivers/pci/probe.c |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 611adcd..b285786 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1730,6 +1730,10 @@ static void pci_configure_mps(struct pci_dev *dev)
 	if (!pci_is_pcie(dev) || !bridge || !pci_is_pcie(bridge))
 		return;
 
+	/* MPS and MRRS fields are of type 'RsvdP' for VFs, short-circuit out */
+	if (dev->is_virtfn)
+		return;
+
 	mps = pcie_get_mps(dev);
 	p_mps = pcie_get_mps(bridge);
 

[PATCH 2/2] PCI: Match Root Port's MPS to endpoint's MPSS as necessary

[Myron Stowe]

In commit 27d868b5e6cf ("PCI: Set MPS to match upstream bridge"), we made
sure every device's MPS setting matches its upstream bridge, making it more
likely that a hot-added device will work in a system with an optimized MPS
configuration.

Recently I've started encountering systems where the endpoint device's MPSS
capability is less than its Root Port's current MPS value, thus the
endpoint is not capable of matching its upstream bridge's MPS setting (see:
bugzilla via "Link:" below).  This leaves the system vulnerable - the
upstream Root Port could respond with larger sized TLPs than the device
can handle, and the device will consider them to be 'Malformed'.

One could use the "pci=pcie_bus_safe" kernel parameter to resolve the
issue, but, it both forces a user to have to supply a kernel parameter to
get the system to function reliable, and may end up limiting MPS settings
of other, non-related, sub-topologies which could benefit from maintaining
their larger values.

This patch augments Keith's approach to include tuning down a Root Port's
MPS setting when its hot-added endpoint device is not capable of matching
it.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=200527
Cc: Keith Busch <keith.busch@intel.com>
Cc: Sinan Kaya <okaya@kernel.org>
Cc: Dongdong Liu <liudongdong3@huawei.com>
Acked-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Myron Stowe <myron.stowe@redhat.com>
---
 drivers/pci/probe.c |   12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index b285786..a1a243e 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1725,7 +1725,7 @@ int pci_setup_device(struct pci_dev *dev)
 static void pci_configure_mps(struct pci_dev *dev)
 {
 	struct pci_dev *bridge = pci_upstream_bridge(dev);
-	int mps, p_mps, rc;
+	int mps, mpss, p_mps, rc;
 
 	if (!pci_is_pcie(dev) || !bridge || !pci_is_pcie(bridge))
 		return;
@@ -1753,6 +1753,14 @@ static void pci_configure_mps(struct pci_dev *dev)
 	if (pcie_bus_config != PCIE_BUS_DEFAULT)
 		return;
 
+	mpss = 128 << dev->pcie_mpss;
+	if (mpss < p_mps && pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT) {
+		pcie_set_mps(bridge, mpss);
+		pci_info(dev, "Upstream bridge's Max Payload Size set to %d (was %d, max %d)\n",
+			 mpss, p_mps, 128 << bridge->pcie_mpss);
+		p_mps = pcie_get_mps(bridge);
+	}
+
 	rc = pcie_set_mps(dev, p_mps);
 	if (rc) {
 		pci_warn(dev, "can't set Max Payload Size to %d; if necessary, use \"pci=pcie_bus_safe\" and report a bug\n",
@@ -1761,7 +1769,7 @@ static void pci_configure_mps(struct pci_dev *dev)
 	}
 
 	pci_info(dev, "Max Payload Size set to %d (was %d, max %d)\n",
-		 p_mps, mps, 128 << dev->pcie_mpss);
+		 p_mps, mps, mpss);
 }
 
 static struct hpp_type0 pci_default_type0 = {

Re: [PATCH 0/2] Augment device matching its upstream Root Port's MPS logic

[Bjorn Helgaas]

On Mon, Aug 13, 2018 at 12:19:31PM -0600, Myron Stowe wrote:
> In commit 27d868b5e6cf ("PCI: Set MPS to match upstream bridge") Keith made
> sure every device's MPS setting matched its upstream bridge, making it more
> likely that a hot-added devices would work in a system with an optimized MPS
> configuration.
> 
> This series augments Keith's approach to include tuning down a Root Port's
> MPS setting in the case where a hot-added device is not capable of matching
> it (see: [1]).
> 
> Testing by Dongdong exposed a bug with the logic including Virtual Functions
> (VFs).  VFs should not be included so a pre-cursor patch, 1/2, was added to
> cover such.
> 
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=200527
> 
> Myron Stowe (2):
>       PCI: Skip MPS logic for Virtual Functions (VFs)
>       PCI: Match Root Port's MPS to endpoint's MPSS as necessary
> 
> 
>  drivers/pci/probe.c |   16 ++++++++++++++--
>  1 file changed, 14 insertions(+), 2 deletions(-)

Applied to pci/enumeration for v4.19, thanks!