[PATCH V4 0/1] Intel Sky Lake-E host root ports check.

[PATCH V4 0/1] Intel Sky Lake-E host root ports check.

[Shlomo Pongratz]

Changes in v4: 

Address Bjorn Helgaas and Jason Gunthorpe comments.
Replace the implementation of pci_is_root_port with a simple check
pci_pcie_type(root) != PCI_EXP_TYPE_ROOT_PORT and remove the added
IS_ROOT_PORT flag.
Update patch text.

Changes in v3: 

Use Jason Gunthorpe suggestion, that is add a flag 'IS_ROOT_PORT'
instead of 'port' and then just ignore the slot number entirely for root ports. 

Changes in v2: 

Change comment and description based on Logan Gunthorpe comments.


v1:

In commit 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to
the whitelist")
Andrew Maier added the Sky Lake-E additional devices
2031, 2032 and 2033 root ports to the already existing 2030 device.

The Intel devices 2030, 2031, 2032 and 2033 which are root ports A, B, C and D,
respectively and if all exist they will occupy slots 0 till 3 in that order.

The original code handled only the case where the devices in the whiltlist are 
host bridges and assumed that they will be found on slot 0.

This assumption doesn't hold for root ports so an explicit test was added to
cover this case.


Shlomo Pongratz (1):
  Intel Sky Lake-E host root ports check.

 drivers/pci/p2pdma.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

-- 
2.17.1

[PATCH V4 1/1] Intel Sky Lake-E host root ports check.

[Shlomo Pongratz]

In commit 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to
the whitelist")
Andrew Maier added the Sky Lake-E additional devices
2031, 2032 and 2033 root ports to the already existing 2030 device.

The Intel devices 2030, 2031, 2032 and 2033 which are root ports A, B, C and D,
respectively and if all exist they will occupy slots 0 till 3 in that order.

The original code handled only the case where the devices in the whiltlist are
host bridges and assumed that they will be found on slot 0.

This assumption doesn't hold for root ports so an explicit test was added to
cover this case.

Signed-off-by: Shlomo Pongratz <shlomop@pliops.com>
---
 drivers/pci/p2pdma.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
index 30b1df3c9d2f..107905dfa229 100644
--- a/drivers/pci/p2pdma.c
+++ b/drivers/pci/p2pdma.c
@@ -327,15 +327,18 @@ static const struct pci_p2pdma_whitelist_entry {
 
 /*
  * This lookup function tries to find the PCI device corresponding to a given
- * host bridge.
+ * host bridge or a root port.
  *
  * It assumes the host bridge device is the first PCI device in the
- * bus->devices list and that the devfn is 00.0. These assumptions should hold
- * for all the devices in the whitelist above.
+ * bus->devices list and that the devfn is 00.0. The first assumption should
+ * hold for all the devices in the whitelist above, however the second one
+ * doesn't always holds for root ports, for example Intel SkyLake-E devices
+ * 2030, 2031, 2032 and 2033 which are root ports (A, B, C and D respectively).
+ * So being a root port is checked explicitly.
  *
- * This function is equivalent to pci_get_slot(host->bus, 0), however it does
- * not take the pci_bus_sem lock seeing __host_bridge_whitelist() must not
- * sleep.
+ * This function is equivalent to pci_get_slot(host->bus, 0) (except for
+ * the root port test), however it does not take the pci_bus_sem lock seeing
+ * __host_bridge_whitelist() must not sleep.
  *
  * For this to be safe, the caller should hold a reference to a device on the
  * bridge, which should ensure the host_bridge device will not be freed
@@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host)
 
 	if (!root)
 		return NULL;
-	if (root->devfn != PCI_DEVFN(0, 0))
+
+	/* Is it a host bridge or a root port? */
+	if (root->devfn != PCI_DEVFN(0, 0) &&
+		pci_pcie_type(root) != PCI_EXP_TYPE_ROOT_PORT)
 		return NULL;
 
 	return root;
-- 
2.17.1

Re: [PATCH V4 1/1] Intel Sky Lake-E host root ports check.

[Logan Gunthorpe]

On 2022-03-30 08:08, Shlomo Pongratz wrote:
> In commit 7b94b53db34f ("PCI/P2PDMA: Add Intel Sky Lake-E Root Ports B, C, D to
> the whitelist")
> Andrew Maier added the Sky Lake-E additional devices
> 2031, 2032 and 2033 root ports to the already existing 2030 device.
> 
> The Intel devices 2030, 2031, 2032 and 2033 which are root ports A, B, C and D,
> respectively and if all exist they will occupy slots 0 till 3 in that order.
> 
> The original code handled only the case where the devices in the whiltlist are

s/whiltlist/whitelist/

> host bridges and assumed that they will be found on slot 0.
> 
> This assumption doesn't hold for root ports so an explicit test was added to
> cover this case.
> 
> Signed-off-by: Shlomo Pongratz <shlomop@pliops.com>

This patch looks good to me. Much cleaner and easier to understand than
the previous ones.

Reviewed-by: Logan Gunthorpe <logang@deltatee.com>

Thanks,

Logan

Re: [PATCH V4 1/1] Intel Sky Lake-E host root ports check.

[Jason Gunthorpe]

On Wed, Mar 30, 2022 at 05:08:23PM +0300, Shlomo Pongratz wrote:
> @@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host)
>  
>  	if (!root)
>  		return NULL;
> -	if (root->devfn != PCI_DEVFN(0, 0))
> +
> +	/* Is it a host bridge or a root port? */

This is a better comment:

/* host bridges must have a 0 devfn, but some of the entries in the
  whilelist are root ports that can have any devfn */

> +	if (root->devfn != PCI_DEVFN(0, 0) &&
> +		pci_pcie_type(root) != PCI_EXP_TYPE_ROOT_PORT)
>  		return NULL;

Indentation is wrong

Jason

Re: [PATCH V4 1/1] Intel Sky Lake-E host root ports check.

[Bjorn Helgaas]

On Wed, Mar 30, 2022 at 12:37:20PM -0300, Jason Gunthorpe wrote:
> On Wed, Mar 30, 2022 at 05:08:23PM +0300, Shlomo Pongratz wrote:
> > @@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host)
> >  
> >  	if (!root)
> >  		return NULL;
> > -	if (root->devfn != PCI_DEVFN(0, 0))
> > +
> > +	/* Is it a host bridge or a root port? */
> 
> This is a better comment:
> 
> /* host bridges must have a 0 devfn, but some of the entries in the
>   whilelist are root ports that can have any devfn */

Is this something in the spec or is it just common practice?  The PCIe
spec says very little about "host bridges" and I don't remember
anything about them having to be devfn 0 or even that they have to be
materialized as PCI devices.

> > +	if (root->devfn != PCI_DEVFN(0, 0) &&
> > +		pci_pcie_type(root) != PCI_EXP_TYPE_ROOT_PORT)
> >  		return NULL;
> 
> Indentation is wrong
> 
> Jason

Re: [PATCH V4 1/1] Intel Sky Lake-E host root ports check.

[Jason Gunthorpe]

On Wed, Mar 30, 2022 at 02:10:17PM -0500, Bjorn Helgaas wrote:
> On Wed, Mar 30, 2022 at 12:37:20PM -0300, Jason Gunthorpe wrote:
> > On Wed, Mar 30, 2022 at 05:08:23PM +0300, Shlomo Pongratz wrote:
> > > @@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host)
> > >  
> > >  	if (!root)
> > >  		return NULL;
> > > -	if (root->devfn != PCI_DEVFN(0, 0))
> > > +
> > > +	/* Is it a host bridge or a root port? */
> > 
> > This is a better comment:
> > 
> > /* host bridges must have a 0 devfn, but some of the entries in the
> >   whilelist are root ports that can have any devfn */
> 
> Is this something in the spec or is it just common practice?  The PCIe
> spec says very little about "host bridges" and I don't remember
> anything about them having to be devfn 0 or even that they have to be
> materialized as PCI devices.

I think we are relying on common practice here, but I don't know why
this check was added in the first place? Logan?

Jason

Re: [PATCH V4 1/1] Intel Sky Lake-E host root ports check.

[Logan Gunthorpe]

On 2022-03-30 13:11, Jason Gunthorpe wrote:
> On Wed, Mar 30, 2022 at 02:10:17PM -0500, Bjorn Helgaas wrote:
>> On Wed, Mar 30, 2022 at 12:37:20PM -0300, Jason Gunthorpe wrote:
>>> On Wed, Mar 30, 2022 at 05:08:23PM +0300, Shlomo Pongratz wrote:
>>>> @@ -350,7 +353,10 @@ static struct pci_dev *pci_host_bridge_dev(struct pci_host_bridge *host)
>>>>  
>>>>  	if (!root)
>>>>  		return NULL;
>>>> -	if (root->devfn != PCI_DEVFN(0, 0))
>>>> +
>>>> +	/* Is it a host bridge or a root port? */
>>>
>>> This is a better comment:
>>>
>>> /* host bridges must have a 0 devfn, but some of the entries in the
>>>   whilelist are root ports that can have any devfn */
>>
>> Is this something in the spec or is it just common practice?  The PCIe
>> spec says very little about "host bridges" and I don't remember
>> anything about them having to be devfn 0 or even that they have to be
>> materialized as PCI devices.
> 
> I think we are relying on common practice here, but I don't know why
> this check was added in the first place? Logan?

Yeah, this was more about common practice than spec. Seeing we only
support a subset of devices it was expected that those added to our
support list would meet this criteria.

The main purpose here is to just get identifying information about the
host bridge. I wouldn't say that the host bridge is dev-fn 0 in any
sense of the word. What a host bridge is vs a root port is also a bit
murky . I'm really not sure what the tree looks like on this SkyLake
system, but on my test system, the two host bridges are also root ports.

00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)
	Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00

The point was really just finding the VID/PID of one of the PCI devices
belonging to the bridge for identifying whether it was safe to use.
Originally this function was implemented as simply

    pci_get_slot(host->bus, PCI_DEVFN(0, 0));

That was replaced with the current function solely to avoid needing to
take pci_bus_sem in this path. The check for dev-fn 0 was kept to ensure
it matched the old method.

Checking simply for PCI_EXP_TYPE_ROOT_PORT instead of a zero devfn is
probably a good idea, assuming it works for all existing systems. I'd
expect it would be set for all the devices currently allowed.

Logan

Re: [PATCH V4 1/1] Intel Sky Lake-E host root ports check.

[Jason Gunthorpe]

On Wed, Mar 30, 2022 at 01:36:25PM -0600, Logan Gunthorpe wrote:

> Checking simply for PCI_EXP_TYPE_ROOT_PORT instead of a zero devfn is
> probably a good idea, assuming it works for all existing systems. I'd
> expect it would be set for all the devices currently allowed.

I think if we find a PCI ID in the white list as we go up the PCI
hierarchy we should just declare success?

Does it matter if it is the top of the tree or if it is a root port or
host bridge? The IDs in this list only exist as part of SOCs, so
seeing them at all should confirm the whole SOC..

Jason

Re: [PATCH V4 1/1] Intel Sky Lake-E host root ports check.

[Logan Gunthorpe]

On 2022-03-30 13:38, Jason Gunthorpe wrote:
> On Wed, Mar 30, 2022 at 01:36:25PM -0600, Logan Gunthorpe wrote:
> 
>> Checking simply for PCI_EXP_TYPE_ROOT_PORT instead of a zero devfn is
>> probably a good idea, assuming it works for all existing systems. I'd
>> expect it would be set for all the devices currently allowed.
> 
> I think if we find a PCI ID in the white list as we go up the PCI
> hierarchy we should just declare success?

Well, in older hardware we also have to ensure that the devices are on
the same host bridge. (See REQ_SAME_HOST_BRIDGE). So it's a bit more
complicated than that...
> Does it matter if it is the top of the tree or if it is a root port or
> host bridge? The IDs in this list only exist as part of SOCs, so
> seeing them at all should confirm the whole SOC..

Not sure if that will always hold with multi-socket systems. The code
currently uses pci_find_host_bridge() for each device, then uses
pci_host_bridge_dev() simply to find identifying information about the
host bridge. It doesn't really matter where in the tree things are, but
it would be add for the host bridge not to be at the top, I would have
thought.

Logan