* [RFC] PCI ACS Flags Clarification @ 2017-04-04 18:47 Sinan Kaya 2017-04-04 19:39 ` Alex Williamson 0 siblings, 1 reply; 5+ messages in thread From: Sinan Kaya @ 2017-04-04 18:47 UTC (permalink / raw) To: Linux PCI, Alex Williamson, iommu I'm looking for a guideline on how Access Control Services (ACS) need to be implemented in HW for cases where peer to peer is and isn't supported. I see conflicting information in the code. iommu.c calls pci_acs_enabled() from pci_device_group() to determine if ACS path is enabled for the following flags. This paragraph lays out the requirements. /* * To consider a PCI device isolated, we require ACS to support Source * Validation, Request Redirection, Completer Redirection, and Upstream * Forwarding. This effectively means that devices cannot spoof their * requester ID, requests and completions cannot be redirected, and all * transactions are forwarded upstream, even as it passes through a * bridge where the target device is downstream. */ #define REQ_ACS_FLAGS (PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF) I also see the following comment in pci_acs_flags_enabled() which gets called from pci_acs_enabled() /* * Except for egress control, capabilities are either required * or only required if controllable. Features missing from the * capability field can therefore be assumed as hard-wired enabled. */ This comment in pci_acs_flags_enabled() is confusing. A capability that reads 0 usually means it is not enabled. This code is assuming enabled. Are we trying to say that we'll use the capability bits as a mask while verifying the requested ACS flags? For systems that don't support P2P, the code will check PCI_ACS_SV only. So, if I summarize this correctly; The requirement is to have an 1. ACS capability with PCI_ACS_SV if p2p is not supported 2. ACS capability with PCI_ACS_SV|PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF if p2p is supported. Did I get this right? -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [RFC] PCI ACS Flags Clarification 2017-04-04 18:47 [RFC] PCI ACS Flags Clarification Sinan Kaya @ 2017-04-04 19:39 ` Alex Williamson 2017-04-05 13:51 ` Sinan Kaya 0 siblings, 1 reply; 5+ messages in thread From: Alex Williamson @ 2017-04-04 19:39 UTC (permalink / raw) To: Sinan Kaya; +Cc: Linux PCI, iommu On Tue, 4 Apr 2017 14:47:58 -0400 Sinan Kaya <okaya@codeaurora.org> wrote: > I'm looking for a guideline on how Access Control Services (ACS) need to be > implemented in HW for cases where peer to peer is and isn't supported. > > I see conflicting information in the code. > > iommu.c calls pci_acs_enabled() from pci_device_group() to determine if > ACS path is enabled for the following flags. > > This paragraph lays out the requirements. > > /* > * To consider a PCI device isolated, we require ACS to support Source > * Validation, Request Redirection, Completer Redirection, and Upstream > * Forwarding. This effectively means that devices cannot spoof their > * requester ID, requests and completions cannot be redirected, and all > * transactions are forwarded upstream, even as it passes through a > * bridge where the target device is downstream. > */ > #define REQ_ACS_FLAGS (PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF) > > I also see the following comment in pci_acs_flags_enabled() which gets > called from pci_acs_enabled() > > /* > * Except for egress control, capabilities are either required > * or only required if controllable. Features missing from the > * capability field can therefore be assumed as hard-wired enabled. > */ > > This comment in pci_acs_flags_enabled() is confusing. A capability that reads > 0 usually means it is not enabled. This code is assuming enabled. > Are we trying to say that we'll use the capability bits as a mask while > verifying the requested ACS flags? See the language in the PCIe spec: 6.12.1.1. ACS Downstream Ports - ACS Source Validation: must be implemented. - ACS Translation Blocking: must be implemented. - ACS P2P Request Redirect: must be implemented by Root Ports that support peer-to-peer traffic with other Root Ports; must be implemented by Switch Downstream Ports. - ACS P2P Completion Redirect: must be implemented by Root Ports that implement ACS P2P Request Redirect; must be implemented by Switch Downstream Ports. - ACS Upstream Forwarding: must be implemented by Root Ports if the RC supports Redirected Request Validation; must be implemented by Switch Downstream Ports. - ACS P2P Egress Control: implementation is optional. Thus for a downstream port, SV and TB must be implemented, period. RR, CR, and UF must be implemented by switch downstream ports, period. On a root port, if RR is not implemented, we can assume it does not support P2P with other root ports. If the root port supports RR, it must also support CR and UF. The kernel code takes a bit of a lax approach to this with the items you've identified above, we assume that if a capability isn't present then the device is compliant to the spec and not trying to circumvent it. It's a generalization trying to also encompass other component types. > For systems that don't support P2P, the code will check PCI_ACS_SV > only. So, if I summarize this correctly; For a downstream port or downstream switch port, this would not be spec compliant. TB would need to be implemented for either (not that the kernel cares for isolation) and all except EC would need to be implemented on a downstream switch port. > The requirement is to have an > 1. ACS capability with PCI_ACS_SV if p2p is not supported > 2. ACS capability with PCI_ACS_SV|PCI_ACS_RR | PCI_ACS_CR | > PCI_ACS_UF if p2p is supported. > > Did I get this right? I'd suggest following the spec, not the code, that way you always have a case for how you interpret the spec and the behavior of your hardware. The code is an attempt to validate the device against the spec, but more thorough implementations may follow. REQ_ACS_FLAGS defines the set of ACS capabilities we think are relevant to device isolation. In particular, UF seems like a key feature and our current test for it may not be fully correct. Note how RR and CR only specify p2p with other root ports while UF requires transaction towards to the RC. Section 6.12.2 further defines Redirected Request Validation as a feature within the context of RR and CR. So rather than look at the code, discuss section 6.12 with the hardware engineers and understand how it behaves relative to each device and transaction type. Implement the capabilities that best match. Attempting a minimal implementation based on the current software interpretation may bite later, for instance if we re-interpret how UF works. Thanks, Alex ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [RFC] PCI ACS Flags Clarification @ 2017-04-05 13:51 ` Sinan Kaya 0 siblings, 0 replies; 5+ messages in thread From: Sinan Kaya @ 2017-04-05 13:51 UTC (permalink / raw) To: Alex Williamson; +Cc: Linux PCI, iommu Hi Alex, Thank you very much for the detailed explanation. On 4/4/2017 3:39 PM, Alex Williamson wrote: > On Tue, 4 Apr 2017 14:47:58 -0400 > Sinan Kaya <okaya@codeaurora.org> wrote: > [cut] >> The requirement is to have an >> 1. ACS capability with PCI_ACS_SV if p2p is not supported >> 2. ACS capability with PCI_ACS_SV|PCI_ACS_RR | PCI_ACS_CR | >> PCI_ACS_UF if p2p is supported. >> >> Did I get this right? > I'm looking for sanity check on OS requirements. According to the PCIE spec, ACS itself is optional. However, Linux requires ACS capability. I want to make sure that we are satisfying the Linux requirements here. I also agree that spec should be the guide. Maybe, a combination of spec+linux is the right answer. > I'd suggest following the spec, not the code, that way you always have a > case for how you interpret the spec and the behavior of your hardware. > The code is an attempt to validate the device against the spec, but more > thorough implementations may follow. REQ_ACS_FLAGS defines the set of > ACS capabilities we think are relevant to device isolation. In > particular, UF seems like a key feature and our current test for it may > not be fully correct. Note how RR and CR only specify p2p with other > root ports while UF requires transaction towards to the RC. Section > 6.12.2 further defines Redirected Request Validation as a feature > within the context of RR and CR. So rather than look at the code, > discuss section 6.12 with the hardware engineers and understand how it > behaves relative to each device and transaction type. Implement the > capabilities that best match. Attempting a minimal implementation > based on the current software interpretation may bite later, for > instance if we re-interpret how UF works. Thanks, I read your reply multiple times. Here is what I got from it. The summary below is for the root ports only. - P2P on root ports is optional. - If P2P is there source validation, translation blocking, upstream forwarding, p2p request redirection and p2p completion redirection ACS capabilities need to be there for spec+linux compliance. - If P2P is not there source validation, translation blocking and upstream forwarding needs to be there for spec+linux compliance. - The code is relaxed to allow any combination of these today based on the ACS capability but it can change tomorrow. Sinan -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [RFC] PCI ACS Flags Clarification @ 2017-04-05 13:51 ` Sinan Kaya 0 siblings, 0 replies; 5+ messages in thread From: Sinan Kaya @ 2017-04-05 13:51 UTC (permalink / raw) To: Alex Williamson Cc: Linux PCI, iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA Hi Alex, Thank you very much for the detailed explanation. On 4/4/2017 3:39 PM, Alex Williamson wrote: > On Tue, 4 Apr 2017 14:47:58 -0400 > Sinan Kaya <okaya-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote: > [cut] >> The requirement is to have an >> 1. ACS capability with PCI_ACS_SV if p2p is not supported >> 2. ACS capability with PCI_ACS_SV|PCI_ACS_RR | PCI_ACS_CR | >> PCI_ACS_UF if p2p is supported. >> >> Did I get this right? > I'm looking for sanity check on OS requirements. According to the PCIE spec, ACS itself is optional. However, Linux requires ACS capability. I want to make sure that we are satisfying the Linux requirements here. I also agree that spec should be the guide. Maybe, a combination of spec+linux is the right answer. > I'd suggest following the spec, not the code, that way you always have a > case for how you interpret the spec and the behavior of your hardware. > The code is an attempt to validate the device against the spec, but more > thorough implementations may follow. REQ_ACS_FLAGS defines the set of > ACS capabilities we think are relevant to device isolation. In > particular, UF seems like a key feature and our current test for it may > not be fully correct. Note how RR and CR only specify p2p with other > root ports while UF requires transaction towards to the RC. Section > 6.12.2 further defines Redirected Request Validation as a feature > within the context of RR and CR. So rather than look at the code, > discuss section 6.12 with the hardware engineers and understand how it > behaves relative to each device and transaction type. Implement the > capabilities that best match. Attempting a minimal implementation > based on the current software interpretation may bite later, for > instance if we re-interpret how UF works. Thanks, I read your reply multiple times. Here is what I got from it. The summary below is for the root ports only. - P2P on root ports is optional. - If P2P is there source validation, translation blocking, upstream forwarding, p2p request redirection and p2p completion redirection ACS capabilities need to be there for spec+linux compliance. - If P2P is not there source validation, translation blocking and upstream forwarding needs to be there for spec+linux compliance. - The code is relaxed to allow any combination of these today based on the ACS capability but it can change tomorrow. Sinan -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [RFC] PCI ACS Flags Clarification 2017-04-05 13:51 ` Sinan Kaya (?) @ 2017-04-05 18:57 ` Alex Williamson -1 siblings, 0 replies; 5+ messages in thread From: Alex Williamson @ 2017-04-05 18:57 UTC (permalink / raw) To: Sinan Kaya; +Cc: Linux PCI, iommu On Wed, 5 Apr 2017 09:51:06 -0400 Sinan Kaya <okaya@codeaurora.org> wrote: > Hi Alex, > > Thank you very much for the detailed explanation. > > On 4/4/2017 3:39 PM, Alex Williamson wrote: > > On Tue, 4 Apr 2017 14:47:58 -0400 > > Sinan Kaya <okaya@codeaurora.org> wrote: > > > [cut] > > >> The requirement is to have an > >> 1. ACS capability with PCI_ACS_SV if p2p is not supported > >> 2. ACS capability with PCI_ACS_SV|PCI_ACS_RR | PCI_ACS_CR | > >> PCI_ACS_UF if p2p is supported. > >> > >> Did I get this right? > > > > I'm looking for sanity check on OS requirements. According to the PCIE spec, > ACS itself is optional. However, Linux requires ACS capability. I want to > make sure that we are satisfying the Linux requirements here. This is like saying Linux requires ACPI, it's simply not true. ACS is a PCIe spec defined feature which gives us a standard mechanism of determining if a device is DMA isolated within a PCIe topology. For those that do not implement the standard, we have numerous quirks implementing platform specific mechanism to provide the equivalent behavior. Even this device isolation is not required by Linux, except for the use case of userspace drivers, where we make the incredibly reasonable requirement (IMO) that we cannot mix devices which are not isolated from one another between kernel and user or one user and another. > I also agree that spec should be the guide. Maybe, a combination of spec+linux > is the right answer. > > > I'd suggest following the spec, not the code, that way you always have a > > case for how you interpret the spec and the behavior of your hardware. > > The code is an attempt to validate the device against the spec, but more > > thorough implementations may follow. REQ_ACS_FLAGS defines the set of > > ACS capabilities we think are relevant to device isolation. In > > particular, UF seems like a key feature and our current test for it may > > not be fully correct. Note how RR and CR only specify p2p with other > > root ports while UF requires transaction towards to the RC. Section > > 6.12.2 further defines Redirected Request Validation as a feature > > within the context of RR and CR. So rather than look at the code, > > discuss section 6.12 with the hardware engineers and understand how it > > behaves relative to each device and transaction type. Implement the > > capabilities that best match. Attempting a minimal implementation > > based on the current software interpretation may bite later, for > > instance if we re-interpret how UF works. Thanks, > > I read your reply multiple times. Here is what I got from it. > > The summary below is for the root ports only. > > - P2P on root ports is optional. And unless told otherwise (by ACS or ACS-equivalent quirks), Linux assumes p2p is possible. > - If P2P is there source validation, translation blocking, upstream forwarding, > p2p request redirection and p2p completion redirection ACS capabilities need to be > there for spec+linux compliance. Again, this is not a Linux requirement. If you want Linux to recognize the isolation of devices downstream of the root port using standard mechanism, this would be the way to do so. > - If P2P is not there source validation, translation blocking and upstream forwarding > needs to be there for spec+linux compliance. This is interpretation of a rather confusing section of the spec where we currently take a fairly liberal approach. The spec says RR must be implemented by RPs that support p2p traffic _with_other_root_ports_. That doesn't necessarily say anything about p2p "reflected" back downstream, however enabling RR clearly states that p2p request must be redirected upstream. Thus supporting RR is clearly the safer choice here for long term compliance if your device uses the RR behavior already. Supporting CR also falls out of that since RR necessitates CR. UF also references RR and CR behavior when enabled, so as per the footnote in 6.12.1.1, it's not entirely clear that we can assume what happens if a RP supports UF but not RR/CR. The safe bet is to implement each capability that matches your hardware behavior. > - The code is relaxed to allow any combination of these today based on the ACS > capability but it can change tomorrow. Yes, we somewhat assume that if a device supports ACS at all, it's probably doing the "right thing". I don't know that the current behavior of assuming p2p is not possible when the capability is not present is actually used by any existing hardware. It seems questionable on a strict reading of the spec. Thanks, Alex ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-04-05 18:57 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-04-04 18:47 [RFC] PCI ACS Flags Clarification Sinan Kaya 2017-04-04 19:39 ` Alex Williamson 2017-04-05 13:51 ` Sinan Kaya 2017-04-05 13:51 ` Sinan Kaya 2017-04-05 18:57 ` Alex Williamson
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.