[RFC] ARM64, PCIe, annd ACPI

[RFC] ARM64, PCIe, annd ACPI

[Ray Jui]

Hi Lorenzo/Bjorn,

I have a question on PCIe controller APCI support on ARM64 based systems.

If my understanding of the implementation of "pci_acpi_scan_root" under 
"arch/arm64/kernel/pci.c" is correct, it appears 
"pci_acpi_setup_ecam_mapping" is called within "pci_acpi_scan_root".

Does that mean for a PCIe host controller on ARM64 to support ACPI, it 
needs to support ECAM and MMIO based access to the configuration space 
registers?

If the above statement is true, does it imply that any PCIe controller 
on ARM64 that does not support MMIO based access to the config space 
register cannot have ACPI support?

I might be missing something here so I'd like to send out this email and 
get your opinion on this. Your feedback is highly appreciated.

Regards,

Ray

Re: [RFC] ARM64, PCIe, annd ACPI

[Bjorn Helgaas]

On Tue, Aug 07, 2018 at 05:10:15PM -0700, Ray Jui wrote:
> Hi Lorenzo/Bjorn,
> 
> I have a question on PCIe controller APCI support on ARM64 based systems.
> 
> If my understanding of the implementation of "pci_acpi_scan_root" under
> "arch/arm64/kernel/pci.c" is correct, it appears
> "pci_acpi_setup_ecam_mapping" is called within "pci_acpi_scan_root".
> 
> Does that mean for a PCIe host controller on ARM64 to support ACPI, it needs
> to support ECAM and MMIO based access to the configuration space registers?
 
Yes.  In my opinion, a PCIe host controller is generally required to
support ECAM even if the platform doesn't support ACPI.  I base this
on PCIe r4.0, sec 7.2.2:

  For systems that are PC-compatible, or that do not implement a
  processor-architecture-specific firmware interface standard that
  allows access to the Configuration Space, the ECAM is required as
  defined in this section.

> If the above statement is true, does it imply that any PCIe controller on
> ARM64 that does not support MMIO based access to the config space register
> cannot have ACPI support?

We would treat that as a non-compliant platform that requires special
quirks to work around the problem.  That does mean we have to add
platform-specific quirks, which of course means we can't run an old
kernel on a new platform that lacks standard ECAM.  There are several
existing systems like this.

The notes and examples here should be a place to start:

  drivers/pci/controller/Makefile
  drivers/pci/controller/dwc/Makefile
  drivers/acpi/pci_mcfg.c

Bjorn

Re: [RFC] ARM64, PCIe, annd ACPI

[Lorenzo Pieralisi]

On Tue, Aug 07, 2018 at 05:10:15PM -0700, Ray Jui wrote:
> Hi Lorenzo/Bjorn,
> 
> I have a question on PCIe controller APCI support on ARM64 based systems.
> 
> If my understanding of the implementation of "pci_acpi_scan_root" under
> "arch/arm64/kernel/pci.c" is correct, it appears
> "pci_acpi_setup_ecam_mapping" is called within "pci_acpi_scan_root".
> 
> Does that mean for a PCIe host controller on ARM64 to support ACPI, it needs
> to support ECAM and MMIO based access to the configuration space registers?
> 
> If the above statement is true, does it imply that any PCIe controller on
> ARM64 that does not support MMIO based access to the config space register
> cannot have ACPI support?

Yes and it is not just MMIO, config space must be ECAM compliant as described
in the PCI firmware specification and enforced here:

http://infocenter.arm.com/help/topic/com.arm.doc.den0029b/Server_Base_System_Architecture_v5_0_ARM_DEN_0029B.pdf

In short: what's in the mainline (+ adequate PCI controller firmware
configuration) must be sufficient to bootstrap your PCI subsystem in
ACPI, it is a very simple litmus test and we won't add anything that
deviates from that to the mainline kernel ARM64 ACPI PCI code support.

Thanks,
Lorenzo

Re: [RFC] ARM64, PCIe, annd ACPI

[Ray Jui]

On 8/7/2018 7:10 PM, Bjorn Helgaas wrote:
> On Tue, Aug 07, 2018 at 05:10:15PM -0700, Ray Jui wrote:
>> Hi Lorenzo/Bjorn,
>>
>> I have a question on PCIe controller APCI support on ARM64 based systems.
>>
>> If my understanding of the implementation of "pci_acpi_scan_root" under
>> "arch/arm64/kernel/pci.c" is correct, it appears
>> "pci_acpi_setup_ecam_mapping" is called within "pci_acpi_scan_root".
>>
>> Does that mean for a PCIe host controller on ARM64 to support ACPI, it needs
>> to support ECAM and MMIO based access to the configuration space registers?
>   
> Yes.  In my opinion, a PCIe host controller is generally required to
> support ECAM even if the platform doesn't support ACPI.  I base this
> on PCIe r4.0, sec 7.2.2:
> 
>    For systems that are PC-compatible, or that do not implement a
>    processor-architecture-specific firmware interface standard that
>    allows access to the Configuration Space, the ECAM is required as
>    defined in this section.
> 
>> If the above statement is true, does it imply that any PCIe controller on
>> ARM64 that does not support MMIO based access to the config space register
>> cannot have ACPI support?
> 
> We would treat that as a non-compliant platform that requires special
> quirks to work around the problem.  That does mean we have to add
> platform-specific quirks, which of course means we can't run an old
> kernel on a new platform that lacks standard ECAM.  There are several
> existing systems like this.
> 
> The notes and examples here should be a place to start:
> 
>    drivers/pci/controller/Makefile
>    drivers/pci/controller/dwc/Makefile
>    drivers/acpi/pci_mcfg.c
> 
> Bjorn
> 

Thanks for the pointers, Bjorn! Very helpful!

Ray

Re: [RFC] ARM64, PCIe, annd ACPI

[Ray Jui]

On 8/8/2018 2:38 AM, Lorenzo Pieralisi wrote:
> On Tue, Aug 07, 2018 at 05:10:15PM -0700, Ray Jui wrote:
>> Hi Lorenzo/Bjorn,
>>
>> I have a question on PCIe controller APCI support on ARM64 based systems.
>>
>> If my understanding of the implementation of "pci_acpi_scan_root" under
>> "arch/arm64/kernel/pci.c" is correct, it appears
>> "pci_acpi_setup_ecam_mapping" is called within "pci_acpi_scan_root".
>>
>> Does that mean for a PCIe host controller on ARM64 to support ACPI, it needs
>> to support ECAM and MMIO based access to the configuration space registers?
>>
>> If the above statement is true, does it imply that any PCIe controller on
>> ARM64 that does not support MMIO based access to the config space register
>> cannot have ACPI support?
> 
> Yes and it is not just MMIO, config space must be ECAM compliant as described
> in the PCI firmware specification and enforced here:
> 
> http://infocenter.arm.com/help/topic/com.arm.doc.den0029b/Server_Base_System_Architecture_v5_0_ARM_DEN_0029B.pdf
> 

Okay, got it, the SBSA spec makes it very clear that the PCIe controller 
needs to support ECAM.

> In short: what's in the mainline (+ adequate PCI controller firmware
> configuration) must be sufficient to bootstrap your PCI subsystem in
> ACPI, it is a very simple litmus test and we won't add anything that
> deviates from that to the mainline kernel ARM64 ACPI PCI code support.
> 

It sounds like I'll have to look into ways to work around this at the 
driver level as Bjorn pointed out with some other examples.

> Thanks,
> Lorenzo
> 

Thanks!

Ray

Re: [RFC] ARM64, PCIe, annd ACPI

[Lorenzo Pieralisi]

On Wed, Aug 08, 2018 at 11:49:46AM -0700, Ray Jui wrote:
> 
> 
> On 8/8/2018 2:38 AM, Lorenzo Pieralisi wrote:
> >On Tue, Aug 07, 2018 at 05:10:15PM -0700, Ray Jui wrote:
> >>Hi Lorenzo/Bjorn,
> >>
> >>I have a question on PCIe controller APCI support on ARM64 based systems.
> >>
> >>If my understanding of the implementation of "pci_acpi_scan_root" under
> >>"arch/arm64/kernel/pci.c" is correct, it appears
> >>"pci_acpi_setup_ecam_mapping" is called within "pci_acpi_scan_root".
> >>
> >>Does that mean for a PCIe host controller on ARM64 to support ACPI, it needs
> >>to support ECAM and MMIO based access to the configuration space registers?
> >>
> >>If the above statement is true, does it imply that any PCIe controller on
> >>ARM64 that does not support MMIO based access to the config space register
> >>cannot have ACPI support?
> >
> >Yes and it is not just MMIO, config space must be ECAM compliant as described
> >in the PCI firmware specification and enforced here:
> >
> >http://infocenter.arm.com/help/topic/com.arm.doc.den0029b/Server_Base_System_Architecture_v5_0_ARM_DEN_0029B.pdf
> >
> 
> Okay, got it, the SBSA spec makes it very clear that the PCIe controller
> needs to support ECAM.
> 
> >In short: what's in the mainline (+ adequate PCI controller firmware
> >configuration) must be sufficient to bootstrap your PCI subsystem in
> >ACPI, it is a very simple litmus test and we won't add anything that
> >deviates from that to the mainline kernel ARM64 ACPI PCI code support.
> >
> 
> It sounds like I'll have to look into ways to work around this at the driver
> level as Bjorn pointed out with some other examples.

As I said ARM64 ACPI PCI kernel support is in the mainline today and I
am not willing to merge any more MCFG quirks at driver level - the
information is out there, it is public and must be followed, either
designers do that or no ACPI PCI support on your platform, sorry.

Lorenzo

Re: [RFC] ARM64, PCIe, annd ACPI

[Ray Jui]

Hi Lorenzo,

On 8/9/2018 1:59 AM, Lorenzo Pieralisi wrote:
> On Wed, Aug 08, 2018 at 11:49:46AM -0700, Ray Jui wrote:
>>
>>
>> On 8/8/2018 2:38 AM, Lorenzo Pieralisi wrote:
>>> On Tue, Aug 07, 2018 at 05:10:15PM -0700, Ray Jui wrote:
>>>> Hi Lorenzo/Bjorn,
>>>>
>>>> I have a question on PCIe controller APCI support on ARM64 based systems.
>>>>
>>>> If my understanding of the implementation of "pci_acpi_scan_root" under
>>>> "arch/arm64/kernel/pci.c" is correct, it appears
>>>> "pci_acpi_setup_ecam_mapping" is called within "pci_acpi_scan_root".
>>>>
>>>> Does that mean for a PCIe host controller on ARM64 to support ACPI, it needs
>>>> to support ECAM and MMIO based access to the configuration space registers?
>>>>
>>>> If the above statement is true, does it imply that any PCIe controller on
>>>> ARM64 that does not support MMIO based access to the config space register
>>>> cannot have ACPI support?
>>>
>>> Yes and it is not just MMIO, config space must be ECAM compliant as described
>>> in the PCI firmware specification and enforced here:
>>>
>>> http://infocenter.arm.com/help/topic/com.arm.doc.den0029b/Server_Base_System_Architecture_v5_0_ARM_DEN_0029B.pdf
>>>
>>
>> Okay, got it, the SBSA spec makes it very clear that the PCIe controller
>> needs to support ECAM.
>>
>>> In short: what's in the mainline (+ adequate PCI controller firmware
>>> configuration) must be sufficient to bootstrap your PCI subsystem in
>>> ACPI, it is a very simple litmus test and we won't add anything that
>>> deviates from that to the mainline kernel ARM64 ACPI PCI code support.
>>>
>>
>> It sounds like I'll have to look into ways to work around this at the driver
>> level as Bjorn pointed out with some other examples.
> 
> As I said ARM64 ACPI PCI kernel support is in the mainline today and I
> am not willing to merge any more MCFG quirks at driver level - the
> information is out there, it is public and must be followed, either
> designers do that or no ACPI PCI support on your platform, sorry.
> 
> Lorenzo
> 

I just want to make sure I interpret this properly. This means if the 
PCIe controller hardware does not support ECAM, there's pretty much no 
way around it with SW based workaround going forward with mainline kernel?

What about for the PCIe controller that was designed a few years back?

Thanks,

Ray

Re: [RFC] ARM64, PCIe, annd ACPI

[Lorenzo Pieralisi]

On Thu, Aug 09, 2018 at 07:57:52AM -0700, Ray Jui wrote:

[...]

> >As I said ARM64 ACPI PCI kernel support is in the mainline today and I
> >am not willing to merge any more MCFG quirks at driver level - the
> >information is out there, it is public and must be followed, either
> >designers do that or no ACPI PCI support on your platform, sorry.
> >
> >Lorenzo
> >
> 
> I just want to make sure I interpret this properly. This means if
> the PCIe controller hardware does not support ECAM, there's pretty
> much no way around it with SW based workaround going forward with
> mainline kernel?

Yes.

> What about for the PCIe controller that was designed a few years back?

It will bootstrap with DT :)

Thanks,
Lorenzo

Re: [RFC] ARM64, PCIe, annd ACPI

[Ray Jui]

Hi Lorenzo,

On 8/9/2018 8:41 AM, Lorenzo Pieralisi wrote:
> On Thu, Aug 09, 2018 at 07:57:52AM -0700, Ray Jui wrote:
> 
> [...]
> 
>>> As I said ARM64 ACPI PCI kernel support is in the mainline today and I
>>> am not willing to merge any more MCFG quirks at driver level - the
>>> information is out there, it is public and must be followed, either
>>> designers do that or no ACPI PCI support on your platform, sorry.
>>>
>>> Lorenzo
>>>
>>
>> I just want to make sure I interpret this properly. This means if
>> the PCIe controller hardware does not support ECAM, there's pretty
>> much no way around it with SW based workaround going forward with
>> mainline kernel?
> 
> Yes.
> 
>> What about for the PCIe controller that was designed a few years back?
> 
> It will bootstrap with DT :)

Okay, one more quick question, what about for distributions that will 
eventually disable DT and force ACPI?

> 
> Thanks,
> Lorenzo
> 

Thanks,

Ray

Re: [RFC] ARM64, PCIe, annd ACPI

[Lorenzo Pieralisi]

On Thu, Aug 09, 2018 at 08:44:34AM -0700, Ray Jui wrote:
> Hi Lorenzo,
> 
> On 8/9/2018 8:41 AM, Lorenzo Pieralisi wrote:
> >On Thu, Aug 09, 2018 at 07:57:52AM -0700, Ray Jui wrote:
> >
> >[...]
> >
> >>>As I said ARM64 ACPI PCI kernel support is in the mainline today and I
> >>>am not willing to merge any more MCFG quirks at driver level - the
> >>>information is out there, it is public and must be followed, either
> >>>designers do that or no ACPI PCI support on your platform, sorry.
> >>>
> >>>Lorenzo
> >>>
> >>
> >>I just want to make sure I interpret this properly. This means if
> >>the PCIe controller hardware does not support ECAM, there's pretty
> >>much no way around it with SW based workaround going forward with
> >>mainline kernel?
> >
> >Yes.
> >
> >>What about for the PCIe controller that was designed a few years back?
> >
> >It will bootstrap with DT :)
> 
> Okay, one more quick question, what about for distributions that
> will eventually disable DT and force ACPI?

ACPI PCI support revolves around a FW model that requires, among other
things, ECAM and its firmware description, MCFG table, to work.

If the host controller is not ECAM compliant it is not PCI firmware
compliant, so it is not ACPI compliant, you can't run ACPI only
distributions on it.

It is a model that has been run for decades in basically all
ACPI based systems out there and there is not a reason to
deviate from it, I understand the frustration but that's the
way it is, sorry.

Thanks,
Lorenzo

Re: [RFC] ARM64, PCIe, annd ACPI

[Ray Jui]

Hi Lorenzo,

On 8/9/2018 9:00 AM, Lorenzo Pieralisi wrote:
> On Thu, Aug 09, 2018 at 08:44:34AM -0700, Ray Jui wrote:
>> Hi Lorenzo,
>>
>> On 8/9/2018 8:41 AM, Lorenzo Pieralisi wrote:
>>> On Thu, Aug 09, 2018 at 07:57:52AM -0700, Ray Jui wrote:
>>>
>>> [...]
>>>
>>>>> As I said ARM64 ACPI PCI kernel support is in the mainline today and I
>>>>> am not willing to merge any more MCFG quirks at driver level - the
>>>>> information is out there, it is public and must be followed, either
>>>>> designers do that or no ACPI PCI support on your platform, sorry.
>>>>>
>>>>> Lorenzo
>>>>>
>>>>
>>>> I just want to make sure I interpret this properly. This means if
>>>> the PCIe controller hardware does not support ECAM, there's pretty
>>>> much no way around it with SW based workaround going forward with
>>>> mainline kernel?
>>>
>>> Yes.
>>>
>>>> What about for the PCIe controller that was designed a few years back?
>>>
>>> It will bootstrap with DT :)
>>
>> Okay, one more quick question, what about for distributions that
>> will eventually disable DT and force ACPI?
> 
> ACPI PCI support revolves around a FW model that requires, among other
> things, ECAM and its firmware description, MCFG table, to work.
> 
> If the host controller is not ECAM compliant it is not PCI firmware
> compliant, so it is not ACPI compliant, you can't run ACPI only
> distributions on it.
> 
> It is a model that has been run for decades in basically all
> ACPI based systems out there and there is not a reason to
> deviate from it, I understand the frustration but that's the
> way it is, sorry.
> 
> Thanks,
> Lorenzo
> 

Okay that draws the line. Thanks for confirming.

Ray