Pointers for writing a good PCIe driver

Pointers for writing a good PCIe driver

[Mason]

Hello,

My platform ( arch/arm/mach-tango ) provides a PCIe controller from PLDA.

I do have access to a driver that works for a few PCIe boards, but it's
an ancient out-of-tree driver (targeting v3.4). Also, I'm not sure it
follows the best-practice guidelines (e.g. it hard-codes a few
work-arounds directly in drivers/pci/probe.c)

If I understand correctly, while PCIe is a standard, it is not codified
as much as USB (XHCI) or SATA (AHCI), which means the register layout
(and possibly other things) are left up to the integrator? Which means
there cannot be some kind of "generic" driver that works for a random
PCIe controller?

I guess my main question is: do you have pointers on getting started
writing a PCIe driver good enough for upstream?

Is Documentation/PCI/PCIEBUS-HOWTO.txt still a relevant resource?

Are there good resources to get up to speed on the PCIe terminology?

Regards.

Re: Pointers for writing a good PCIe driver

[Mason]

On 06/02/2017 16:54, Mason wrote:

> My platform ( arch/arm/mach-tango ) provides a PCIe controller from PLDA.
> 
> I do have access to a driver that works for a few PCIe boards, but it's
> an ancient out-of-tree driver (targeting v3.4). Also, I'm not sure it
> follows the best-practice guidelines (e.g. it hard-codes a few
> work-arounds directly in drivers/pci/probe.c)
> 
> If I understand correctly, while PCIe is a standard, it is not codified
> as much as USB (XHCI) or SATA (AHCI), which means the register layout
> (and possibly other things) are left up to the integrator? Which means
> there cannot be some kind of "generic" driver that works for a random
> PCIe controller?
> 
> I guess my main question is: do you have pointers on getting started
> writing a PCIe driver good enough for upstream?
> 
> Is Documentation/PCI/PCIEBUS-HOWTO.txt still a relevant resource?
> 
> Are there good resources to get up to speed on the PCIe terminology?

I'm also wondering: in many Linux subsystems, there are often
several drivers of different quality; some are obsolescent and
have not been updated in a while, while some are recent and
follow all the latest best-practice guidelines (and the rest
is somewhere in the middle).

Are there 1 or 2 very good PCIe drivers to take as examples
as good starting points? (Perhaps a simple driver, and a
more complex driver.)

Does anyone know if any SoC with a driver upstream is using
a PLDA controller?

Regards.

Re: Pointers for writing a good PCIe driver

[Bjorn Helgaas]

On Tue, Feb 07, 2017 at 04:55:28PM +0100, Mason wrote:
> On 06/02/2017 16:54, Mason wrote:
> 
> > My platform ( arch/arm/mach-tango ) provides a PCIe controller from PLDA.
> > 
> > I do have access to a driver that works for a few PCIe boards, but it's
> > an ancient out-of-tree driver (targeting v3.4). Also, I'm not sure it
> > follows the best-practice guidelines (e.g. it hard-codes a few
> > work-arounds directly in drivers/pci/probe.c)

Indeed, having to change drivers/pci/probe.c doesn't sound like a best
practice :)  If you can share details of what changes you need, people
could probably suggest other ways to do it within the generic PCI
framework.

> > If I understand correctly, while PCIe is a standard, it is not codified
> > as much as USB (XHCI) or SATA (AHCI), which means the register layout
> > (and possibly other things) are left up to the integrator? Which means
> > there cannot be some kind of "generic" driver that works for a random
> > PCIe controller?

There is no standardization of registers in MMIO or I/O port space.
The size, location, and type of those registers is configurable in a
generic way via the BARs, of course.  But that tells you nothing about
the *functionality* of those MMIO or I/O port registers.

For config space, the first 64 bytes (the PCIe r3.0 spec calls it the
"PCI 3.0 Compatible Configuration Space Header") are completely
standardized.  This is where device IDs, BARs, and other generic
registers are.

PCIe devices can implement up to 4096 bytes of config space.  The
space after the 64-byte header can contain a list of standardized
PCI/PCIe Capabilities, non-standardized device-specific registers, or
both.  There is a way to include device-specific registers in a
"Vendor Defined" Capability -- that way drivers can use generic
mechanisms to find registers so they don't have to hard-code register
layout assumptions.

> > I guess my main question is: do you have pointers on getting started
> > writing a PCIe driver good enough for upstream?
> > 
> > Is Documentation/PCI/PCIEBUS-HOWTO.txt still a relevant resource?

PCIEBUS-HOWTO.txt is still relevant, but only to drivers for services
of PCIe ports, e.g., hotplug, error reporting, etc.  It's not relevant
to PCIe endpoint drivers.

> > Are there good resources to get up to speed on the PCIe terminology?
> 
> I'm also wondering: in many Linux subsystems, there are often
> several drivers of different quality; some are obsolescent and
> have not been updated in a while, while some are recent and
> follow all the latest best-practice guidelines (and the rest
> is somewhere in the middle).
> 
> Are there 1 or 2 very good PCIe drivers to take as examples
> as good starting points? (Perhaps a simple driver, and a
> more complex driver.)

I'm not really the best person to ask about this because I deal more
with the PCI core than with the drivers that use the core.  If I were
looking, I might start with drivers/nvme/host/pci.c or
drivers/usb/host/xhci-pci.c.

Bjorn

Re: Pointers for writing a good PCIe driver

[Bjorn Helgaas]

On Tue, Feb 07, 2017 at 02:06:56PM -0600, Bjorn Helgaas wrote:
> On Tue, Feb 07, 2017 at 04:55:28PM +0100, Mason wrote:
> > On 06/02/2017 16:54, Mason wrote:
> > 
> > > My platform ( arch/arm/mach-tango ) provides a PCIe controller from PLDA.
> > > 
> > > I do have access to a driver that works for a few PCIe boards, but it's
> > > an ancient out-of-tree driver (targeting v3.4). Also, I'm not sure it
> > > follows the best-practice guidelines (e.g. it hard-codes a few
> > > work-arounds directly in drivers/pci/probe.c)
> 
> Indeed, having to change drivers/pci/probe.c doesn't sound like a best
> practice :)  If you can share details of what changes you need, people
> could probably suggest other ways to do it within the generic PCI
> framework.
> 
> > > If I understand correctly, while PCIe is a standard, it is not codified
> > > as much as USB (XHCI) or SATA (AHCI), which means the register layout
> > > (and possibly other things) are left up to the integrator? Which means
> > > there cannot be some kind of "generic" driver that works for a random
> > > PCIe controller?
> 
> There is no standardization of registers in MMIO or I/O port space.
> The size, location, and type of those registers is configurable in a
> generic way via the BARs, of course.  But that tells you nothing about
> the *functionality* of those MMIO or I/O port registers.
> 
> For config space, the first 64 bytes (the PCIe r3.0 spec calls it the
> "PCI 3.0 Compatible Configuration Space Header") are completely
> standardized.  This is where device IDs, BARs, and other generic
> registers are.
> 
> PCIe devices can implement up to 4096 bytes of config space.  The
> space after the 64-byte header can contain a list of standardized
> PCI/PCIe Capabilities, non-standardized device-specific registers, or
> both.  There is a way to include device-specific registers in a
> "Vendor Defined" Capability -- that way drivers can use generic
> mechanisms to find registers so they don't have to hard-code register
> layout assumptions.
> 
> > > I guess my main question is: do you have pointers on getting started
> > > writing a PCIe driver good enough for upstream?
> > > 
> > > Is Documentation/PCI/PCIEBUS-HOWTO.txt still a relevant resource?
> 
> PCIEBUS-HOWTO.txt is still relevant, but only to drivers for services
> of PCIe ports, e.g., hotplug, error reporting, etc.  It's not relevant
> to PCIe endpoint drivers.
> 
> > > Are there good resources to get up to speed on the PCIe terminology?
> > 
> > I'm also wondering: in many Linux subsystems, there are often
> > several drivers of different quality; some are obsolescent and
> > have not been updated in a while, while some are recent and
> > follow all the latest best-practice guidelines (and the rest
> > is somewhere in the middle).
> > 
> > Are there 1 or 2 very good PCIe drivers to take as examples
> > as good starting points? (Perhaps a simple driver, and a
> > more complex driver.)
> 
> I'm not really the best person to ask about this because I deal more
> with the PCI core than with the drivers that use the core.  If I were
> looking, I might start with drivers/nvme/host/pci.c or
> drivers/usb/host/xhci-pci.c.

I should have also mentioned Documentation/PCI/pci.txt.  That's a
pretty good place to start.

Re: Pointers for writing a good PCIe driver

[Mason]

On 07/02/2017 22:47, Bjorn Helgaas wrote:
> On Tue, Feb 07, 2017 at 02:06:56PM -0600, Bjorn Helgaas wrote:
>> On Tue, Feb 07, 2017 at 04:55:28PM +0100, Mason wrote:
>>> On 06/02/2017 16:54, Mason wrote:
>>>
>>>> My platform ( arch/arm/mach-tango ) provides a PCIe controller from PLDA.
>>>>
>>>> I do have access to a driver that works for a few PCIe boards, but it's
>>>> an ancient out-of-tree driver (targeting v3.4). Also, I'm not sure it
>>>> follows the best-practice guidelines (e.g. it hard-codes a few
>>>> work-arounds directly in drivers/pci/probe.c)
>>
>> Indeed, having to change drivers/pci/probe.c doesn't sound like a best
>> practice :)  If you can share details of what changes you need, people
>> could probably suggest other ways to do it within the generic PCI
>> framework.
>>
>>>> If I understand correctly, while PCIe is a standard, it is not codified
>>>> as much as USB (XHCI) or SATA (AHCI), which means the register layout
>>>> (and possibly other things) are left up to the integrator? Which means
>>>> there cannot be some kind of "generic" driver that works for a random
>>>> PCIe controller?
>>
>> There is no standardization of registers in MMIO or I/O port space.
>> The size, location, and type of those registers is configurable in a
>> generic way via the BARs, of course.  But that tells you nothing about
>> the *functionality* of those MMIO or I/O port registers.
>>
>> For config space, the first 64 bytes (the PCIe r3.0 spec calls it the
>> "PCI 3.0 Compatible Configuration Space Header") are completely
>> standardized.  This is where device IDs, BARs, and other generic
>> registers are.
>>
>> PCIe devices can implement up to 4096 bytes of config space.  The
>> space after the 64-byte header can contain a list of standardized
>> PCI/PCIe Capabilities, non-standardized device-specific registers, or
>> both.  There is a way to include device-specific registers in a
>> "Vendor Defined" Capability -- that way drivers can use generic
>> mechanisms to find registers so they don't have to hard-code register
>> layout assumptions.
>>
>>>> I guess my main question is: do you have pointers on getting started
>>>> writing a PCIe driver good enough for upstream?
>>>>
>>>> Is Documentation/PCI/PCIEBUS-HOWTO.txt still a relevant resource?
>>
>> PCIEBUS-HOWTO.txt is still relevant, but only to drivers for services
>> of PCIe ports, e.g., hotplug, error reporting, etc.  It's not relevant
>> to PCIe endpoint drivers.
>>
>>>> Are there good resources to get up to speed on the PCIe terminology?
>>>
>>> I'm also wondering: in many Linux subsystems, there are often
>>> several drivers of different quality; some are obsolescent and
>>> have not been updated in a while, while some are recent and
>>> follow all the latest best-practice guidelines (and the rest
>>> is somewhere in the middle).
>>>
>>> Are there 1 or 2 very good PCIe drivers to take as examples
>>> as good starting points? (Perhaps a simple driver, and a
>>> more complex driver.)
>>
>> I'm not really the best person to ask about this because I deal more
>> with the PCI core than with the drivers that use the core.  If I were
>> looking, I might start with drivers/nvme/host/pci.c or
>> drivers/usb/host/xhci-pci.c.
> 
> I should have also mentioned Documentation/PCI/pci.txt.  That's a
> pretty good place to start.

Hello Bjorn,

Thanks for these pointers, I will study them with care.

Who would review my PCIe driver if I ever submitted one? :-)

Regards.

Re: Pointers for writing a good PCIe driver

[Greg KH]

On Tue, Feb 07, 2017 at 11:56:31PM +0100, Mason wrote:
> On 07/02/2017 22:47, Bjorn Helgaas wrote:
> > On Tue, Feb 07, 2017 at 02:06:56PM -0600, Bjorn Helgaas wrote:
> >> On Tue, Feb 07, 2017 at 04:55:28PM +0100, Mason wrote:
> >>> On 06/02/2017 16:54, Mason wrote:
> >>>
> >>>> My platform ( arch/arm/mach-tango ) provides a PCIe controller from PLDA.
> >>>>
> >>>> I do have access to a driver that works for a few PCIe boards, but it's
> >>>> an ancient out-of-tree driver (targeting v3.4). Also, I'm not sure it
> >>>> follows the best-practice guidelines (e.g. it hard-codes a few
> >>>> work-arounds directly in drivers/pci/probe.c)
> >>
> >> Indeed, having to change drivers/pci/probe.c doesn't sound like a best
> >> practice :)  If you can share details of what changes you need, people
> >> could probably suggest other ways to do it within the generic PCI
> >> framework.
> >>
> >>>> If I understand correctly, while PCIe is a standard, it is not codified
> >>>> as much as USB (XHCI) or SATA (AHCI), which means the register layout
> >>>> (and possibly other things) are left up to the integrator? Which means
> >>>> there cannot be some kind of "generic" driver that works for a random
> >>>> PCIe controller?
> >>
> >> There is no standardization of registers in MMIO or I/O port space.
> >> The size, location, and type of those registers is configurable in a
> >> generic way via the BARs, of course.  But that tells you nothing about
> >> the *functionality* of those MMIO or I/O port registers.
> >>
> >> For config space, the first 64 bytes (the PCIe r3.0 spec calls it the
> >> "PCI 3.0 Compatible Configuration Space Header") are completely
> >> standardized.  This is where device IDs, BARs, and other generic
> >> registers are.
> >>
> >> PCIe devices can implement up to 4096 bytes of config space.  The
> >> space after the 64-byte header can contain a list of standardized
> >> PCI/PCIe Capabilities, non-standardized device-specific registers, or
> >> both.  There is a way to include device-specific registers in a
> >> "Vendor Defined" Capability -- that way drivers can use generic
> >> mechanisms to find registers so they don't have to hard-code register
> >> layout assumptions.
> >>
> >>>> I guess my main question is: do you have pointers on getting started
> >>>> writing a PCIe driver good enough for upstream?
> >>>>
> >>>> Is Documentation/PCI/PCIEBUS-HOWTO.txt still a relevant resource?
> >>
> >> PCIEBUS-HOWTO.txt is still relevant, but only to drivers for services
> >> of PCIe ports, e.g., hotplug, error reporting, etc.  It's not relevant
> >> to PCIe endpoint drivers.
> >>
> >>>> Are there good resources to get up to speed on the PCIe terminology?
> >>>
> >>> I'm also wondering: in many Linux subsystems, there are often
> >>> several drivers of different quality; some are obsolescent and
> >>> have not been updated in a while, while some are recent and
> >>> follow all the latest best-practice guidelines (and the rest
> >>> is somewhere in the middle).
> >>>
> >>> Are there 1 or 2 very good PCIe drivers to take as examples
> >>> as good starting points? (Perhaps a simple driver, and a
> >>> more complex driver.)
> >>
> >> I'm not really the best person to ask about this because I deal more
> >> with the PCI core than with the drivers that use the core.  If I were
> >> looking, I might start with drivers/nvme/host/pci.c or
> >> drivers/usb/host/xhci-pci.c.
> > 
> > I should have also mentioned Documentation/PCI/pci.txt.  That's a
> > pretty good place to start.
> 
> Hello Bjorn,
> 
> Thanks for these pointers, I will study them with care.
> 
> Who would review my PCIe driver if I ever submitted one? :-)

It all depends on what subsystem it would belong to, what type of device
are you writing a driver for?

thanks,

greg k-h

Re: Pointers for writing a good PCIe driver

[Mason]

On 12/02/2017 17:50, Greg KH wrote:
> On Tue, Feb 07, 2017 at 11:56:31PM +0100, Mason wrote:
>> On 07/02/2017 22:47, Bjorn Helgaas wrote:
>>> On Tue, Feb 07, 2017 at 02:06:56PM -0600, Bjorn Helgaas wrote:
>>>> On Tue, Feb 07, 2017 at 04:55:28PM +0100, Mason wrote:
>>>>> On 06/02/2017 16:54, Mason wrote:
>>>>>
>>>>>> My platform ( arch/arm/mach-tango ) provides a PCIe controller from PLDA.
>>>>>>
>>>>>> I do have access to a driver that works for a few PCIe boards, but it's
>>>>>> an ancient out-of-tree driver (targeting v3.4). Also, I'm not sure it
>>>>>> follows the best-practice guidelines (e.g. it hard-codes a few
>>>>>> work-arounds directly in drivers/pci/probe.c)
>>>>
>>>> Indeed, having to change drivers/pci/probe.c doesn't sound like a best
>>>> practice :)  If you can share details of what changes you need, people
>>>> could probably suggest other ways to do it within the generic PCI
>>>> framework.
>>>>
>>>>>> If I understand correctly, while PCIe is a standard, it is not codified
>>>>>> as much as USB (XHCI) or SATA (AHCI), which means the register layout
>>>>>> (and possibly other things) are left up to the integrator? Which means
>>>>>> there cannot be some kind of "generic" driver that works for a random
>>>>>> PCIe controller?
>>>>
>>>> There is no standardization of registers in MMIO or I/O port space.
>>>> The size, location, and type of those registers is configurable in a
>>>> generic way via the BARs, of course.  But that tells you nothing about
>>>> the *functionality* of those MMIO or I/O port registers.
>>>>
>>>> For config space, the first 64 bytes (the PCIe r3.0 spec calls it the
>>>> "PCI 3.0 Compatible Configuration Space Header") are completely
>>>> standardized.  This is where device IDs, BARs, and other generic
>>>> registers are.
>>>>
>>>> PCIe devices can implement up to 4096 bytes of config space.  The
>>>> space after the 64-byte header can contain a list of standardized
>>>> PCI/PCIe Capabilities, non-standardized device-specific registers, or
>>>> both.  There is a way to include device-specific registers in a
>>>> "Vendor Defined" Capability -- that way drivers can use generic
>>>> mechanisms to find registers so they don't have to hard-code register
>>>> layout assumptions.
>>>>
>>>>>> I guess my main question is: do you have pointers on getting started
>>>>>> writing a PCIe driver good enough for upstream?
>>>>>>
>>>>>> Is Documentation/PCI/PCIEBUS-HOWTO.txt still a relevant resource?
>>>>
>>>> PCIEBUS-HOWTO.txt is still relevant, but only to drivers for services
>>>> of PCIe ports, e.g., hotplug, error reporting, etc.  It's not relevant
>>>> to PCIe endpoint drivers.
>>>>
>>>>>> Are there good resources to get up to speed on the PCIe terminology?
>>>>>
>>>>> I'm also wondering: in many Linux subsystems, there are often
>>>>> several drivers of different quality; some are obsolescent and
>>>>> have not been updated in a while, while some are recent and
>>>>> follow all the latest best-practice guidelines (and the rest
>>>>> is somewhere in the middle).
>>>>>
>>>>> Are there 1 or 2 very good PCIe drivers to take as examples
>>>>> as good starting points? (Perhaps a simple driver, and a
>>>>> more complex driver.)
>>>>
>>>> I'm not really the best person to ask about this because I deal more
>>>> with the PCI core than with the drivers that use the core.  If I were
>>>> looking, I might start with drivers/nvme/host/pci.c or
>>>> drivers/usb/host/xhci-pci.c.
>>>
>>> I should have also mentioned Documentation/PCI/pci.txt.  That's a
>>> pretty good place to start.
>>
>> Hello Bjorn,
>>
>> Thanks for these pointers, I will study them with care.
>>
>> Who would review my PCIe driver if I ever submitted one? :-)
> 
> It all depends on what subsystem it would belong to, what type of device
> are you writing a driver for?

Hello Greg,

There might be some kind of misunderstanding.

I don't plan to write a driver for a device that plugs into a PCIe slot.

I intend to write a driver for the PCIe controller itself, which is
embedded in the SoC. (In order to support different types of PCIe
devices, with their respective drivers already upstream.)

I think the controller driver belongs in drivers/pci/host ?

Regards.

Re: Pointers for writing a good PCIe driver

[Bjorn Helgaas]

On Fri, Feb 17, 2017 at 10:20:20AM +0100, Mason wrote:
> On 12/02/2017 17:50, Greg KH wrote:
> > On Tue, Feb 07, 2017 at 11:56:31PM +0100, Mason wrote:
> >> On 07/02/2017 22:47, Bjorn Helgaas wrote:
> >>> On Tue, Feb 07, 2017 at 02:06:56PM -0600, Bjorn Helgaas wrote:
> >>>> On Tue, Feb 07, 2017 at 04:55:28PM +0100, Mason wrote:
> >>>>> On 06/02/2017 16:54, Mason wrote:
> >>>>>
> >>>>>> My platform ( arch/arm/mach-tango ) provides a PCIe controller from PLDA.
> >>>>>>
> >>>>>> I do have access to a driver that works for a few PCIe boards, but it's
> >>>>>> an ancient out-of-tree driver (targeting v3.4). Also, I'm not sure it
> >>>>>> follows the best-practice guidelines (e.g. it hard-codes a few
> >>>>>> work-arounds directly in drivers/pci/probe.c)
> >>>>
> >>>> Indeed, having to change drivers/pci/probe.c doesn't sound like a best
> >>>> practice :)  If you can share details of what changes you need, people
> >>>> could probably suggest other ways to do it within the generic PCI
> >>>> framework.
> >>>>
> >>>>>> If I understand correctly, while PCIe is a standard, it is not codified
> >>>>>> as much as USB (XHCI) or SATA (AHCI), which means the register layout
> >>>>>> (and possibly other things) are left up to the integrator? Which means
> >>>>>> there cannot be some kind of "generic" driver that works for a random
> >>>>>> PCIe controller?
> >>>>
> >>>> There is no standardization of registers in MMIO or I/O port space.
> >>>> The size, location, and type of those registers is configurable in a
> >>>> generic way via the BARs, of course.  But that tells you nothing about
> >>>> the *functionality* of those MMIO or I/O port registers.
> >>>>
> >>>> For config space, the first 64 bytes (the PCIe r3.0 spec calls it the
> >>>> "PCI 3.0 Compatible Configuration Space Header") are completely
> >>>> standardized.  This is where device IDs, BARs, and other generic
> >>>> registers are.
> >>>>
> >>>> PCIe devices can implement up to 4096 bytes of config space.  The
> >>>> space after the 64-byte header can contain a list of standardized
> >>>> PCI/PCIe Capabilities, non-standardized device-specific registers, or
> >>>> both.  There is a way to include device-specific registers in a
> >>>> "Vendor Defined" Capability -- that way drivers can use generic
> >>>> mechanisms to find registers so they don't have to hard-code register
> >>>> layout assumptions.
> >>>>
> >>>>>> I guess my main question is: do you have pointers on getting started
> >>>>>> writing a PCIe driver good enough for upstream?
> >>>>>>
> >>>>>> Is Documentation/PCI/PCIEBUS-HOWTO.txt still a relevant resource?
> >>>>
> >>>> PCIEBUS-HOWTO.txt is still relevant, but only to drivers for services
> >>>> of PCIe ports, e.g., hotplug, error reporting, etc.  It's not relevant
> >>>> to PCIe endpoint drivers.
> >>>>
> >>>>>> Are there good resources to get up to speed on the PCIe terminology?
> >>>>>
> >>>>> I'm also wondering: in many Linux subsystems, there are often
> >>>>> several drivers of different quality; some are obsolescent and
> >>>>> have not been updated in a while, while some are recent and
> >>>>> follow all the latest best-practice guidelines (and the rest
> >>>>> is somewhere in the middle).
> >>>>>
> >>>>> Are there 1 or 2 very good PCIe drivers to take as examples
> >>>>> as good starting points? (Perhaps a simple driver, and a
> >>>>> more complex driver.)
> >>>>
> >>>> I'm not really the best person to ask about this because I deal more
> >>>> with the PCI core than with the drivers that use the core.  If I were
> >>>> looking, I might start with drivers/nvme/host/pci.c or
> >>>> drivers/usb/host/xhci-pci.c.
> >>>
> >>> I should have also mentioned Documentation/PCI/pci.txt.  That's a
> >>> pretty good place to start.
> >>
> >> Hello Bjorn,
> >>
> >> Thanks for these pointers, I will study them with care.
> >>
> >> Who would review my PCIe driver if I ever submitted one? :-)
> > 
> > It all depends on what subsystem it would belong to, what type of device
> > are you writing a driver for?
> 
> Hello Greg,
> 
> There might be some kind of misunderstanding.
> 
> I don't plan to write a driver for a device that plugs into a PCIe slot.
> 
> I intend to write a driver for the PCIe controller itself, which is
> embedded in the SoC. (In order to support different types of PCIe
> devices, with their respective drivers already upstream.)
> 
> I think the controller driver belongs in drivers/pci/host ?

Sorry, I indeed misunderstood you.

The native PCI host controller drivers indeed live in
drivers/pci/host/.

I don't know anything about your hardware or environment, but I highly
encourage you to use ACPI (drivers/acpi/pci_root.c, with a little bit
of arch support) or generic DT (drivers/pci/host/pci-host-generic.c)
instead of writing a custom host controller driver.

The native drivers in drivers/pci/host are a huge maintenance hassle
for no real benefit.

Bjorn

Re: Pointers for writing a good PCIe driver

[Mason]

On 17/02/2017 15:38, Bjorn Helgaas wrote:

> On Fri, Feb 17, 2017 at 10:20:20AM +0100, Mason wrote:
>
>> There might be some kind of misunderstanding.
>>
>> I don't plan to write a driver for a device that plugs into a PCIe slot.
>>
>> I intend to write a driver for the PCIe controller itself, which is
>> embedded in the SoC. (In order to support different types of PCIe
>> devices, with their respective drivers already upstream.)
>>
>> I think the controller driver belongs in drivers/pci/host ?
> 
> Sorry, I indeed misunderstood you.
> 
> The native PCI host controller drivers indeed live in
> drivers/pci/host/.
> 
> I don't know anything about your hardware or environment, but I highly
> encourage you to use ACPI (drivers/acpi/pci_root.c, with a little bit
> of arch support) or generic DT (drivers/pci/host/pci-host-generic.c)
> instead of writing a custom host controller driver.
> 
> The native drivers in drivers/pci/host are a huge maintenance hassle
> for no real benefit.

My SoC is built around an ARM Cortex A9.
I don't think (?) this platform has any support for ACPI.

I'm currently using this DT description:
http://lxr.free-electrons.com/source/arch/arm/boot/dts/tango4-common.dtsi

The "legacy" driver sits at ~700 lines. It would be awesome to be able
to discard all of it, and rely on generic upstream code. But I don't
understand how it is possible for a generic driver to support whatever
crazy solution some random vendor has come up with?

Regards.

Pointers for writing a good PCIe driver

[Mason]

On 17/02/2017 15:38, Bjorn Helgaas wrote:

> On Fri, Feb 17, 2017 at 10:20:20AM +0100, Mason wrote:
>
>> There might be some kind of misunderstanding.
>>
>> I don't plan to write a driver for a device that plugs into a PCIe slot.
>>
>> I intend to write a driver for the PCIe controller itself, which is
>> embedded in the SoC. (In order to support different types of PCIe
>> devices, with their respective drivers already upstream.)
>>
>> I think the controller driver belongs in drivers/pci/host ?
> 
> Sorry, I indeed misunderstood you.
> 
> The native PCI host controller drivers indeed live in
> drivers/pci/host/.
> 
> I don't know anything about your hardware or environment, but I highly
> encourage you to use ACPI (drivers/acpi/pci_root.c, with a little bit
> of arch support) or generic DT (drivers/pci/host/pci-host-generic.c)
> instead of writing a custom host controller driver.
> 
> The native drivers in drivers/pci/host are a huge maintenance hassle
> for no real benefit.

My SoC is built around an ARM Cortex A9.
I don't think (?) this platform has any support for ACPI.

I'm currently using this DT description:
http://lxr.free-electrons.com/source/arch/arm/boot/dts/tango4-common.dtsi

The "legacy" driver sits at ~700 lines. It would be awesome to be able
to discard all of it, and rely on generic upstream code. But I don't
understand how it is possible for a generic driver to support whatever
crazy solution some random vendor has come up with?

Regards.

Re: Pointers for writing a good PCIe driver

[Bjorn Helgaas]

On Fri, Feb 17, 2017 at 06:00:53PM +0100, Mason wrote:
> On 17/02/2017 15:38, Bjorn Helgaas wrote:
> 
> > On Fri, Feb 17, 2017 at 10:20:20AM +0100, Mason wrote:
> >
> >> There might be some kind of misunderstanding.
> >>
> >> I don't plan to write a driver for a device that plugs into a PCIe slot.
> >>
> >> I intend to write a driver for the PCIe controller itself, which is
> >> embedded in the SoC. (In order to support different types of PCIe
> >> devices, with their respective drivers already upstream.)
> >>
> >> I think the controller driver belongs in drivers/pci/host ?
> > 
> > Sorry, I indeed misunderstood you.
> > 
> > The native PCI host controller drivers indeed live in
> > drivers/pci/host/.
> > 
> > I don't know anything about your hardware or environment, but I highly
> > encourage you to use ACPI (drivers/acpi/pci_root.c, with a little bit
> > of arch support) or generic DT (drivers/pci/host/pci-host-generic.c)
> > instead of writing a custom host controller driver.
> > 
> > The native drivers in drivers/pci/host are a huge maintenance hassle
> > for no real benefit.
> 
> My SoC is built around an ARM Cortex A9.
> I don't think (?) this platform has any support for ACPI.

There's ACPI support on ARM64, but none that I know of for ARM32.

> I'm currently using this DT description:
> http://lxr.free-electrons.com/source/arch/arm/boot/dts/tango4-common.dtsi
> 
> The "legacy" driver sits at ~700 lines. It would be awesome to be able
> to discard all of it, and rely on generic upstream code. But I don't
> understand how it is possible for a generic driver to support whatever
> crazy solution some random vendor has come up with?

You're right that the programming model of the host bridge itself is
not specified by PCI specs, so it's impossible to have a generic
driver that can manage it completely by itself.

If you have firmware that initializes the bridge and tells the OS what
the windows are (bus numbers, memory space, I/O space) and where the
PCIe-specified ECAM space is, a generic driver can take it from there.

Bjorn

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Pointers for writing a good PCIe driver

[Bjorn Helgaas]

On Fri, Feb 17, 2017 at 06:00:53PM +0100, Mason wrote:
> On 17/02/2017 15:38, Bjorn Helgaas wrote:
> 
> > On Fri, Feb 17, 2017 at 10:20:20AM +0100, Mason wrote:
> >
> >> There might be some kind of misunderstanding.
> >>
> >> I don't plan to write a driver for a device that plugs into a PCIe slot.
> >>
> >> I intend to write a driver for the PCIe controller itself, which is
> >> embedded in the SoC. (In order to support different types of PCIe
> >> devices, with their respective drivers already upstream.)
> >>
> >> I think the controller driver belongs in drivers/pci/host ?
> > 
> > Sorry, I indeed misunderstood you.
> > 
> > The native PCI host controller drivers indeed live in
> > drivers/pci/host/.
> > 
> > I don't know anything about your hardware or environment, but I highly
> > encourage you to use ACPI (drivers/acpi/pci_root.c, with a little bit
> > of arch support) or generic DT (drivers/pci/host/pci-host-generic.c)
> > instead of writing a custom host controller driver.
> > 
> > The native drivers in drivers/pci/host are a huge maintenance hassle
> > for no real benefit.
> 
> My SoC is built around an ARM Cortex A9.
> I don't think (?) this platform has any support for ACPI.

There's ACPI support on ARM64, but none that I know of for ARM32.

> I'm currently using this DT description:
> http://lxr.free-electrons.com/source/arch/arm/boot/dts/tango4-common.dtsi
> 
> The "legacy" driver sits at ~700 lines. It would be awesome to be able
> to discard all of it, and rely on generic upstream code. But I don't
> understand how it is possible for a generic driver to support whatever
> crazy solution some random vendor has come up with?

You're right that the programming model of the host bridge itself is
not specified by PCI specs, so it's impossible to have a generic
driver that can manage it completely by itself.

If you have firmware that initializes the bridge and tells the OS what
the windows are (bus numbers, memory space, I/O space) and where the
PCIe-specified ECAM space is, a generic driver can take it from there.

Bjorn

Re: Pointers for writing a good PCIe driver

[Mason]

On 17/02/2017 18:50, Bjorn Helgaas wrote:

> On Fri, Feb 17, 2017 at 06:00:53PM +0100, Mason wrote:
>
>> On 17/02/2017 15:38, Bjorn Helgaas wrote:
>>
>>> The native PCI host controller drivers indeed live in drivers/pci/host/
>>>
>>> I don't know anything about your hardware or environment, but I highly
>>> encourage you to use ACPI (drivers/acpi/pci_root.c, with a little bit
>>> of arch support) or generic DT (drivers/pci/host/pci-host-generic.c)
>>> instead of writing a custom host controller driver.
>>>
>>> The native drivers in drivers/pci/host are a huge maintenance hassle
>>> for no real benefit.
>>
>> I'm currently using this DT description:
>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/tango4-common.dtsi
>>
>> The "legacy" driver sits at ~700 lines. It would be awesome to be able
>> to discard all of it, and rely on generic upstream code. But I don't
>> understand how it is possible for a generic driver to support whatever
>> crazy solution some random vendor has come up with?
> 
> You're right that the programming model of the host bridge itself is
> not specified by PCI specs, so it's impossible to have a generic
> driver that can manage it completely by itself.
> 
> If you have firmware that initializes the bridge and tells the OS what
> the windows are (bus numbers, memory space, I/O space) and where the
> PCIe-specified ECAM space is, a generic driver can take it from there.

(Mostly walking myself through the code and documentation,
noting references along the way.)

I took a look at drivers/pci/host/pci-host-generic.c
which is a tiny file, so the magic must be happening elsewhere.
=> drivers/pci/host/pci-host-common.c (for starters)

gen_pci_cfg_cam_bus_ops and pci_generic_ecam_ops have the same
.pci_ops, but different .bus_shift

$ git grep pci-host-cam-generic
Documentation/devicetree/bindings/pci/host-generic-pci.txt:- compatible     : Must be "pci-host-cam-generic" or "pci-host-ecam-generic"
Documentation/devicetree/bindings/pci/host-generic-pci.txt:    compatible = "pci-host-cam-generic"
drivers/pci/host/pci-host-generic.c:    { .compatible = "pci-host-cam-generic",

$ git grep pci-host-ecam-generic
Documentation/devicetree/bindings/pci/host-generic-pci.txt:- compatible     : Must be "pci-host-cam-generic" or "pci-host-ecam-generic"
arch/arm/boot/dts/alpine.dtsi:                  compatible = "pci-host-ecam-generic";
arch/arm64/boot/dts/al/alpine-v2.dtsi:                  compatible = "pci-host-ecam-generic";
arch/arm64/boot/dts/amd/amd-seattle-soc.dtsi:                   compatible = "pci-host-ecam-generic";
arch/arm64/boot/dts/arm/juno-base.dtsi:         compatible = "arm,juno-r1-pcie", "plda,xpressrich3-axi", "pci-host-ecam-generic";
arch/arm64/boot/dts/broadcom/vulcan.dtsi:               compatible = "pci-host-ecam-generic";
drivers/pci/host/pci-host-generic.c:    { .compatible = "pci-host-ecam-generic",

https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/host-generic-pci.txt

> Firmware-initialised PCI host controllers and PCI emulations, such as the
> virtio-pci implementations found in kvmtool and other para-virtualised
> systems, do not require driver support for complexities such as regulator
> and clock management. In fact, the controller may not even require the
> configuration of a control interface by the operating system, instead
> presenting a set of fixed windows describing a subset of IO, Memory and
> Configuration Spaces.

For the only arm32 user, arch/arm/boot/dts/alpine.dtsi:

		/* Internal PCIe Controller */
		pcie-internal@0xfbc00000 {
			compatible = "pci-host-ecam-generic";
			device_type = "pci";
			#size-cells = <2>;
			#address-cells = <3>;
			#interrupt-cells = <1>;
			reg = <0x0 0xfbc00000 0x0 0x100000>;
			interrupt-map-mask = <0xf800 0 0 7>;
			/* Add legacy interrupts for SATA devices only */
			interrupt-map =	<0x4000 0 0 1 &gic 0 43 4>,
					<0x4800 0 0 1 &gic 0 44 4>;

			/* 32 bit non prefetchable memory space */
			ranges = <0x02000000 0x0 0xfe000000 0x0 0xfe000000 0x0 0x1000000>;

			bus-range = <0x00 0x00>;
			msi-parent = <&msix>;
		};


Compatible string depends on "the layout of configuration space
(CAM vs ECAM respectively)"

https://en.wikipedia.org/wiki/PCI_configuration_space


The register interface to my PCIe controller is:

0x2e000	REGION_0_BASE_REG
0x2e004	REGION_1_BASE_REG
0x2e008	REGION_2_BASE_REG
0x2e00c	REGION_3_BASE_REG
0x2e010	REGION_4_BASE_REG
0x2e014	REGION_5_BASE_REG
0x2e018	REGION_6_BASE_REG
0x2e01c	REGION_7_BASE_REG
0x2e020	DMA_RD_MBUS_ADDR_REG
0x2e024	DMA_RD_ADDR_LO_REG
0x2e028	DMA_RD_ADDR_HI_REG
0x2e02c	DMA_RD_CNT_REG
0x2e030	DMA_WR_MBUS_ADDR_REG
0x2e034	DMA_WR_ADDR_LO_REG
0x2e038	DMA_WR_ADDR_HI_REG
0x2e03c	DMA_WR_CNT_REG
0x2e040	DMA_RD_EN_REG
0x2e044	DMA_WR_EN_REG
0x2e048	PCI_ADDR_LO_BASE_REG
0x2e04c	PCI_ADDR_HI_BASE_REG
0x2e050	PM_REG
0x2e054	MAX_CPL_TIMEOUT_REG
0x2e058	CORE_CONF_0_REG
0x2e05c	CORE_CONF_1_REG
0x2e060	CBA_ADDR_REG
0x2e064	CBA_DATA_REG
0x2e068	DEBUG_0_REG
0x2e06c	DEBUG_1_REG
0x2e070	DEBUG_2_REG
0x2e074	CORE_TEST_OUT_REG
0x2e078	INT_NOT_MSI_REG
0x2e07c	MSI_REG
0x2e080	MSI_0_REG
0x2e084	MSI_1_REG
0x2e088	MSI_2_REG
0x2e08c	MSI_3_REG
0x2e090	MSI_4_REG
0x2e094	MSI_5_REG
0x2e098	MSI_6_REG
0x2e09c	MSI_7_REG
0x2e0a0	MSI_0_MASK_REG
0x2e0a4	MSI_1_MASK_REG
0x2e0a8	MSI_2_MASK_REG
0x2e0ac	MSI_3_MASK_REG
0x2e0b0	MSI_4_MASK_REG
0x2e0b4	MSI_5_MASK_REG
0x2e0b8	MSI_6_MASK_REG
0x2e0bc	MSI_7_MASK_REG
0x2e0c0	PHY_HW_TRAPS_REG
0x2e0c4	COMMON_BIST_0_REG
0x2e0c8	COMMON_BIST_1_REG
0x2e0cc	LANE_0_BIST_REG
0x2e0d0	CUSTOM_CONF_LINK_REG
0x2e0d4	CUSTOM_CONF_LANE_0_REG
0x2e0e0	PLL_SETTING_REG
0x2e0e4	DEBUG_3_REG
0x2e0e8	DEBUG_4_REG
0x2e0ec	DEBUG_5_REG


In the "legacy" driver (for kernel v3.4), I see

static struct pci_ops tangox_pcie_ops = {
	.read	= tangox_pcie_read_conf,
	.write	= tangox_pcie_write_conf,
};

#define TANGOX_PCIE_ENABLE_CONFIG_IO()      WR_PCIE_REG32(TANGOX_PCIE_PCI_ADDR_LO_BASE_REG, ((TANGOX_PCIE_ADDR_BASE & 0xff000000) | 0x1))
#define TANGOX_PCIE_DISABLE_CONFIG_IO()     WR_PCIE_REG32(TANGOX_PCIE_PCI_ADDR_LO_BASE_REG, ((TANGOX_PCIE_ADDR_BASE & 0xff000000) & ~0x1))

static int tangox_pcie_read_conf(struct pci_bus *bus, unsigned int devfn,
                                 int where, int size, u32 *val)
{
    void __iomem *addr;

    TANGOX_PCIE_ENABLE_CONFIG_IO();

    addr =  (void __iomem *)(
            tangox_pcie.regs +
            PCIE_CONF_BUS(bus->number) +
            PCIE_CONF_DEV(PCI_SLOT(devfn)) +
            PCIE_CONF_FUNC(PCI_FUNC(devfn)) +
            PCIE_CONF_REG(where));

    *val = readl( addr );

	if (size == 1)
		*val = (*val >> (8 * (where & 3))) & 0xff;
	else if (size == 2)
		*val = (*val >> (8 * (where & 3))) & 0xffff;

    TANGOX_PCIE_DISABLE_CONFIG_IO();

    return PCIBIOS_SUCCESSFUL;
}


I see that pci-host-common.c calls:
pci_scan_root_bus(dev, cfg->busr.start, &ops->pci_ops, cfg, &resources);

whereas the legacy driver calls:
pci_scan_root_bus(NULL, sys->busnr, &tangox_pcie_ops, sys, &sys->resources);

So the generic equivalent of
tangox_pcie_read_conf() and tangox_pcie_write_conf()
should be
pci_generic_config_read() and pci_generic_config_write()


int pci_generic_config_read(struct pci_bus *bus, unsigned int devfn,
			    int where, int size, u32 *val)
{
	void __iomem *addr;

	addr = bus->ops->map_bus(bus, devfn, where);
	if (!addr) {
		*val = ~0;
		return PCIBIOS_DEVICE_NOT_FOUND;
	}

	if (size == 1)
		*val = readb(addr);
	else if (size == 2)
		*val = readw(addr);
	else
		*val = readl(addr);

	return PCIBIOS_SUCCESSFUL;
}


Problem might come from the
TANGOX_PCIE_ENABLE_CONFIG_IO() / TANGOX_PCIE_DISABLE_CONFIG_IO()
shenanigans.

Also, I see that the legacy implementation reads 32 bits,
then performs the shift & mask in software. I'm hoping that
the ARM core does the right thing for readb/readw.


Taking a closer look at TANGOX_PCIE_ENABLE_CONFIG_IO()
WR_PCIE_REG32(TANGOX_PCIE_PCI_ADDR_LO_BASE_REG, ((TANGOX_PCIE_ADDR_BASE & 0xff000000) | 0x1))

Register "PCI_ADDR_LO_BASE_REG" is composed of
pci_addr_base_lo[31:28] and pci_conf[0]

  pci_conf : if 1, the access by LBUS is a configuration request. If 0, the access by LBUS is a memory request.
  pci_addr_base_lo : lower part of the PCI address for access by LBUS.

So it looks like I need to fudge something, perhaps override map_bus method?

What is a "memory request", in PCIe lingo?
In other words, when do I need to clear the "pci_conf" bit?
(Maybe firmware can just set it to 1, and be done with it?)

Regards.

Pointers for writing a good PCIe driver

[Mason]

On 17/02/2017 18:50, Bjorn Helgaas wrote:

> On Fri, Feb 17, 2017 at 06:00:53PM +0100, Mason wrote:
>
>> On 17/02/2017 15:38, Bjorn Helgaas wrote:
>>
>>> The native PCI host controller drivers indeed live in drivers/pci/host/
>>>
>>> I don't know anything about your hardware or environment, but I highly
>>> encourage you to use ACPI (drivers/acpi/pci_root.c, with a little bit
>>> of arch support) or generic DT (drivers/pci/host/pci-host-generic.c)
>>> instead of writing a custom host controller driver.
>>>
>>> The native drivers in drivers/pci/host are a huge maintenance hassle
>>> for no real benefit.
>>
>> I'm currently using this DT description:
>> http://lxr.free-electrons.com/source/arch/arm/boot/dts/tango4-common.dtsi
>>
>> The "legacy" driver sits at ~700 lines. It would be awesome to be able
>> to discard all of it, and rely on generic upstream code. But I don't
>> understand how it is possible for a generic driver to support whatever
>> crazy solution some random vendor has come up with?
> 
> You're right that the programming model of the host bridge itself is
> not specified by PCI specs, so it's impossible to have a generic
> driver that can manage it completely by itself.
> 
> If you have firmware that initializes the bridge and tells the OS what
> the windows are (bus numbers, memory space, I/O space) and where the
> PCIe-specified ECAM space is, a generic driver can take it from there.

(Mostly walking myself through the code and documentation,
noting references along the way.)

I took a look at drivers/pci/host/pci-host-generic.c
which is a tiny file, so the magic must be happening elsewhere.
=> drivers/pci/host/pci-host-common.c (for starters)

gen_pci_cfg_cam_bus_ops and pci_generic_ecam_ops have the same
.pci_ops, but different .bus_shift

$ git grep pci-host-cam-generic
Documentation/devicetree/bindings/pci/host-generic-pci.txt:- compatible     : Must be "pci-host-cam-generic" or "pci-host-ecam-generic"
Documentation/devicetree/bindings/pci/host-generic-pci.txt:    compatible = "pci-host-cam-generic"
drivers/pci/host/pci-host-generic.c:    { .compatible = "pci-host-cam-generic",

$ git grep pci-host-ecam-generic
Documentation/devicetree/bindings/pci/host-generic-pci.txt:- compatible     : Must be "pci-host-cam-generic" or "pci-host-ecam-generic"
arch/arm/boot/dts/alpine.dtsi:                  compatible = "pci-host-ecam-generic";
arch/arm64/boot/dts/al/alpine-v2.dtsi:                  compatible = "pci-host-ecam-generic";
arch/arm64/boot/dts/amd/amd-seattle-soc.dtsi:                   compatible = "pci-host-ecam-generic";
arch/arm64/boot/dts/arm/juno-base.dtsi:         compatible = "arm,juno-r1-pcie", "plda,xpressrich3-axi", "pci-host-ecam-generic";
arch/arm64/boot/dts/broadcom/vulcan.dtsi:               compatible = "pci-host-ecam-generic";
drivers/pci/host/pci-host-generic.c:    { .compatible = "pci-host-ecam-generic",

https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/host-generic-pci.txt

> Firmware-initialised PCI host controllers and PCI emulations, such as the
> virtio-pci implementations found in kvmtool and other para-virtualised
> systems, do not require driver support for complexities such as regulator
> and clock management. In fact, the controller may not even require the
> configuration of a control interface by the operating system, instead
> presenting a set of fixed windows describing a subset of IO, Memory and
> Configuration Spaces.

For the only arm32 user, arch/arm/boot/dts/alpine.dtsi:

		/* Internal PCIe Controller */
		pcie-internal at 0xfbc00000 {
			compatible = "pci-host-ecam-generic";
			device_type = "pci";
			#size-cells = <2>;
			#address-cells = <3>;
			#interrupt-cells = <1>;
			reg = <0x0 0xfbc00000 0x0 0x100000>;
			interrupt-map-mask = <0xf800 0 0 7>;
			/* Add legacy interrupts for SATA devices only */
			interrupt-map =	<0x4000 0 0 1 &gic 0 43 4>,
					<0x4800 0 0 1 &gic 0 44 4>;

			/* 32 bit non prefetchable memory space */
			ranges = <0x02000000 0x0 0xfe000000 0x0 0xfe000000 0x0 0x1000000>;

			bus-range = <0x00 0x00>;
			msi-parent = <&msix>;
		};


Compatible string depends on "the layout of configuration space
(CAM vs ECAM respectively)"

https://en.wikipedia.org/wiki/PCI_configuration_space


The register interface to my PCIe controller is:

0x2e000	REGION_0_BASE_REG
0x2e004	REGION_1_BASE_REG
0x2e008	REGION_2_BASE_REG
0x2e00c	REGION_3_BASE_REG
0x2e010	REGION_4_BASE_REG
0x2e014	REGION_5_BASE_REG
0x2e018	REGION_6_BASE_REG
0x2e01c	REGION_7_BASE_REG
0x2e020	DMA_RD_MBUS_ADDR_REG
0x2e024	DMA_RD_ADDR_LO_REG
0x2e028	DMA_RD_ADDR_HI_REG
0x2e02c	DMA_RD_CNT_REG
0x2e030	DMA_WR_MBUS_ADDR_REG
0x2e034	DMA_WR_ADDR_LO_REG
0x2e038	DMA_WR_ADDR_HI_REG
0x2e03c	DMA_WR_CNT_REG
0x2e040	DMA_RD_EN_REG
0x2e044	DMA_WR_EN_REG
0x2e048	PCI_ADDR_LO_BASE_REG
0x2e04c	PCI_ADDR_HI_BASE_REG
0x2e050	PM_REG
0x2e054	MAX_CPL_TIMEOUT_REG
0x2e058	CORE_CONF_0_REG
0x2e05c	CORE_CONF_1_REG
0x2e060	CBA_ADDR_REG
0x2e064	CBA_DATA_REG
0x2e068	DEBUG_0_REG
0x2e06c	DEBUG_1_REG
0x2e070	DEBUG_2_REG
0x2e074	CORE_TEST_OUT_REG
0x2e078	INT_NOT_MSI_REG
0x2e07c	MSI_REG
0x2e080	MSI_0_REG
0x2e084	MSI_1_REG
0x2e088	MSI_2_REG
0x2e08c	MSI_3_REG
0x2e090	MSI_4_REG
0x2e094	MSI_5_REG
0x2e098	MSI_6_REG
0x2e09c	MSI_7_REG
0x2e0a0	MSI_0_MASK_REG
0x2e0a4	MSI_1_MASK_REG
0x2e0a8	MSI_2_MASK_REG
0x2e0ac	MSI_3_MASK_REG
0x2e0b0	MSI_4_MASK_REG
0x2e0b4	MSI_5_MASK_REG
0x2e0b8	MSI_6_MASK_REG
0x2e0bc	MSI_7_MASK_REG
0x2e0c0	PHY_HW_TRAPS_REG
0x2e0c4	COMMON_BIST_0_REG
0x2e0c8	COMMON_BIST_1_REG
0x2e0cc	LANE_0_BIST_REG
0x2e0d0	CUSTOM_CONF_LINK_REG
0x2e0d4	CUSTOM_CONF_LANE_0_REG
0x2e0e0	PLL_SETTING_REG
0x2e0e4	DEBUG_3_REG
0x2e0e8	DEBUG_4_REG
0x2e0ec	DEBUG_5_REG


In the "legacy" driver (for kernel v3.4), I see

static struct pci_ops tangox_pcie_ops = {
	.read	= tangox_pcie_read_conf,
	.write	= tangox_pcie_write_conf,
};

#define TANGOX_PCIE_ENABLE_CONFIG_IO()      WR_PCIE_REG32(TANGOX_PCIE_PCI_ADDR_LO_BASE_REG, ((TANGOX_PCIE_ADDR_BASE & 0xff000000) | 0x1))
#define TANGOX_PCIE_DISABLE_CONFIG_IO()     WR_PCIE_REG32(TANGOX_PCIE_PCI_ADDR_LO_BASE_REG, ((TANGOX_PCIE_ADDR_BASE & 0xff000000) & ~0x1))

static int tangox_pcie_read_conf(struct pci_bus *bus, unsigned int devfn,
                                 int where, int size, u32 *val)
{
    void __iomem *addr;

    TANGOX_PCIE_ENABLE_CONFIG_IO();

    addr =  (void __iomem *)(
            tangox_pcie.regs +
            PCIE_CONF_BUS(bus->number) +
            PCIE_CONF_DEV(PCI_SLOT(devfn)) +
            PCIE_CONF_FUNC(PCI_FUNC(devfn)) +
            PCIE_CONF_REG(where));

    *val = readl( addr );

	if (size == 1)
		*val = (*val >> (8 * (where & 3))) & 0xff;
	else if (size == 2)
		*val = (*val >> (8 * (where & 3))) & 0xffff;

    TANGOX_PCIE_DISABLE_CONFIG_IO();

    return PCIBIOS_SUCCESSFUL;
}


I see that pci-host-common.c calls:
pci_scan_root_bus(dev, cfg->busr.start, &ops->pci_ops, cfg, &resources);

whereas the legacy driver calls:
pci_scan_root_bus(NULL, sys->busnr, &tangox_pcie_ops, sys, &sys->resources);

So the generic equivalent of
tangox_pcie_read_conf() and tangox_pcie_write_conf()
should be
pci_generic_config_read() and pci_generic_config_write()


int pci_generic_config_read(struct pci_bus *bus, unsigned int devfn,
			    int where, int size, u32 *val)
{
	void __iomem *addr;

	addr = bus->ops->map_bus(bus, devfn, where);
	if (!addr) {
		*val = ~0;
		return PCIBIOS_DEVICE_NOT_FOUND;
	}

	if (size == 1)
		*val = readb(addr);
	else if (size == 2)
		*val = readw(addr);
	else
		*val = readl(addr);

	return PCIBIOS_SUCCESSFUL;
}


Problem might come from the
TANGOX_PCIE_ENABLE_CONFIG_IO() / TANGOX_PCIE_DISABLE_CONFIG_IO()
shenanigans.

Also, I see that the legacy implementation reads 32 bits,
then performs the shift & mask in software. I'm hoping that
the ARM core does the right thing for readb/readw.


Taking a closer look at TANGOX_PCIE_ENABLE_CONFIG_IO()
WR_PCIE_REG32(TANGOX_PCIE_PCI_ADDR_LO_BASE_REG, ((TANGOX_PCIE_ADDR_BASE & 0xff000000) | 0x1))

Register "PCI_ADDR_LO_BASE_REG" is composed of
pci_addr_base_lo[31:28] and pci_conf[0]

  pci_conf : if 1, the access by LBUS is a configuration request. If 0, the access by LBUS is a memory request.
  pci_addr_base_lo : lower part of the PCI address for access by LBUS.

So it looks like I need to fudge something, perhaps override map_bus method?

What is a "memory request", in PCIe lingo?
In other words, when do I need to clear the "pci_conf" bit?
(Maybe firmware can just set it to 1, and be done with it?)

Regards.