local bus enumeration beyond a PCI device

local bus enumeration beyond a PCI device

[Chris Packham]

Hi,

We've got a custom x86_64 based design that is using an ASIX9100 to 
provide a PCI to local bus bridge. Attached to that local bus is an FPGA 
which mostly provides some GPIOs accessed via registers on the local 
bus. Right now we've got a custom driver that bundles everything 
together so effectively we've got a PCI device that provides GPIOs.

But as things can change based on the FPGA program I'd like some 
flexibility to treat it separately from the PCI bridge. So really I'd 
like to have a PCI device driver for the ASIX9100 that provides a local 
bus controller and a (platform?) driver for the FPGA that provides the 
GPIOs where I can have different compatibles for the different 
implementations.

Then in the ACPI overlay I'd have something like

     Scope (\_SB.PCI0.D0B0)
     {
         Device (ASIX)
         {
             Name (_ADR, 0x0000)

             Device (FPGA)
             {
                         Name (_HID, "PRP0001")
                         Name (_DSD, Package ()
                         {
ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
                                     Package ()
                                     {
                                                 Package () { 
"compatible", "my-platform-driver-for-fpga" },
                                     }
                         })
             }
         }
     }

    Scope(\_SB)
    {
         Device(OTHR)
         {
             GpioIo (Exclusive, PullUp, 0, 0, IoRestrictionInputOnly, 
"\\_SB.PCI0.D0B0.ASIX.FPGA",) { 0 }
         }
    }

Is it even possible to register a host controller for another platform bus?

Thanks,
Chris

Re: local bus enumeration beyond a PCI device

[Bjorn Helgaas]

[+cc Herve]

On Thu, Apr 18, 2024 at 12:24:06AM +0000, Chris Packham wrote:
> Hi,
> 
> We've got a custom x86_64 based design that is using an ASIX9100 to 
> provide a PCI to local bus bridge. Attached to that local bus is an FPGA 
> which mostly provides some GPIOs accessed via registers on the local 
> bus. Right now we've got a custom driver that bundles everything 
> together so effectively we've got a PCI device that provides GPIOs.

What's the local bus?  The ASIX9100 (for which Google doesn't find any
details) would have a PCI interface on the primary (upstream) side.
What's the local bus on the secondary (downstream) side?  Below you
mention "PCI bridge", which normally means both the primary and
secondary sides are PCI buses.

If the local bus is not PCI, I guess the ASIX9100 would look to the OS
like an endpoint, i.e., PCI_HEADER_TYPE_NORMAL, and the ASIX9100
driver would handle any "bridge" functionality completely internally?

Maybe Herve's work at
https://lore.kernel.org/r/20240325153919.199337-1-herve.codina@bootlin.com
would be relevant?

> But as things can change based on the FPGA program I'd like some 
> flexibility to treat it separately from the PCI bridge. So really I'd 
> like to have a PCI device driver for the ASIX9100 that provides a local 
> bus controller and a (platform?) driver for the FPGA that provides the 
> GPIOs where I can have different compatibles for the different 
> implementations.
> 
> Then in the ACPI overlay I'd have something like
> 
>      Scope (\_SB.PCI0.D0B0)
>      {
>          Device (ASIX)
>          {
>              Name (_ADR, 0x0000)
> 
>              Device (FPGA)
>              {
>                          Name (_HID, "PRP0001")
>                          Name (_DSD, Package ()
>                          {
> ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
>                                      Package ()
>                                      {
>                                                  Package () { 
> "compatible", "my-platform-driver-for-fpga" },
>                                      }
>                          })
>              }
>          }
>      }
> 
>     Scope(\_SB)
>     {
>          Device(OTHR)
>          {
>              GpioIo (Exclusive, PullUp, 0, 0, IoRestrictionInputOnly, 
> "\\_SB.PCI0.D0B0.ASIX.FPGA",) { 0 }
>          }
>     }
> 
> Is it even possible to register a host controller for another platform bus?
> 
> Thanks,
> Chris

Re: local bus enumeration beyond a PCI device

[Andy Shevchenko]

Thu, Apr 18, 2024 at 01:45:31PM -0500, Bjorn Helgaas kirjoitti:
> On Thu, Apr 18, 2024 at 12:24:06AM +0000, Chris Packham wrote:

> > We've got a custom x86_64 based design that is using an ASIX9100 to 
> > provide a PCI to local bus bridge. Attached to that local bus is an FPGA 
> > which mostly provides some GPIOs accessed via registers on the local 
> > bus. Right now we've got a custom driver that bundles everything 
> > together so effectively we've got a PCI device that provides GPIOs.
> 
> What's the local bus?  The ASIX9100 (for which Google doesn't find any
> details) would have a PCI interface on the primary (upstream) side.
> What's the local bus on the secondary (downstream) side?  Below you
> mention "PCI bridge", which normally means both the primary and
> secondary sides are PCI buses.
> 
> If the local bus is not PCI, I guess the ASIX9100 would look to the OS
> like an endpoint, i.e., PCI_HEADER_TYPE_NORMAL, and the ASIX9100
> driver would handle any "bridge" functionality completely internally?
> 
> Maybe Herve's work at
> https://lore.kernel.org/r/20240325153919.199337-1-herve.codina@bootlin.com
> would be relevant?

+1 here. Thought the same when seeing the original message from Chris.

> > But as things can change based on the FPGA program I'd like some 
> > flexibility to treat it separately from the PCI bridge. So really I'd 
> > like to have a PCI device driver for the ASIX9100 that provides a local 
> > bus controller and a (platform?) driver for the FPGA that provides the 
> > GPIOs where I can have different compatibles for the different 
> > implementations.
> > 
> > Then in the ACPI overlay I'd have something like
> > 
> >      Scope (\_SB.PCI0.D0B0)
> >      {
> >          Device (ASIX)
> >          {
> >              Name (_ADR, 0x0000)
> > 
> >              Device (FPGA)
> >              {
> >                          Name (_HID, "PRP0001")
> >                          Name (_DSD, Package ()
> >                          {
> > ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
> >                                      Package ()
> >                                      {
> >                                                  Package () { 
> > "compatible", "my-platform-driver-for-fpga" },
> >                                      }
> >                          })
> >              }
> >          }
> >      }
> > 
> >     Scope(\_SB)
> >     {
> >          Device(OTHR)
> >          {
> >              GpioIo (Exclusive, PullUp, 0, 0, IoRestrictionInputOnly, 
> > "\\_SB.PCI0.D0B0.ASIX.FPGA",) { 0 }
> >          }
> >     }
> > 
> > Is it even possible to register a host controller for another platform bus?

-- 
With Best Regards,
Andy Shevchenko

Re: local bus enumeration beyond a PCI device

[Chris Packham]

Hi,

On 19/04/24 06:45, Bjorn Helgaas wrote:
> [+cc Herve]
>
> On Thu, Apr 18, 2024 at 12:24:06AM +0000, Chris Packham wrote:
>> Hi,
>>
>> We've got a custom x86_64 based design that is using an ASIX9100 to
>> provide a PCI to local bus bridge. Attached to that local bus is an FPGA
>> which mostly provides some GPIOs accessed via registers on the local
>> bus. Right now we've got a custom driver that bundles everything
>> together so effectively we've got a PCI device that provides GPIOs.
> What's the local bus?  The ASIX9100 (for which Google doesn't find any
> details) would have a PCI interface on the primary (upstream) side.
> What's the local bus on the secondary (downstream) side?  Below you
> mention "PCI bridge", which normally means both the primary and
> secondary sides are PCI buses.

The ASIX9100 is a multi purpose chip with a few options for providing 
GPIO/I2C/SPI/etc. In our hardware configuration we're using it to 
provide a 16-bit parallel bus which we're using for register access to 
the FPGA. There's an outside chance someone might ask us to hang other 
parallel devices like an SRAM or some NOR Flash off it as well. These 
are all things we've done on other products with more traditional 
embedded SoCs that have a parallel bus interface.

> If the local bus is not PCI, I guess the ASIX9100 would look to the OS
> like an endpoint, i.e., PCI_HEADER_TYPE_NORMAL, and the ASIX9100
> driver would handle any "bridge" functionality completely internally?

I might be using the terminology wrong. Essentially the ASIX9100 is a 
PCI-e endpoint (so a device from the PCI subsystems point of view) but 
that device can be a I2C, SPI or parallel bus controller onto which 
other devices are connected.Hence it's providing a bridge between PCI 
and 16-bit parallel busses.

> Maybe Herve's work at
> https://lore.kernel.org/r/20240325153919.199337-1-herve.codina@bootlin.com
> would be relevant?

I've had a quick look at that series. I'm not 100% sure it'll help. The 
platform I'm dealing with is x86_64 and doesn't have device tree support 
(if someone can say "hey do xyz and you'll get DT on x86" that'd be 
great).But I'll take a closer look at it and at the original "Generate 
device tree node for pci devices" series.

>> But as things can change based on the FPGA program I'd like some
>> flexibility to treat it separately from the PCI bridge. So really I'd
>> like to have a PCI device driver for the ASIX9100 that provides a local
>> bus controller and a (platform?) driver for the FPGA that provides the
>> GPIOs where I can have different compatibles for the different
>> implementations.
>>
>> Then in the ACPI overlay I'd have something like
>>
>>       Scope (\_SB.PCI0.D0B0)
>>       {
>>           Device (ASIX)
>>           {
>>               Name (_ADR, 0x0000)
>>
>>               Device (FPGA)
>>               {
>>                           Name (_HID, "PRP0001")
>>                           Name (_DSD, Package ()
>>                           {
>> ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
>>                                       Package ()
>>                                       {
>>                                                   Package () {
>> "compatible", "my-platform-driver-for-fpga" },
>>                                       }
>>                           })
>>               }
>>           }
>>       }
>>
>>      Scope(\_SB)
>>      {
>>           Device(OTHR)
>>           {
>>               GpioIo (Exclusive, PullUp, 0, 0, IoRestrictionInputOnly,
>> "\\_SB.PCI0.D0B0.ASIX.FPGA",) { 0 }
>>           }
>>      }
>>
>> Is it even possible to register a host controller for another platform bus?
>>
>> Thanks,
>> Chris

Re: local bus enumeration beyond a PCI device

[Herve Codina]

Hi Chris,

On Tue, 23 Apr 2024 01:05:21 +0000
Chris Packham <Chris.Packham@alliedtelesis.co.nz> wrote:

> Hi,
> 
> On 19/04/24 06:45, Bjorn Helgaas wrote:
> > [+cc Herve]
> >
> > On Thu, Apr 18, 2024 at 12:24:06AM +0000, Chris Packham wrote:  
> >> Hi,
> >>
> >> We've got a custom x86_64 based design that is using an ASIX9100 to
> >> provide a PCI to local bus bridge. Attached to that local bus is an FPGA
> >> which mostly provides some GPIOs accessed via registers on the local
> >> bus. Right now we've got a custom driver that bundles everything
> >> together so effectively we've got a PCI device that provides GPIOs.  
> > What's the local bus?  The ASIX9100 (for which Google doesn't find any
> > details) would have a PCI interface on the primary (upstream) side.
> > What's the local bus on the secondary (downstream) side?  Below you
> > mention "PCI bridge", which normally means both the primary and
> > secondary sides are PCI buses.  
> 
> The ASIX9100 is a multi purpose chip with a few options for providing 
> GPIO/I2C/SPI/etc. In our hardware configuration we're using it to 
> provide a 16-bit parallel bus which we're using for register access to 
> the FPGA. There's an outside chance someone might ask us to hang other 
> parallel devices like an SRAM or some NOR Flash off it as well. These 
> are all things we've done on other products with more traditional 
> embedded SoCs that have a parallel bus interface.
> 
> > If the local bus is not PCI, I guess the ASIX9100 would look to the OS
> > like an endpoint, i.e., PCI_HEADER_TYPE_NORMAL, and the ASIX9100
> > driver would handle any "bridge" functionality completely internally?  
> 
> I might be using the terminology wrong. Essentially the ASIX9100 is a 
> PCI-e endpoint (so a device from the PCI subsystems point of view) but 
> that device can be a I2C, SPI or parallel bus controller onto which 
> other devices are connected.Hence it's providing a bridge between PCI 
> and 16-bit parallel busses.
> 
> > Maybe Herve's work at
> > https://lore.kernel.org/r/20240325153919.199337-1-herve.codina@bootlin.com
> > would be relevant?  
> 
> I've had a quick look at that series. I'm not 100% sure it'll help. The 
> platform I'm dealing with is x86_64 and doesn't have device tree support 
> (if someone can say "hey do xyz and you'll get DT on x86" that'd be 
> great).But I'll take a closer look at it and at the original "Generate 
> device tree node for pci devices" series.

You can have DT support on x86.
CONFIG_OF=y
CONFIG_PCI_DYNAMIC_OF_NODES = y

I use a PCIe board from Microchip (Lan966x) and I describe the internal board
IPs using a DT overlay.
My PCI driver loads the overlay at probe() and call of_platform_default_populate()
to instantiate the devices.

My DT overlay contains nodes for:
- clock controller
- reset
- i2c bus
- pinctrl and gpio
- Network switch
- MDIO controller

I have it working on x86 system.
What is mainly missing upstream for x86 system is the DT node creation for the
host PCI bridge (root complex). Indeed, on x86 the description comes from ACPI
and a DT description of this root bridge is needed.
Got a draft code locally (not sent upstream yet) that creates this first bridge
on ACPI based systems.

PCI-to-PCI bridges and PCI devices are already handled:
  https://elixir.bootlin.com/linux/v6.9-rc5/source/drivers/pci/of.c#L625
At runtime, based on the PCI enumeration, corresponding DT nodes are created.

Best regards,
Hervé

> 
> >> But as things can change based on the FPGA program I'd like some
> >> flexibility to treat it separately from the PCI bridge. So really I'd
> >> like to have a PCI device driver for the ASIX9100 that provides a local
> >> bus controller and a (platform?) driver for the FPGA that provides the
> >> GPIOs where I can have different compatibles for the different
> >> implementations.
> >>
> >> Then in the ACPI overlay I'd have something like
> >>
> >>       Scope (\_SB.PCI0.D0B0)
> >>       {
> >>           Device (ASIX)
> >>           {
> >>               Name (_ADR, 0x0000)
> >>
> >>               Device (FPGA)
> >>               {
> >>                           Name (_HID, "PRP0001")
> >>                           Name (_DSD, Package ()
> >>                           {
> >> ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
> >>                                       Package ()
> >>                                       {
> >>                                                   Package () {
> >> "compatible", "my-platform-driver-for-fpga" },
> >>                                       }
> >>                           })
> >>               }
> >>           }
> >>       }
> >>
> >>      Scope(\_SB)
> >>      {
> >>           Device(OTHR)
> >>           {
> >>               GpioIo (Exclusive, PullUp, 0, 0, IoRestrictionInputOnly,
> >> "\\_SB.PCI0.D0B0.ASIX.FPGA",) { 0 }
> >>           }
> >>      }
> >>
> >> Is it even possible to register a host controller for another platform bus?

With DT overlay, you can register any devices available on your PCI board.
You just need to describe them in the overlay and be sure that a driver exist to
handle them based on their compatible property.

Also, this solution was designed to work on both DT and ACPI based systems.
My Microchip board works on x86 and ARM.

Best regards,
Hervé

Re: local bus enumeration beyond a PCI device

[Andy Shevchenko]

Thu, Apr 18, 2024 at 12:24:06AM +0000, Chris Packham kirjoitti:
> Hi,
> 
> We've got a custom x86_64 based design that is using an ASIX9100 to 
> provide a PCI to local bus bridge. Attached to that local bus is an FPGA 
> which mostly provides some GPIOs accessed via registers on the local 
> bus. Right now we've got a custom driver that bundles everything 
> together so effectively we've got a PCI device that provides GPIOs.
> 
> But as things can change based on the FPGA program I'd like some 
> flexibility to treat it separately from the PCI bridge.

Why? AFAIU the architecture, you have an FPGA with a real PCI bridge / switch,
no? If it's the case, the software shouldn't care if the respective IP comes
from FPGA or SoC.

> So really I'd 
> like to have a PCI device driver for the ASIX9100 that provides a local 
> bus controller and a (platform?) driver for the FPGA that provides the 
> GPIOs where I can have different compatibles for the different 
> implementations.
> 
> Then in the ACPI overlay I'd have something like
> 
>      Scope (\_SB.PCI0.D0B0)
>      {
>          Device (ASIX)
>          {
>              Name (_ADR, 0x0000)
> 
>              Device (FPGA)
>              {
>                          Name (_HID, "PRP0001")
>                          Name (_DSD, Package ()
>                          {
> ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
>                                      Package ()
>                                      {
>                                                  Package () { 
> "compatible", "my-platform-driver-for-fpga" },
>                                      }
>                          })
>              }
>          }
>      }
> 
>     Scope(\_SB)
>     {
>          Device(OTHR)
>          {
>              GpioIo (Exclusive, PullUp, 0, 0, IoRestrictionInputOnly, 
> "\\_SB.PCI0.D0B0.ASIX.FPGA",) { 0 }
>          }
>     }
> 
> Is it even possible to register a host controller for another platform bus?

AFAIK there is an FPGA framework in the kernel and the idea is that each FPGA
configuration provides a complimentary DT to describe the hardware setup. As
Bjorn Cc'ed this to Herve you may get the answer on what's going on there much
better as I'm not involved in the development of that topic.

-- 
With Best Regards,
Andy Shevchenko

Re: local bus enumeration beyond a PCI device

[Chris Packham]

On 22/04/24 19:59, Andy Shevchenko wrote:
> Thu, Apr 18, 2024 at 12:24:06AM +0000, Chris Packham kirjoitti:
>> Hi,
>>
>> We've got a custom x86_64 based design that is using an ASIX9100 to
>> provide a PCI to local bus bridge. Attached to that local bus is an FPGA
>> which mostly provides some GPIOs accessed via registers on the local
>> bus. Right now we've got a custom driver that bundles everything
>> together so effectively we've got a PCI device that provides GPIOs.
>>
>> But as things can change based on the FPGA program I'd like some
>> flexibility to treat it separately from the PCI bridge.
> Why? AFAIU the architecture, you have an FPGA with a real PCI bridge / switch,
> no? If it's the case, the software shouldn't care if the respective IP comes
> from FPGA or SoC.

No the FPGA has a parallel interface. We did try to push the HW 
designers into using a FPGA with a proper PCI-e interface but they 
sighted reasons of cost and schedule so they ended up adding the 
ASIX9100 so they could re-use a lot of the design they had from an older 
product.

>
>> So really I'd
>> like to have a PCI device driver for the ASIX9100 that provides a local
>> bus controller and a (platform?) driver for the FPGA that provides the
>> GPIOs where I can have different compatibles for the different
>> implementations.
>>
>> Then in the ACPI overlay I'd have something like
>>
>>       Scope (\_SB.PCI0.D0B0)
>>       {
>>           Device (ASIX)
>>           {
>>               Name (_ADR, 0x0000)
>>
>>               Device (FPGA)
>>               {
>>                           Name (_HID, "PRP0001")
>>                           Name (_DSD, Package ()
>>                           {
>> ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
>>                                       Package ()
>>                                       {
>>                                                   Package () {
>> "compatible", "my-platform-driver-for-fpga" },
>>                                       }
>>                           })
>>               }
>>           }
>>       }
>>
>>      Scope(\_SB)
>>      {
>>           Device(OTHR)
>>           {
>>               GpioIo (Exclusive, PullUp, 0, 0, IoRestrictionInputOnly,
>> "\\_SB.PCI0.D0B0.ASIX.FPGA",) { 0 }
>>           }
>>      }
>>
>> Is it even possible to register a host controller for another platform bus?
> AFAIK there is an FPGA framework in the kernel and the idea is that each FPGA
> configuration provides a complimentary DT to describe the hardware setup. As
> Bjorn Cc'ed this to Herve you may get the answer on what's going on there much
> better as I'm not involved in the development of that topic.
I do need to have a better look at the FPGA framework in the kernel. 
We've mostly done without it on other products.