From: Tom Rix <trix@redhat.com>
To: Frank Rowand <frowand.list@gmail.com>,
Lizhi Hou <lizhi.hou@amd.com>,
linux-pci@vger.kernel.org, devicetree@vger.kernel.org,
linux-kernel@vger.kernel.org, robh@kernel.org,
helgaas@kernel.org
Cc: clement.leger@bootlin.com, max.zhen@amd.com,
sonal.santan@amd.com, larry.liu@amd.com, brian.xu@amd.com,
stefano.stabellini@xilinx.com
Subject: Re: [PATCH RFC 0/2] Generate device tree node for pci devices
Date: Sat, 17 Sep 2022 11:36:24 -0700 [thread overview]
Message-ID: <7393f2f3-3af5-edf9-4afb-8acc0a8db1e7@redhat.com> (raw)
In-Reply-To: <f042a8d7-fa2b-3e75-a2f2-6e6ba28a9b38@gmail.com>
Frank,
On 9/16/22 7:23 PM, Frank Rowand wrote:
> On 9/13/22 16:02, Lizhi Hou wrote:
>> On 9/13/22 10:41, Frank Rowand wrote:
>>> On 9/13/22 12:10, Lizhi Hou wrote:
>>>> On 9/13/22 00:00, Frank Rowand wrote:
>>>>> On 8/29/22 16:43, Lizhi Hou wrote:
>>>>>> This patch series introduces OF overlay support for PCI devices which
>>>>>> primarily addresses two use cases. First, it provides a data driven method
>>>>>> to describe hardware peripherals that are present in a PCI endpoint and
>>>>>> hence can be accessed by the PCI host. An example device is Xilinx/AMD
>>>>>> Alveo PCIe accelerators. Second, it allows reuse of a OF compatible
>>>>>> driver -- often used in SoC platforms -- in a PCI host based system. An
>>>>>> example device is Microchip LAN9662 Ethernet Controller.
>>>>>>
>>>>>> This patch series consolidates previous efforts to define such an
>>>>>> infrastructure:
>>>>>> https://lore.kernel.org/lkml/20220305052304.726050-1-lizhi.hou@xilinx.com/
>>>>>> https://lore.kernel.org/lkml/20220427094502.456111-1-clement.leger@bootlin.com/
>>>>>>
>>>>>> Normally, the PCI core discovers PCI devices and their BARs using the
>>>>>> PCI enumeration process. However, the process does not provide a way to
>>>>>> discover the hardware peripherals that are present in a PCI device, and
>>>>>> which can be accessed through the PCI BARs. Also, the enumeration process
>>>>>> does not provide a way to associate MSI-X vectors of a PCI device with the
>>>>>> hardware peripherals that are present in the device. PCI device drivers
>>>>>> often use header files to describe the hardware peripherals and their
>>>>>> resources as there is no standard data driven way to do so. This patch
>>>>>> series proposes to use flattened device tree blob to describe the
>>>>>> peripherals in a data driven way. Based on previous discussion, using
>>>>>> device tree overlay is the best way to unflatten the blob and populate
>>>>>> platform devices. To use device tree overlay, there are three obvious
>>>>>> problems that need to be resolved.
>>>>>>
>>>>>> First, we need to create a base tree for non-DT system such as x86_64. A
>>>>>> patch series has been submitted for this:
>>>>>> https://lore.kernel.org/lkml/20220624034327.2542112-1-frowand.list@gmail.com/
>>>>>> https://lore.kernel.org/lkml/20220216050056.311496-1-lizhi.hou@xilinx.com/
>>>>>>
>>>>>> Second, a device tree node corresponding to the PCI endpoint is required
>>>>>> for overlaying the flattened device tree blob for that PCI endpoint.
>>>>>> Because PCI is a self-discoverable bus, a device tree node is usually not
>>>>>> created for PCI devices. This series adds support to generate a device
>>>>>> tree node for a PCI device which advertises itself using PCI quirks
>>>>>> infrastructure.
>>>>>>
>>>>>> Third, we need to generate device tree nodes for PCI bridges since a child
>>>>>> PCI endpoint may choose to have a device tree node created.
>>>>>>
>>>>>> This patch series is made up of two patches.
>>>>>>
>>>>>> The first patch is adding OF interface to allocate an OF node. It is copied
>>>>>> from:
>>>>>> https://lore.kernel.org/lkml/20220620104123.341054-5-clement.leger@bootlin.com/
>>>>>>
>>>>>> The second patch introduces a kernel option, CONFIG_PCI_OF. When the option
>>>>>> is turned on, the kernel will generate device tree nodes for all PCI
>>>>>> bridges unconditionally. The patch also shows how to use the PCI quirks
>>>>>> infrastructure, DECLARE_PCI_FIXUP_FINAL to generate a device tree node for
>>>>>> a device. Specifically, the patch generates a device tree node for Xilinx
>>>>>> Alveo U50 PCIe accelerator device. The generated device tree nodes do not
>>>>>> have any property. Future patches will add the necessary properties.
>>>>>>
>>>>>> Clément Léger (1):
>>>>>> of: dynamic: add of_node_alloc()
>>>>>>
>>>>>> Lizhi Hou (1):
>>>>>> pci: create device tree node for selected devices
>>>>>>
>>>>>> drivers/of/dynamic.c | 50 +++++++++++++----
>>>>>> drivers/pci/Kconfig | 11 ++++
>>>>>> drivers/pci/bus.c | 2 +
>>>>>> drivers/pci/msi/irqdomain.c | 6 +-
>>>>>> drivers/pci/of.c | 106 ++++++++++++++++++++++++++++++++++++
>>>>>> drivers/pci/pci-driver.c | 3 +-
>>>>>> drivers/pci/pci.h | 16 ++++++
>>>>>> drivers/pci/quirks.c | 11 ++++
>>>>>> drivers/pci/remove.c | 1 +
>>>>>> include/linux/of.h | 7 +++
>>>>>> 10 files changed, 200 insertions(+), 13 deletions(-)
>>>>>>
>>>>> The patch description leaves out the most important piece of information.
>>>>>
>>>>> The device located at the PCI endpoint is implemented via FPGA
>>>>> - which is programmed after Linux boots (or somewhere late in the boot process)
>>>>> - (A) and thus can not be described by static data available pre-boot because
>>>>> it is dynamic (and the FPGA program will often change while the Linux
>>>>> kernel is already booted
>>>>> - (B) can be described by static data available pre-boot because the FPGA
>>>>> program will always be the same for this device on this system
>>>>>
>>>>> I am not positive what part of what I wrote above is correct and would appreciate
>>>>> some confirmation of what is correct or incorrect.
>>>> There are 2 series devices rely on this patch:
>>>>
>>>> 1) Xilinx Alveo Accelerator cards (FPGA based device)
>>>>
>>>> 2) lan9662 PCIe card
>>>>
>>>> please see: https://lore.kernel.org/lkml/20220427094502.456111-1-clement.leger@bootlin.com/
>>> Thanks. Please include this information in future versions of the patch series.
>>>
>>> For device 2 I have strongly recommended using pre-boot apply of the overlay to the base
>>> device tree. I realize that this suggestion is only a partial solution if one wants to
>>> use hotplug to change system configuration (as opposed to using hotplug only to replace
>>> an existing device (eg a broken device) with another instance of the same device). I
>>> also realize that this increased the system administration overhead. On the other hand
>>> an overlay based solution is likely to be fragile and possibly flaky.
>> Can you clarify the pre-boot apply approach? How will it work for PCI devices?
>>>> For Xilinx Alveo device, it is (A). The FPGA partitions can be programmed dynamically after boot.
>>> I looked at the Xilinx Alveo web page, and there are a variety of types of Alveo cards
>>> available. So the answer to my next question may vary by type of card.
>>>
>>> Is it expected that the fpga program on a given card will change frequently (eg multiple
>>> times per day), where the changed program results in a new device that would require a
>>> different hardware description in the device tree?
>> Different images may be loaded to a FPGA partition several times a
>> day. The PCI topology (Device IDs, BARs, MSIx, etc) does not change.
>> New IPs may appear (and old IPs may disappear) on the BARs when a new
>> image is loaded. We would like to use flattened device tree to
>> describe the IPs on the BARs.
> That was kind of a non-answer. I know that images _may_ change at
> some frequency. I was trying to get a sense of whether the images
> were _likely_ to be changing on a frequent basis for these types
> of boards, or whether frequent image changes are likely to be a
> rare edge use case.
>
> If there is a good design for the 99.999% use case that does not
> support the 0.001% use case then it may be better to not create
> an inferior design that also supports the 0.001% use case.
>
> I hope that gives a better idea of the reason why I was asking the
> question and how the answer could impact design and implementation
> decisions.
>
> As a point of reference, some other fpga users have indicated a
> desire to change images many times per second. The current driver
> and overlay architecture did not seem to me to be a good match to
> that use case (depending on the definition of "many").
I would rather we cover 99.999% now.
My understanding is that the subdevices are flexible but fairly static
and the frequency Lizhi mentions would cover development uses.
In production I would expect the image to change about once a year with
the same order of magnitude as firmware.
Can you point me to a reference of a user case with high frequency
images changing that also depends on pci io device changing?
Tom
> -Frank
>
>> Thanks,
>>
>> Lizhi
>>
>>> Or is the fpga program expected to change on an infrequent basis (eg monthly, quarterly,
>>> annually), in the same way as device firmware and operating systems are updated on a regular
>>> basis for bug fixes and new functionality?
>>>
>>>
>>>> Thanks,
>>>>
>>>> Lzhi
>>>>
>>>>> -Frank
next prev parent reply other threads:[~2022-09-17 18:36 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-29 21:43 [PATCH RFC 0/2] Generate device tree node for pci devices Lizhi Hou
2022-08-29 21:43 ` [PATCH RFC 1/2] of: dynamic: add of_node_alloc() Lizhi Hou
2022-09-16 23:15 ` Frank Rowand
2022-08-29 21:43 ` [PATCH RFC 2/2] pci: create device tree node for selected devices Lizhi Hou
2022-09-02 18:54 ` Rob Herring
2022-09-12 6:33 ` Frank Rowand
2022-09-13 7:03 ` Frank Rowand
2022-09-16 23:20 ` Frank Rowand
2022-09-13 5:49 ` Lizhi Hou
2022-09-02 20:43 ` [PATCH RFC 0/2] Generate device tree node for pci devices Bjorn Helgaas
2022-09-09 23:06 ` Lizhi Hou
2022-09-13 7:00 ` Frank Rowand
2022-09-13 17:10 ` Lizhi Hou
2022-09-13 17:41 ` Frank Rowand
2022-09-13 21:02 ` Lizhi Hou
2022-09-17 2:23 ` Frank Rowand
2022-09-17 18:36 ` Tom Rix [this message]
2022-09-20 3:12 ` Frank Rowand
2022-09-26 3:03 ` Sonal Santan
2022-10-14 21:25 ` Frank Rowand
2022-10-10 8:42 ` [PATCH RFC 0/2] Generate device tree node for pci devicesgain, Clément Léger
2022-10-13 6:05 ` Frank Rowand
2022-10-13 8:02 ` Clément Léger
2022-10-13 17:28 ` Frank Rowand
2022-10-14 17:33 ` Rob Herring
2022-10-14 18:52 ` Frank Rowand
2022-10-17 7:18 ` Clément Léger
2022-10-26 21:20 ` Sonal Santan
2022-09-14 13:35 ` [PATCH RFC 0/2] Generate device tree node for pci devices Jeremi Piotrowski
2022-09-14 18:08 ` Rob Herring
2022-09-16 23:15 ` Frank Rowand
2022-09-26 22:44 ` Rob Herring
2022-09-30 19:29 ` Sonal Santan
2022-10-06 15:10 ` Rob Herring
2022-10-07 22:45 ` Sonal Santan
2022-10-10 8:58 ` Clément Léger
2022-10-13 6:08 ` Frank Rowand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7393f2f3-3af5-edf9-4afb-8acc0a8db1e7@redhat.com \
--to=trix@redhat.com \
--cc=brian.xu@amd.com \
--cc=clement.leger@bootlin.com \
--cc=devicetree@vger.kernel.org \
--cc=frowand.list@gmail.com \
--cc=helgaas@kernel.org \
--cc=larry.liu@amd.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=lizhi.hou@amd.com \
--cc=max.zhen@amd.com \
--cc=robh@kernel.org \
--cc=sonal.santan@amd.com \
--cc=stefano.stabellini@xilinx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).