RE: [PATCH v1 00/12] add FPGA hotplug manager driver

["Zhang, Tianfei" <tianfei.zhang@intel.com>]

> -----Original Message-----
> From: Pali Rohár <pali@kernel.org>
> Sent: Thursday, January 19, 2023 4:06 PM
> To: Zhang, Tianfei <tianfei.zhang@intel.com>
> Cc: bhelgaas@google.com; linux-pci@vger.kernel.org; linux-fpga@vger.kernel.org;
> lukas@wunner.de; kabel@kernel.org; mani@kernel.org; mdf@kernel.org; Wu, Hao
> <hao.wu@intel.com>; Xu, Yilun <yilun.xu@intel.com>; Rix, Tom <trix@redhat.com>;
> jgg@ziepe.ca; Weiny, Ira <ira.weiny@intel.com>;
> andriy.shevchenko@linux.intel.com; Williams, Dan J <dan.j.williams@intel.com>;
> keescook@chromium.org; rafael@kernel.org; Weight, Russell H
> <russell.h.weight@intel.com>; corbet@lwn.net; linux-doc@vger.kernel.org;
> ilpo.jarvinen@linux.intel.com; lee@kernel.org; gregkh@linuxfoundation.org;
> matthew.gerlach@linux.intel.com
> Subject: Re: [PATCH v1 00/12] add FPGA hotplug manager driver
> 
> Hello!
> 
> On Wednesday 18 January 2023 20:35:50 Tianfei Zhang wrote:
> > This patchset introduces the FPGA hotplug manager (fpgahp) driver
> > which has been verified on the Intel N3000 card.
> >
> > When a PCIe-based FPGA card is reprogrammed, it temporarily disappears
> > from the PCIe bus. This needs to be managed to avoid PCIe errors and
> > to reprobe the device after reprogramming.
> >
> > To change the FPGA image, the kernel burns a new image into the flash
> > on the card, and then triggers the card BMC to load the new image into FPGA.
> > A new FPGA hotplug manager driver is introduced that leverages the
> > PCIe hotplug framework to trigger and manage the update of the FPGA
> > image, including the disappearance and reappearance of the card on the PCIe bus.
> > The fpgahp driver uses APIs from the pciehp driver.
> 
> Just I'm thinking about one thing. PCIe cards can support PCIe hotplug mechanism
> (via standard PCIe capabilities). So what would happen when FPGA based PCIe card is
> also hotplug-able? Will be there two PCI hotplug drivers/devices (one fpgahp and
> one pciehp)? Or just one and which?

For our Intel PAC N3000 and N6000 FPGA card, there are not support PCIe hotplug capability from hardware side now,
but from software perspective, the process of FPGA image load is very similar with PCIe hotplug, like removing all of devices under 
PCIe bridge, re-scan the PCIe device under the bridge, so we are looking for the PCIe hotplug framework and APIs from pciehp 
driver to manager this process, and reduce some duplicate code.

> 
> > Two new operation
> > callbacks are defined in hotplug_slot_ops:
> >
> >   - available_images: Optional: available FPGA images
> >   - image_load: Optional: trigger the FPGA to load a new image
> >
> >
> > The process of reprogramming an FPGA card begins by removing all
> > devices associated with the card that are not required for the
> > reprogramming of the card. This includes PCIe devices (PFs and VFs)
> > associated with the card as well as any other types of devices
> > (platform, etc.) defined within the FPGA. The remaining devices are referred to
> here as "reserved" devices.
> > After triggering the update of the FPGA card, the reserved devices are
> > also removed.
> >
> > The complete process for reprogramming the FPGA are:
> >     1. remove all PFs and VFs except for PF0 (reserved).
> >     2. remove all non-reserved devices of PF0.
> >     3. trigger FPGA card to do the image update.
> >     4. disable the link of the hotplug bridge.
> >     5. remove all reserved devices under hotplug bridge.
> >     6. wait for image reload done via BMC, e.g. 10s.
> >     7. re-enable the link of hotplug bridge
> >     8. enumerate PCI devices below the hotplug bridge
> >
> > usage example:
> > [root@localhost]# cd /sys/bus/pci/slot/X-X/
> >
> > Get the available images.
> > [root@localhost 2-1]# cat available_images bmc_factory bmc_user
> > retimer_fw
> >
> > Load the request images for FPGA Card, for example load the BMC user image:
> > [root@localhost 2-1]# echo bmc_user > image_load
> >
> > Tianfei Zhang (12):
> >   PCI: hotplug: add new callbacks on hotplug_slot_ops
> >   PCI: hotplug: expose APIs from pciehp driver
> >   PCI: hotplug: add and expose link disable API
> >   PCI: hotplug: add FPGA PCI hotplug manager driver
> >   fpga: dfl: register dfl-pci device into fpgahph driver
> >   driver core: expose device_is_ancestor() API
> >   PCI: hotplug: add register/unregister function for BMC device
> >   fpga: m10bmc-sec: register BMC device into fpgahp driver
> >   fpga: dfl: remove non-reserved devices
> >   PCI: hotplug: implement the hotplug_slot_ops callback for fpgahp
> >   fpga: m10bmc-sec: add m10bmc_sec_retimer_load callback
> >   Documentation: fpga: add description of fpgahp driver
> >
> >  Documentation/ABI/testing/sysfs-driver-fpgahp |  21 +
> >  Documentation/fpga/fpgahp.rst                 |  29 +
> >  Documentation/fpga/index.rst                  |   1 +
> >  MAINTAINERS                                   |  10 +
> >  drivers/base/core.c                           |   3 +-
> >  drivers/fpga/Kconfig                          |   2 +
> >  drivers/fpga/dfl-pci.c                        |  95 +++-
> >  drivers/fpga/dfl.c                            |  58 ++
> >  drivers/fpga/dfl.h                            |   4 +
> >  drivers/fpga/intel-m10-bmc-sec-update.c       | 246 ++++++++
> >  drivers/pci/hotplug/Kconfig                   |  14 +
> >  drivers/pci/hotplug/Makefile                  |   1 +
> >  drivers/pci/hotplug/fpgahp.c                  | 526 ++++++++++++++++++
> >  drivers/pci/hotplug/pci_hotplug_core.c        |  88 +++
> >  drivers/pci/hotplug/pciehp.h                  |   3 +
> >  drivers/pci/hotplug/pciehp_hpc.c              |  11 +-
> >  drivers/pci/hotplug/pciehp_pci.c              |   2 +
> >  include/linux/device.h                        |   1 +
> >  include/linux/fpga/fpgahp_manager.h           | 100 ++++
> >  include/linux/mfd/intel-m10-bmc.h             |  31 ++
> >  include/linux/pci_hotplug.h                   |   5 +
> >  21 files changed, 1243 insertions(+), 8 deletions(-)  create mode
> > 100644 Documentation/ABI/testing/sysfs-driver-fpgahp
> >  create mode 100644 Documentation/fpga/fpgahp.rst  create mode 100644
> > drivers/pci/hotplug/fpgahp.c  create mode 100644
> > include/linux/fpga/fpgahp_manager.h
> >
> >
> > base-commit: 5dc4c995db9eb45f6373a956eb1f69460e69e6d4
> > --
> > 2.38.1
> >