Re: [PATCH v3 01/21] docs: fpga: add a document for Intel FPGA driver overview

[Alan Tull <atull@kernel.org>]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> Add a document for Intel FPGA driver overview.
>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
>     updated open discussion per current implementation.
>     fixed some typos.
> v3: use FPGA base region as container device instead of fpga-dev class.
>     split common enumeration code from pcie driver to functions exposed by
>     device feature list framework.
>     update FME performance reporting which supports both integrated (iperf/)
>     and discrete (dperf/) FPGA solutions.
> ---
>  Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 261 insertions(+)
>  create mode 100644 Documentation/fpga/intel-fpga.txt
>
> diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> new file mode 100644
> index 0000000..0754733
> --- /dev/null
> +++ b/Documentation/fpga/intel-fpga.txt
> @@ -0,0 +1,261 @@
> +===============================================================================
> +                    Intel FPGA driver Overview

This doesn't look Intel specific to me.  This could all be 'DFL FPGA Framework'

> +-------------------------------------------------------------------------------
> +                Enno Luebbers <enno.luebbers@intel.com>
> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> +                Wu Hao <hao.wu@intel.com>
> +
> +The Intel FPGA driver provides interfaces for userspace applications to
> +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> +with Intel(R) FPGA PCIe based solutions and enables system level management
> +functions such as FPGA reconfiguration, power management, and virtualization.
> +
> +HW Architecture
> +===============
> +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> +The FPGA device memory is organized using a predefined data structure (Device
> +Feature List). Features supported by the particular FPGA device are exposed
> +through these data structures, as illustrated below:
> +
> +  +-------------------------------+  +-------------+
> +  |              PF               |  |     VF      |
> +  +-------------------------------+  +-------------+
> +      ^            ^         ^              ^
> +      |            |         |              |
> ++-----|------------|---------|--------------|-------+
> +|     |            |         |              |       |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|                  ^         ^              ^       |
> +|                  |         |              |       |
> +|              +-------+ +------+       +-------+   |
> +|              |  AFU  | |  AFU |       |  AFU  |   |
> +|              +-------+ +------+       +-------+   |
> +|                                                   |
> +|                 FPGA PCIe Device                  |
> ++---------------------------------------------------+
> +
> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> +used to assign individual accelerators to virtual machines.
> +
> +FME (FPGA Management Engine)
> +============================
> +The FPGA Management Engine performs power and thermal management, error
> +reporting, reconfiguration, performance reporting for integrated and discrete
> +solution, and other infrastructure functions. Each FPGA has one FME, which is
> +always accessed through the physical function (PF).
> +
> +User-space applications can acquire exclusive access to the FME using open(),
> +and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +       Get driver API version (FPGA_GET_API_VERSION)
> +       Check for extensions (FPGA_CHECK_EXTENSION)
> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> +       Release port from PF (FPGA_FME_PORT_RELEASE)
> +       Program bitstream (FPGA_FME_PORT_PR)
> +
> +More functions are exposed through sysfs
> +(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/):

I see that /sys/class/fpga/* has changed to /sys/class/fpga_region/*
now as requested (thanks!).  It looks like it ended up being pretty
straightforward (so far, just diffing this doc with the previous v2).

> +
> +       Read bitstream ID (bitstream_id)
> +       Read bitstream metadata (bitstream_metadata)
> +       Read number of ports (ports_num)
> +       Read socket ID (socket_id)
> +       Read performance counters (iperf/ and dperf/)
> +       Power management (power_mgmt/)
> +       Thermal management (thermal_mgmt/)
> +       Error reporting (errors/)
> +
> +PORT
> +====
> +A port represents the interface between the static FPGA fabric (the "blue
> +bitstream") and a partially reconfigurable region containing an AFU (the "green
> +bitstream"). It controls the communication from SW to the accelerator and
> +exposes features such as reset and debug.
> +
> +A PCIe device may have several ports and each port can be released from PF by
> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> +sysfs interface.
> +
> +AFU
> +===
> +An AFU is attached to a port and exposes a 256k MMIO region to be used for
> +accelerator-specific control registers.
> +
> +User-space applications can acquire exclusive access to an AFU attached to a
> +port by using open() on the port device node, and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +       Get driver API version (FPGA_GET_API_VERSION)
> +       Check for extensions (FPGA_CHECK_EXTENSION)
> +       Get port info (FPGA_PORT_GET_INFO)
> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> +       Reset AFU (FPGA_PORT_RESET)
> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> +
> +User-space applications can also mmap() accelerator MMIO regions.
> +
> +More functions are exposed through sysfs:
> +(/sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/):
> +
> +       Read Accelerator GUID (afu_id)
> +       Error reporting (errors/)
> +
> +Partial Reconfiguration
> +=======================
> +As mentioned above, accelerators can be reconfigured through partial
> +reconfiguration of a green bitstream file (GBS). The green bitstream must have
> +been generated for the exact blue bitstream and targeted reconfigurable region
> +(port) of the FPGA; otherwise, the reconfiguration operation will fail and
> +possibly cause system instability. This compatibility can be checked by
> +comparing the interface ID noted in the GBS header against the interface ID
> +exposed by the FME through sysfs (see above). This check is usually done by
> +user-space before calling the reconfiguration IOCTL.
> +
> +FPGA virtualization
> +===================
> +To enable accessing an accelerator from applications running in a VM, the
> +respective AFU's port needs to be assigned to a VF using the following steps:
> +
> + a) The PF owns all AFU ports by default. Any port that needs to be reassigned
> + to a VF must first be released through the FPGA_FME_PORT_RELEASE ioctl on the
> + FME device.
> +
> + b) Once N ports are released from PF, then user can use command below to
> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> +
> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> +
> + c) Pass through the VFs to VMs
> +
> + d) The AFU under VF is accessible from applications in VM (using the same
> + driver inside the VF).
> +
> +Note that an FME can't be assigned to a VF, thus PR and other management
> +functions are only available via the PF.
> +
> +
> +Driver organization
> +===================
> +
> +  +-------++------++------+             |
> +  | FME   || FME  || FME  |             |
> +  | FPGA  || FPGA || FPGA |             |
> +  |Manager||Bridge||Region|             |
> +  +-------++------++------+             |
> +  +-----------------------+  +--------+ |             +--------+
> +  |          FME          |  |  AFU   | |             |  AFU   |
> +  |         Module        |  | Module | |             | Module |
> +  +-----------------------+  +--------+ |             +--------+
> +        +-----------------------+       |       +-----------------------+
> +        | FPGA Container Device |       |       | FPGA Container Device |
> +        |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
> +        +-----------------------+       |       +-----------------------+
> +          +------------------+          |         +------------------+
> +          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
> +          +------------------+   Host   | Machine +------------------+
> + -------------------------------------- | ------------------------------
> +           +---------------+            |          +---------------+
> +           | PCI PF Device |            |          | PCI VF Device |
> +           +---------------+            |          +---------------+
> +
> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
> +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
> +driver plays an infrastructural role in the driver architecture.  It:
> +
> +       a) locates the Device Feature Lists in PCIE device BAR memory, handles
> +          them and related resources to common interfaces from DFL framework
> +          for enumeration.
> +       b) supports SRIOV.
> +
> +The feature device infrastructure provides common interfaces to create container
> +device (FPGA base region), discover feature devices and their sub features from
> +the given Device Feature Lists, and create platform devices for feature devices
> +with related resources under the container device. It also abstracts operations
> +for sub features and exposes common interfaces to feature device drivers.
> +
> +The FPGA Management Engine (FME) driver is a platform driver which is loaded
> +automatically after FME platform device creation from the PCIE driver. It
> +provides the key features for FPGA management, including:
> +
> +       a) Power and thermal management, error reporting, performance reporting
> +          and other infrastructure functions. Users can access these functions
> +          via sysfs interfaces exposed by FME driver.
> +       b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
> +          bridges and FPGA regions during PR sub feature initialization; Once
> +          it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the
> +          common interface function from FPGA Region to complete the partial
> +          reconfiguration of the bitstream to the given port.
> +       c) Port management for virtualization. The FME driver introduces two
> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
> +          released from the PF, it can be assigned to the VF through the SRIOV
> +          interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
> +          for more details).
> +
> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
> +is probed once the AFU platform device is created. The main function of this
> +module is to provide an interface for userspace applications to access the
> +individual accelerators, including basic reset control on port, AFU MMIO region
> +export, dma buffer mapping service, UMsg notification, and remote debug
> +functions (see above).
> +
> +
> +Device enumeration
> +==================
> +This section introduces how applications enumerate the fpga device from
> +the sysfs hierarchy under /sys/class/fpga_region.
> +
> +In the example below, two Intel(R) FPGA devices are installed in the host. Each
> +fpga device has one FME and two ports (AFUs).
> +
> +FPGA regions are created under /sys/class/fpga_region/
> +
> +       /sys/class/fpga_region/region0
> +       /sys/class/fpga_region/region1
> +       /sys/class/fpga_region/region2
> +       ...
> +
> +Application needs to search each regionX folder, if feature device is found,
> +(e.g "fpga-dfl-port.n" or "fpga-dfl-fme.m" is found), then it's the base
> +fpga region which represents the FPGA device.
> +
> +Each base region has one FME and two ports (AFUs) as child devices:
> +
> +       /sys/class/fpga_region/region0/fpga-dfl-fme.0
> +       /sys/class/fpga_region/region0/fpga-dfl-port.0
> +       /sys/class/fpga_region/region0/fpga-dfl-port.1
> +       ...
> +
> +       /sys/class/fpga_region/region3/fpga-dfl-fme.1
> +       /sys/class/fpga_region/region3/fpga-dfl-port.2
> +       /sys/class/fpga_region/region3/fpga-dfl-port.3
> +       ...
> +
> +In general, the FME/AFU sysfs interfaces are named as follows:
> +
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/
> +
> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
> +ports.
> +
> +The device nodes used for ioctl() or mmap() can be referenced through:
> +
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/dev
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-port.n>/dev
> +
> +Open discussion
> +===============
> +FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
> +user now. In the future, if unified user interfaces for reconfiguration are
> +added, FME driver should switch to them from ioctl interface.
> --
> 1.8.3.1
>