From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751418AbdK0GwS (ORCPT ); Mon, 27 Nov 2017 01:52:18 -0500 Received: from mga14.intel.com ([192.55.52.115]:50513 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751350AbdK0GwO (ORCPT ); Mon, 27 Nov 2017 01:52:14 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.44,463,1505804400"; d="scan'208";a="178124993" From: Wu Hao To: atull@kernel.org, mdf@kernel.org, linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org Cc: linux-api@vger.kernel.org, luwei.kang@intel.com, yi.z.zhang@intel.com, hao.wu@intel.com, Enno Luebbers , Xiao Guangrong Subject: [PATCH v3 01/21] docs: fpga: add a document for Intel FPGA driver overview Date: Mon, 27 Nov 2017 14:42:08 +0800 Message-Id: <1511764948-20972-2-git-send-email-hao.wu@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1511764948-20972-1-git-send-email-hao.wu@intel.com> References: <1511764948-20972-1-git-send-email-hao.wu@intel.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add a document for Intel FPGA driver overview. Signed-off-by: Enno Luebbers Signed-off-by: Xiao Guangrong Signed-off-by: Wu Hao ---- v2: added FME fpga-mgr/bridge/region platform driver to driver organization. updated open discussion per current implementation. fixed some typos. v3: use FPGA base region as container device instead of fpga-dev class. split common enumeration code from pcie driver to functions exposed by device feature list framework. update FME performance reporting which supports both integrated (iperf/) and discrete (dperf/) FPGA solutions. --- Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 261 insertions(+) create mode 100644 Documentation/fpga/intel-fpga.txt diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt new file mode 100644 index 0000000..0754733 --- /dev/null +++ b/Documentation/fpga/intel-fpga.txt @@ -0,0 +1,261 @@ +=============================================================================== + Intel FPGA driver Overview +------------------------------------------------------------------------------- + Enno Luebbers + Xiao Guangrong + Wu Hao + +The Intel FPGA driver provides interfaces for userspace applications to +configure, enumerate, open, and access FPGA accelerators on platforms equipped +with Intel(R) FPGA PCIe based solutions and enables system level management +functions such as FPGA reconfiguration, power management, and virtualization. + +HW Architecture +=============== +From the OS's point of view, the FPGA hardware appears as a regular PCIe device. +The FPGA device memory is organized using a predefined data structure (Device +Feature List). Features supported by the particular FPGA device are exposed +through these data structures, as illustrated below: + + +-------------------------------+ +-------------+ + | PF | | VF | + +-------------------------------+ +-------------+ + ^ ^ ^ ^ + | | | | ++-----|------------|---------|--------------|-------+ +| | | | | | +| +-----+ +-------+ +-------+ +-------+ | +| | FME | | Port0 | | Port1 | | Port2 | | +| +-----+ +-------+ +-------+ +-------+ | +| ^ ^ ^ | +| | | | | +| +-------+ +------+ +-------+ | +| | AFU | | AFU | | AFU | | +| +-------+ +------+ +-------+ | +| | +| FPGA PCIe Device | ++---------------------------------------------------+ + +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be +used to assign individual accelerators to virtual machines. + +FME (FPGA Management Engine) +============================ +The FPGA Management Engine performs power and thermal management, error +reporting, reconfiguration, performance reporting for integrated and discrete +solution, and other infrastructure functions. Each FPGA has one FME, which is +always accessed through the physical function (PF). + +User-space applications can acquire exclusive access to the FME using open(), +and release it using close(). + +The following functions are exposed through ioctls: + + Get driver API version (FPGA_GET_API_VERSION) + Check for extensions (FPGA_CHECK_EXTENSION) + Assign port to PF (FPGA_FME_PORT_ASSIGN) + Release port from PF (FPGA_FME_PORT_RELEASE) + Program bitstream (FPGA_FME_PORT_PR) + +More functions are exposed through sysfs +(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/): + + Read bitstream ID (bitstream_id) + Read bitstream metadata (bitstream_metadata) + Read number of ports (ports_num) + Read socket ID (socket_id) + Read performance counters (iperf/ and dperf/) + Power management (power_mgmt/) + Thermal management (thermal_mgmt/) + Error reporting (errors/) + +PORT +==== +A port represents the interface between the static FPGA fabric (the "blue +bitstream") and a partially reconfigurable region containing an AFU (the "green +bitstream"). It controls the communication from SW to the accelerator and +exposes features such as reset and debug. + +A PCIe device may have several ports and each port can be released from PF by +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov +sysfs interface. + +AFU +=== +An AFU is attached to a port and exposes a 256k MMIO region to be used for +accelerator-specific control registers. + +User-space applications can acquire exclusive access to an AFU attached to a +port by using open() on the port device node, and release it using close(). + +The following functions are exposed through ioctls: + + Get driver API version (FPGA_GET_API_VERSION) + Check for extensions (FPGA_CHECK_EXTENSION) + Get port info (FPGA_PORT_GET_INFO) + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) + Map DMA buffer (FPGA_PORT_DMA_MAP) + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) + Reset AFU (FPGA_PORT_RESET) + Enable UMsg (FPGA_PORT_UMSG_ENABLE) + Disable UMsg (FPGA_PORT_UMSG_DISABLE) + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) + +User-space applications can also mmap() accelerator MMIO regions. + +More functions are exposed through sysfs: +(/sys/class/fpga_region///): + + Read Accelerator GUID (afu_id) + Error reporting (errors/) + +Partial Reconfiguration +======================= +As mentioned above, accelerators can be reconfigured through partial +reconfiguration of a green bitstream file (GBS). The green bitstream must have +been generated for the exact blue bitstream and targeted reconfigurable region +(port) of the FPGA; otherwise, the reconfiguration operation will fail and +possibly cause system instability. This compatibility can be checked by +comparing the interface ID noted in the GBS header against the interface ID +exposed by the FME through sysfs (see above). This check is usually done by +user-space before calling the reconfiguration IOCTL. + +FPGA virtualization +=================== +To enable accessing an accelerator from applications running in a VM, the +respective AFU's port needs to be assigned to a VF using the following steps: + + a) The PF owns all AFU ports by default. Any port that needs to be reassigned + to a VF must first be released through the FPGA_FME_PORT_RELEASE ioctl on the + FME device. + + b) Once N ports are released from PF, then user can use command below to + enable SRIOV and VFs. Each VF owns only one Port with AFU. + + echo N > $PCI_DEVICE_PATH/sriov_numvfs + + c) Pass through the VFs to VMs + + d) The AFU under VF is accessible from applications in VM (using the same + driver inside the VF). + +Note that an FME can't be assigned to a VF, thus PR and other management +functions are only available via the PF. + + +Driver organization +=================== + + +-------++------++------+ | + | FME || FME || FME | | + | FPGA || FPGA || FPGA | | + |Manager||Bridge||Region| | + +-------++------++------+ | + +-----------------------+ +--------+ | +--------+ + | FME | | AFU | | | AFU | + | Module | | Module | | | Module | + +-----------------------+ +--------+ | +--------+ + +-----------------------+ | +-----------------------+ + | FPGA Container Device | | | FPGA Container Device | + | (FPGA Base Region) | | | (FPGA Base Region) | + +-----------------------+ | +-----------------------+ + +------------------+ | +------------------+ + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | + +------------------+ Host | Machine +------------------+ + -------------------------------------- | ------------------------------ + +---------------+ | +---------------+ + | PCI PF Device | | | PCI VF Device | + +---------------+ | +---------------+ + +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This +driver plays an infrastructural role in the driver architecture. It: + + a) locates the Device Feature Lists in PCIE device BAR memory, handles + them and related resources to common interfaces from DFL framework + for enumeration. + b) supports SRIOV. + +The feature device infrastructure provides common interfaces to create container +device (FPGA base region), discover feature devices and their sub features from +the given Device Feature Lists, and create platform devices for feature devices +with related resources under the container device. It also abstracts operations +for sub features and exposes common interfaces to feature device drivers. + +The FPGA Management Engine (FME) driver is a platform driver which is loaded +automatically after FME platform device creation from the PCIE driver. It +provides the key features for FPGA management, including: + + a) Power and thermal management, error reporting, performance reporting + and other infrastructure functions. Users can access these functions + via sysfs interfaces exposed by FME driver. + b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA + bridges and FPGA regions during PR sub feature initialization; Once + it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the + common interface function from FPGA Region to complete the partial + reconfiguration of the bitstream to the given port. + c) Port management for virtualization. The FME driver introduces two + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is + released from the PF, it can be assigned to the VF through the SRIOV + interfaces provided by PCIE driver. (Refer to "FPGA virtualization" + for more details). + +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver +is probed once the AFU platform device is created. The main function of this +module is to provide an interface for userspace applications to access the +individual accelerators, including basic reset control on port, AFU MMIO region +export, dma buffer mapping service, UMsg notification, and remote debug +functions (see above). + + +Device enumeration +================== +This section introduces how applications enumerate the fpga device from +the sysfs hierarchy under /sys/class/fpga_region. + +In the example below, two Intel(R) FPGA devices are installed in the host. Each +fpga device has one FME and two ports (AFUs). + +FPGA regions are created under /sys/class/fpga_region/ + + /sys/class/fpga_region/region0 + /sys/class/fpga_region/region1 + /sys/class/fpga_region/region2 + ... + +Application needs to search each regionX folder, if feature device is found, +(e.g "fpga-dfl-port.n" or "fpga-dfl-fme.m" is found), then it's the base +fpga region which represents the FPGA device. + +Each base region has one FME and two ports (AFUs) as child devices: + + /sys/class/fpga_region/region0/fpga-dfl-fme.0 + /sys/class/fpga_region/region0/fpga-dfl-port.0 + /sys/class/fpga_region/region0/fpga-dfl-port.1 + ... + + /sys/class/fpga_region/region3/fpga-dfl-fme.1 + /sys/class/fpga_region/region3/fpga-dfl-port.2 + /sys/class/fpga_region/region3/fpga-dfl-port.3 + ... + +In general, the FME/AFU sysfs interfaces are named as follows: + + /sys/class/fpga_region/// + /sys/class/fpga_region/// + +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all +ports. + +The device nodes used for ioctl() or mmap() can be referenced through: + + /sys/class/fpga_region///dev + /sys/class/fpga_region///dev + +Open discussion +=============== +FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to +user now. In the future, if unified user interfaces for reconfiguration are +added, FME driver should switch to them from ioctl interface. -- 1.8.3.1