From: Wu Hao <hao.wu@intel.com>
To: atull@kernel.org, moritz.fischer@ettus.com,
linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org
Cc: luwei.kang@intel.com, yi.z.zhang@intel.com, hao.wu@intel.com,
Enno Luebbers <enno.luebbers@intel.com>,
Xiao Guangrong <guangrong.xiao@linux.intel.com>
Subject: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
Date: Thu, 30 Mar 2017 20:08:01 +0800 [thread overview]
Message-ID: <1490875696-15145-2-git-send-email-hao.wu@intel.com> (raw)
In-Reply-To: <1490875696-15145-1-git-send-email-hao.wu@intel.com>
Add a document for Intel FPGA driver overview.
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
Documentation/fpga/intel-fpga.txt | 259 ++++++++++++++++++++++++++++++++++++++
1 file changed, 259 insertions(+)
create mode 100644 Documentation/fpga/intel-fpga.txt
diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
new file mode 100644
index 0000000..9396cea
--- /dev/null
+++ b/Documentation/fpga/intel-fpga.txt
@@ -0,0 +1,259 @@
+===============================================================================
+ Intel FPGA driver Overview
+-------------------------------------------------------------------------------
+ Enno Luebbers <enno.luebbers@intel.com>
+ Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ Wu Hao <hao.wu@intel.com>
+
+The Intel FPGA driver provides interfaces for userspace applications to
+configure, enumerate, open, and access FPGA accelerators on platforms equipped
+with Intel(R) FPGA solutions and enables system level management functions such
+as FPGA reconfiguration, power management, and virtualization.
+
+HW Architecture
+===============
+From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
+The FPGA device memory is organized using a predefined data structure (Device
+Feature List). Features supported by the particular FPGA device are exposed
+through these data structures, as illustrated below:
+
+ +-------------------------------+ +-------------+
+ | PF | | VF |
+ +-------------------------------+ +-------------+
+ ^ ^ ^ ^
+ | | | |
++-----|------------|---------|--------------|-------+
+| | | | | |
+| +-----+ +-------+ +-------+ +-------+ |
+| | FME | | Port0 | | Port1 | | Port2 | |
+| +-----+ +-------+ +-------+ +-------+ |
+| ^ ^ ^ |
+| | | | |
+| +-------+ +------+ +-------+ |
+| | AFU | | AFU | | AFU | |
+| +-------+ +------+ +-------+ |
+| |
+| FPGA PCIe Device |
++---------------------------------------------------+
+
+The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
+used to assign individual accelerators to virtual machines .
+
+FME (FPGA Management Engine)
+============================
+The FPGA Management Enging performs power and thermal management, error
+reporting, reconfiguration, performance reporting, and other infrastructure
+functions. Each FPGA has one FME, which is always accessed through the physical
+function (PF).
+
+User-space applications can acquire exclusive access to the FME using open(),
+and release it using close().
+
+The following functions are exposed through ioctls:
+
+ Get driver API version (FPGA_GET_API_VERSION)
+ Check for extensions (FPGA_CHECK_EXTENSION)
+ Assign port to PF (FPGA_FME_PORT_ASSIGN)
+ Release port from PF (FPGA_FME_PORT_RELEASE)
+ Program bitstream (FPGA_FME_PORT_PR)
+
+More functions are exposed through sysfs
+(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
+
+ Read bitstream ID (bitstream_id)
+ Read bitstream metadata (bitstream_metadata)
+ Read number of ports (ports_num)
+ Read socket ID (socket_id)
+ Read performance counters (perf/)
+ Power management (power_mgmt/)
+ Thermal management (thermal_mgmt/)
+ Error reporting (errors/)
+
+PORT
+====
+A port represents the interface between the static FPGA fabric (the "blue
+bitstream") and a partially reconfigurable region containing an AFU (the "green
+bitstream"). It controls the communication from SW to the accelerator and
+exposes features such as reset and debug.
+
+A PCIe device may have several ports and each port can be released from PF by
+FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
+sysfs interface.
+
+AFU
+===
+An AFU is attached to a port and exposes a 256k MMIO region to be used for
+accelerator-specific control registers.
+
+User-space applications can acquire exclusive access to an AFU attached to a
+port by using open() on the port device node, and release it using close().
+
+The following functions are exposed through ioctls:
+
+ Get driver API version (FPGA_GET_API_VERSION)
+ Check for extensions (FPGA_CHECK_EXTENSION)
+ Get port info (FPGA_PORT_GET_INFO)
+ Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
+ Map DMA buffer (FPGA_PORT_DMA_MAP)
+ Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
+ Reset AFU (FPGA_PORT_RESET)
+ Enable UMsg (FPGA_PORT_UMSG_ENABLE)
+ Disable UMsg (FPGA_PORT_UMSG_DISABLE)
+ Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
+ Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
+
+User-space applications can also mmap() accelerator MMIO regions.
+
+More functions are exposed through sysfs:
+(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
+
+ Read Accelerator GUID (afu_id)
+ Error reporting (errors/)
+
+Partial Reconfiguration
+=======================
+As mentioned above, accelerators can be reconfigured through partial
+reconfiguration of a green bitstream file (GBS). The green bitstream must have
+been generated for the exact blue bitstream and targeted reconfigurable region
+(port) of the FPGA; otherwise, the reconfiguration operation will fail and
+possibly cause system instability. This compatibility can be checked by
+comparing the interface ID noted in the GBS header against the interface ID
+exposed by the FME through sysfs (see above). This check is usually done by
+user-space before calling the reconfiguration IOCTL.
+
+FPGA virtualization
+===================
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+ a) The PF owns all AFU ports by default. Any port that needs to be reassigned
+ to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE
+ ioctl on the FME device.
+
+ b) Once N ports are released from PF, then user can use below command to
+ enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+ echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+ c) Pass through the VFs to VMs
+
+ d) The AFU under VF is accessiable from applications in VM (using the same
+ driver inside the VF).
+
+Note the an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+
+Driver organization
+===================
+
+ +------------------+ +---------+ | +---------+
+ | +-------+ | | | | | |
+ | | FPGA | FME | | AFU | | | AFU |
+ | |Manager| Module | | Module | | | Module |
+ | +-------+ | | | | | |
+ +------------------+ +---------+ | +---------+
+ +-----------------------+ | +-----------------------+
+ | FPGA Container Device | | | FPGA Container Device |
+ +-----------------------+ | +-----------------------+
+ +------------------+ | +------------------+
+ | FPGA PCIE Module | | Virtual | FPGA PCIE Module |
+ +------------------+ Host | Machine +------------------+
+ ------------------------------------ | ------------------------------
+ +---------------+ | +---------------+
+ | PCI PF Device | | | PCI VF Device |
+ +---------------+ | +---------------+
+
+The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
+driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
+driver plays an infrastructural role in the driver architecuture. It:
+
+ a) creates FPGA container device as parent of the feature devices.
+ b) walks through the Device Feature List, which is implemented in PCIE
+ device BAR memory, to discover feature devices and their sub features
+ and create platform device for them under the container device.
+ c) supports SRIOV.
+ d) introduces the feature device infrastructure, which abstracts
+ operations for sub features and exposes common functions to feature
+ device drivers.
+
+The FPGA Management Engine (FME) driver is a platform driver which is loaded
+automatically after FME platform device creation from the PCIE driver. It
+provides the key features for FPGA management, including:
+
+ a) Power and thermal management, error reporting, performance reporting
+ and other infrastructure functions. Users can access these functions
+ via sysfs interfaces exposed by FME driver.
+ b) Paritial Reconfiguration. The FME driver registers a FPGA Manager
+ during PR sub feature initialization; once it receives an
+ FPGA_FME_PORT_PR ioctl from user, it invokes the common interface
+ function from FPGA Manager to complete the partial reconfiguration of
+ the bitstream to the given port.
+ c) Port management for virtualization. The FME driver introduces two
+ ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
+ FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
+ released from the PF, it can be assigned to the VF through the SRIOV
+ interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
+ for more details).
+
+Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
+is probed once the AFU platform device is created. The main function of this
+module is to provide an interface for userspace applications to access the
+individual accelerators, including basic reset control on port, AFU MMIO region
+export, dma buffer mapping service, UMsg notification, and remote debug
+functions (see above).
+
+
+Device enumeration
+==================
+This section introduces how applications enumerate the fpga device from
+the sysfs hierarchy under /sys/class/fpga.
+
+In the example below, two Intel(R) FPGA devices are installed in the host. Each
+fpga device has one FME and two ports (AFUs).
+
+For each FPGA device, a device director is created under /sys/class/fpga/:
+
+ /sys/class/fpga/fpga.0
+ /sys/class/fpga/fpga.1
+
+The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name.
+Application can retrieve name information via the sysfs interface:
+
+ /sys/class/fpga/fpga.0/name
+
+Each node has one FME and two ports (AFUs) as child devices:
+
+ /sys/class/fpga/fpga.0/intel-fpga-fme.0
+ /sys/class/fpga/fpga.0/intel-fpga-port.0
+ /sys/class/fpga/fpga.0/intel-fpga-port.1
+
+ /sys/class/fpga/fpga.1/intel-fpga-fme.1
+ /sys/class/fpga/fpga.1/intel-fpga-port.2
+ /sys/class/fpga/fpga.1/intel-fpga-port.3
+
+In general, the FME/AFU sysfs interfaces are named as follows:
+
+ /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
+ /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
+
+with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
+ports.
+
+The device nodes used for ioctl() or mmap() can be referenced through:
+
+ /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
+ /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
+
+
+Open discussions
+================
+The current FME driver does not provide user space access to the FME MMIO
+region, but exposes access through sysfs and ioctls. It also provides an FPGA
+manger interface for partial reconfiguration (PR), but does not make use of
+fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled inside
+the FME, and fpga-region depends on device tree which is not used at all. There
+are patches from Alan Tull to separate the device tree specific code and
+introduce a sysfs interface for PR. We plan to add fpga-regions support in the
+driver once the related patches get merged. Then the FME driver should create
+one fpga-region for each Port/AFU.
--
2.7.4
next prev parent reply other threads:[~2017-03-30 12:15 UTC|newest]
Thread overview: 93+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
2017-03-30 12:08 ` Wu Hao [this message]
2017-03-31 18:24 ` [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview matthew.gerlach
2017-03-31 18:38 ` Alan Tull
2017-04-01 11:16 ` Wu Hao
2017-04-02 14:41 ` Moritz Fischer
2017-04-03 20:44 ` Alan Tull
2017-04-04 5:24 ` Wu Hao
2017-04-04 5:06 ` Wu Hao
2017-04-11 18:02 ` Alan Tull
2017-04-12 3:22 ` Wu, Hao
2017-03-30 12:08 ` [PATCH 02/16] fpga: add FPGA device framework Wu Hao
2017-03-31 6:09 ` Greg KH
2017-03-31 7:48 ` Wu Hao
2017-03-31 9:03 ` Greg KH
2017-03-31 12:19 ` Wu Hao
2017-03-31 19:01 ` matthew.gerlach
2017-04-01 12:18 ` Wu Hao
2017-07-25 21:32 ` Alan Tull
2017-07-26 9:50 ` Wu Hao
2017-07-26 14:20 ` Alan Tull
2017-07-26 22:29 ` Alan Tull
2017-07-27 4:54 ` Wu Hao
2017-03-31 6:13 ` Greg KH
[not found] ` <82D7661F83C1A047AF7DC287873BF1E167C90F1B@SHSMSX101.ccr.corp.intel.com>
2017-03-31 13:31 ` Wu Hao
2017-03-31 14:10 ` Greg KH
2017-04-01 11:36 ` Wu Hao
2017-03-30 12:08 ` [PATCH 03/16] fpga: intel: add FPGA PCIe device driver Wu Hao
2017-04-04 2:10 ` Moritz Fischer
2017-04-05 13:14 ` Wu, Hao
2017-03-30 12:08 ` [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features Wu Hao
2017-04-03 21:44 ` Alan Tull
2017-04-05 11:58 ` Wu Hao
2017-04-11 20:21 ` Alan Tull
2017-04-13 4:12 ` Wu, Hao
2017-04-04 2:44 ` Moritz Fischer
2017-04-05 12:57 ` Wu Hao
2017-04-04 22:09 ` Alan Tull
2017-04-05 14:09 ` Wu Hao
2017-05-04 15:13 ` Li, Yi
2017-05-05 3:03 ` Wu Hao
2017-03-30 12:08 ` [PATCH 05/16] fpga: intel: pcie: add chardev support for feature devices Wu Hao
2017-03-30 12:08 ` [PATCH 06/16] fpga: intel: pcie: adds fpga_for_each_port callback for fme device Wu Hao
2017-03-30 12:08 ` [PATCH 07/16] fpga: intel: add feature device infrastructure Wu Hao
2017-03-30 12:08 ` [PATCH 08/16] fpga: intel: add FPGA Management Engine driver basic framework Wu Hao
2017-03-30 12:08 ` [PATCH 09/16] fpga: intel: fme: add header sub feature support Wu Hao
2017-03-30 12:08 ` [PATCH 10/16] fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao
2017-03-30 12:08 ` [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao
2017-03-30 20:30 ` Alan Tull
2017-03-31 4:11 ` Xiao Guangrong
2017-03-31 8:50 ` Wu Hao
2017-04-03 20:26 ` Alan Tull
2017-04-04 5:25 ` Wu Hao
2017-03-31 19:10 ` Alan Tull
2017-04-01 11:08 ` Wu Hao
2017-04-03 16:30 ` Alan Tull
2017-04-04 6:05 ` Wu Hao
2017-04-04 22:37 ` Alan Tull
2017-04-05 11:40 ` Wu, Hao
2017-04-05 15:26 ` Alan Tull
2017-04-05 15:39 ` Alan Tull
2017-04-06 10:57 ` Wu Hao
2017-04-06 19:27 ` Alan Tull
2017-04-07 5:56 ` Wu Hao
2017-03-31 23:45 ` kbuild test robot
2017-04-01 1:12 ` kbuild test robot
2017-04-03 21:24 ` Alan Tull
2017-04-03 22:49 ` matthew.gerlach
2017-04-04 6:48 ` Wu Hao
2017-04-04 6:28 ` Wu Hao
2017-03-30 12:08 ` [PATCH 12/16] fpga: intel: add FPGA Accelerated Function Unit driver basic framework Wu Hao
2017-03-30 12:08 ` [PATCH 13/16] fpga: intel: afu: add header sub feature support Wu Hao
2017-03-30 12:08 ` [PATCH 14/16] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao
2017-03-30 12:08 ` [PATCH 15/16] fpga: intel: afu: add user afu sub feature support Wu Hao
2017-03-30 12:08 ` [PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support Wu Hao
2017-04-01 0:00 ` kbuild test robot
2017-04-01 1:33 ` kbuild test robot
2017-03-30 17:17 ` [PATCH 00/16] Intel FPGA Device Drivers Moritz Fischer
2017-04-06 20:27 ` Jerome Glisse
2017-04-11 19:38 ` Luebbers, Enno
2017-04-12 13:29 ` Jerome Glisse
2017-04-12 14:46 ` Moritz Fischer
2017-04-12 15:37 ` Jerome Glisse
2017-04-14 19:48 ` Luebbers, Enno
2017-04-14 20:49 ` Jerome Glisse
2017-04-17 15:35 ` Alan Tull
2017-04-17 15:57 ` Jerome Glisse
2017-04-17 16:22 ` Alan Tull
2017-04-17 17:15 ` Jerome Glisse
2017-04-18 13:36 ` Alan Cox
2017-04-18 14:59 ` Jerome Glisse
2017-04-25 20:02 ` One Thousand Gnomes
2017-05-01 16:41 ` Jerome Glisse
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1490875696-15145-2-git-send-email-hao.wu@intel.com \
--to=hao.wu@intel.com \
--cc=atull@kernel.org \
--cc=enno.luebbers@intel.com \
--cc=guangrong.xiao@linux.intel.com \
--cc=linux-fpga@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=luwei.kang@intel.com \
--cc=moritz.fischer@ettus.com \
--cc=yi.z.zhang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).