From: Wu Hao <hao.wu@intel.com>
To: atull@kernel.org, mdf@kernel.org, linux-fpga@vger.kernel.org,
linux-kernel@vger.kernel.org
Cc: linux-api@vger.kernel.org, Wu Hao <hao.wu@intel.com>,
Xu Yilun <yilun.xu@intel.com>
Subject: [PATCH v3 05/16] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
Date: Mon, 27 May 2019 13:22:15 +0800 [thread overview]
Message-ID: <1558934546-12171-6-git-send-email-hao.wu@intel.com> (raw)
In-Reply-To: <1558934546-12171-1-git-send-email-hao.wu@intel.com>
This patch adds virtualization support description for DFL based
FPGA devices (based on PCIe SRIOV), and introductions to new
interfaces added by new dfl private feature drivers.
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
Acked-by: Alan Tull <atull@kernel.org>
---
v2: update description for thermal/power management user interfaces.
v3: remove thermal/power management feature description.
---
Documentation/fpga/dfl.txt | 105 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 105 insertions(+)
diff --git a/Documentation/fpga/dfl.txt b/Documentation/fpga/dfl.txt
index 6df4621..9e912a0 100644
--- a/Documentation/fpga/dfl.txt
+++ b/Documentation/fpga/dfl.txt
@@ -84,6 +84,8 @@ The following functions are exposed through ioctls:
Get driver API version (DFL_FPGA_GET_API_VERSION)
Check for extensions (DFL_FPGA_CHECK_EXTENSION)
Program bitstream (DFL_FPGA_FME_PORT_PR)
+ Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
+ Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
More functions are exposed through sysfs
(/sys/class/fpga_region/regionX/dfl-fme.n/):
@@ -99,6 +101,14 @@ More functions are exposed through sysfs
one FPGA device may have more than one port, this sysfs interface indicates
how many ports the FPGA device has.
+ Global error reporting management (errors/)
+ error reporting sysfs interfaces allow user to read errors detected by the
+ hardware, and clear the logged errors.
+
+ Performance counters (perf/)
+ performance counters sysfs interfaces allow user to use different counters
+ to get performance data.
+
FIU - PORT
==========
@@ -139,6 +149,10 @@ More functions are exposed through sysfs:
Read Accelerator GUID (afu_id)
afu_id indicates which PR bitstream is programmed to this AFU.
+ Error reporting (errors/)
+ error reporting sysfs interfaces allow user to read port/afu errors
+ detected by the hardware, and clear the logged errors.
+
DFL Framework Overview
======================
@@ -212,6 +226,97 @@ the compat_id exposed by the target FPGA region. This check is usually done by
userspace before calling the reconfiguration IOCTL.
+FPGA virtualization - PCIe SRIOV
+================================
+This section describes the virtualization support on DFL based FPGA device to
+enable accessing an accelerator from applications running in a virtual machine
+(VM). This section only describes the PCIe based FPGA device with SRIOV support.
+
+Features supported by the particular FPGA device are exposed through Device
+Feature Lists, as illustrated below:
+
+ +-------------------------------+ +-------------+
+ | PF | | VF |
+ +-------------------------------+ +-------------+
+ ^ ^ ^ ^
+ | | | |
++-----|------------|---------|--------------|-------+
+| | | | | |
+| +-----+ +-------+ +-------+ +-------+ |
+| | FME | | Port0 | | Port1 | | Port2 | |
+| +-----+ +-------+ +-------+ +-------+ |
+| ^ ^ ^ |
+| | | | |
+| +-------+ +------+ +-------+ |
+| | AFU | | AFU | | AFU | |
+| +-------+ +------+ +-------+ |
+| |
+| DFL based FPGA PCIe Device |
++---------------------------------------------------+
+
+FME is always accessed through the physical function (PF).
+
+Ports (and related AFUs) are accessed via PF by default, but could be exposed
+through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
+1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
+created via PCIe SRIOV interface, to virtual machines.
+
+The driver organization in virtualization case is illustrated below:
+
+ +-------++------++------+ |
+ | FME || FME || FME | |
+ | FPGA || FPGA || FPGA | |
+ |Manager||Bridge||Region| |
+ +-------++------++------+ |
+ +-----------------------+ +--------+ | +--------+
+ | FME | | AFU | | | AFU |
+ | Module | | Module | | | Module |
+ +-----------------------+ +--------+ | +--------+
+ +-----------------------+ | +-----------------------+
+ | FPGA Container Device | | | FPGA Container Device |
+ | (FPGA Base Region) | | | (FPGA Base Region) |
+ +-----------------------+ | +-----------------------+
+ +------------------+ | +------------------+
+ | FPGA PCIE Module | | Virtual | FPGA PCIE Module |
+ +------------------+ Host | Machine +------------------+
+ -------------------------------------- | ------------------------------
+ +---------------+ | +---------------+
+ | PCI PF Device | | | PCI VF Device |
+ +---------------+ | +---------------+
+
+FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
+is detected. It:
+
+ a) finish enumeration on both FPGA PCIe PF and VF device using common
+ interfaces from DFL framework.
+ b) supports SRIOV.
+
+The FME device driver plays a management role in this driver architecture, it
+provides ioctls to release Port from PF and assign Port to PF. After release
+a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
+sysfs interface.
+
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+ a) The PF owns all AFU ports by default. Any port that needs to be
+ reassigned to a VF must first be released through the
+ DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.
+
+ b) Once N ports are released from PF, then user can use command below
+ to enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+ echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+ c) Pass through the VFs to VMs
+
+ d) The AFU under VF is accessible from applications in VM (using the
+ same driver inside the VF).
+
+Note that an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+
Device enumeration
==================
This section introduces how applications enumerate the fpga device from
--
1.8.3.1
next prev parent reply other threads:[~2019-05-27 5:40 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-27 5:22 [PATCH v3 00/16] add new features for FPGA DFL drivers Wu Hao
2019-05-27 5:22 ` [PATCH v3 01/16] fpga: dfl-fme-mgr: fix FME_PR_INTFC_ID register address Wu Hao
2019-05-27 5:22 ` [PATCH v3 02/16] fpga: dfl: fme: remove copy_to_user() in ioctl for PR Wu Hao
2019-05-27 5:22 ` [PATCH v3 03/16] fpga: dfl: fme: align PR buffer size per PR datawidth Wu Hao
2019-05-27 5:22 ` [PATCH v3 04/16] fpga: dfl: fme: support 512bit data width PR Wu Hao
2019-05-27 5:22 ` Wu Hao [this message]
2019-05-27 5:22 ` [PATCH v3 06/16] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support Wu Hao
2019-05-27 5:22 ` [PATCH v3 07/16] fpga: dfl: pci: enable SRIOV support Wu Hao
2019-05-27 5:22 ` [PATCH v3 08/16] fpga: dfl: afu: add AFU state related sysfs interfaces Wu Hao
2019-05-27 5:22 ` [PATCH v3 09/16] fpga: dfl: afu: add userclock " Wu Hao
2019-05-27 5:22 ` [PATCH v3 10/16] fpga: dfl: add id_table for dfl private feature driver Wu Hao
2019-05-27 5:22 ` [PATCH v3 11/16] fpga: dfl: afu: export __port_enable/disable function Wu Hao
2019-05-27 5:22 ` [PATCH v3 12/16] fpga: dfl: afu: add error reporting support Wu Hao
2019-05-27 5:22 ` [PATCH v3 13/16] fpga: dfl: afu: add STP (SignalTap) support Wu Hao
2019-05-27 5:22 ` [PATCH v3 14/16] fpga: dfl: fme: add capability sysfs interfaces Wu Hao
2019-05-27 5:22 ` [PATCH v3 15/16] fpga: dfl: fme: add global error reporting support Wu Hao
2019-05-27 5:22 ` [PATCH v3 16/16] fpga: dfl: fme: add performance " Wu Hao
2019-05-30 18:53 ` Alan Tull
2019-06-01 8:04 ` Wu Hao
2019-05-30 19:03 ` Greg KH
2019-06-01 9:11 ` Wu Hao
2019-06-01 9:42 ` Greg KH
2019-06-04 14:47 ` Wu Hao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1558934546-12171-6-git-send-email-hao.wu@intel.com \
--to=hao.wu@intel.com \
--cc=atull@kernel.org \
--cc=linux-api@vger.kernel.org \
--cc=linux-fpga@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mdf@kernel.org \
--cc=yilun.xu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).