From: Moritz Fischer <mdf@kernel.org>
To: linux-fpga@vger.kernel.org, gregkh@linuxfoundation.org
Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
Wu Hao <hao.wu@intel.com>, Xu Yilun <yilun.xu@intel.com>,
Alan Tull <atull@kernel.org>, Moritz Fischer <mdf@kernel.org>
Subject: [PATCH 05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
Date: Thu, 27 Jun 2019 17:49:41 -0700 [thread overview]
Message-ID: <20190628004951.6202-6-mdf@kernel.org> (raw)
In-Reply-To: <20190628004951.6202-1-mdf@kernel.org>
From: Wu Hao <hao.wu@intel.com>
This patch adds virtualization support description for DFL based
FPGA devices (based on PCIe SRIOV), and introductions to new
interfaces added by new dfl private feature drivers.
[mdf@kernel.org: Fixed up to make it work with new reStructuredText docs]
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++
1 file changed, 100 insertions(+)
diff --git a/Documentation/fpga/dfl.rst b/Documentation/fpga/dfl.rst
index 2f125abd777f..be9929dd7251 100644
--- a/Documentation/fpga/dfl.rst
+++ b/Documentation/fpga/dfl.rst
@@ -87,6 +87,8 @@ The following functions are exposed through ioctls:
- Get driver API version (DFL_FPGA_GET_API_VERSION)
- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
- Program bitstream (DFL_FPGA_FME_PORT_PR)
+- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
+- Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
More functions are exposed through sysfs
(/sys/class/fpga_region/regionX/dfl-fme.n/):
@@ -143,6 +145,9 @@ More functions are exposed through sysfs:
Read Accelerator GUID (afu_id)
afu_id indicates which PR bitstream is programmed to this AFU.
+ Global error reporting management (errors/)
+ error reporting sysfs interfaces allow user to read errors detected by the
+ hardware, and clear the logged errors.
DFL Framework Overview
======================
@@ -218,6 +223,101 @@ the compat_id exposed by the target FPGA region. This check is usually done by
userspace before calling the reconfiguration IOCTL.
+FPGA virtualization - PCIe SRIOV
+================================
+This section describes the virtualization support on DFL based FPGA device to
+enable accessing an accelerator from applications running in a virtual machine
+(VM). This section only describes the PCIe based FPGA device with SRIOV support.
+
+Features supported by the particular FPGA device are exposed through Device
+Feature Lists, as illustrated below:
+
+::
+
+ +-------------------------------+ +-------------+
+ | PF | | VF |
+ +-------------------------------+ +-------------+
+ ^ ^ ^ ^
+ | | | |
+ +-----|------------|---------|--------------|-------+
+ | | | | | |
+ | +-----+ +-------+ +-------+ +-------+ |
+ | | FME | | Port0 | | Port1 | | Port2 | |
+ | +-----+ +-------+ +-------+ +-------+ |
+ | ^ ^ ^ |
+ | | | | |
+ | +-------+ +------+ +-------+ |
+ | | AFU | | AFU | | AFU | |
+ | +-------+ +------+ +-------+ |
+ | |
+ | DFL based FPGA PCIe Device |
+ +---------------------------------------------------+
+
+FME is always accessed through the physical function (PF).
+
+Ports (and related AFUs) are accessed via PF by default, but could be exposed
+through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
+1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
+created via PCIe SRIOV interface, to virtual machines.
+
+The driver organization in virtualization case is illustrated below:
+::
+
+ +-------++------++------+ |
+ | FME || FME || FME | |
+ | FPGA || FPGA || FPGA | |
+ |Manager||Bridge||Region| |
+ +-------++------++------+ |
+ +-----------------------+ +--------+ | +--------+
+ | FME | | AFU | | | AFU |
+ | Module | | Module | | | Module |
+ +-----------------------+ +--------+ | +--------+
+ +-----------------------+ | +-----------------------+
+ | FPGA Container Device | | | FPGA Container Device |
+ | (FPGA Base Region) | | | (FPGA Base Region) |
+ +-----------------------+ | +-----------------------+
+ +------------------+ | +------------------+
+ | FPGA PCIE Module | | Virtual | FPGA PCIE Module |
+ +------------------+ Host | Machine +------------------+
+ -------------------------------------- | ------------------------------
+ +---------------+ | +---------------+
+ | PCI PF Device | | | PCI VF Device |
+ +---------------+ | +---------------+
+
+FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
+is detected. It:
+
+* Finishes enumeration on both FPGA PCIe PF and VF device using common
+ interfaces from DFL framework.
+* Supports SRIOV.
+
+The FME device driver plays a management role in this driver architecture, it
+provides ioctls to release Port from PF and assign Port to PF. After release
+a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
+sysfs interface.
+
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+#. The PF owns all AFU ports by default. Any port that needs to be
+ reassigned to a VF must first be released through the
+ DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.
+
+#. Once N ports are released from PF, then user can use command below
+ to enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+ ::
+
+ echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+#. Pass through the VFs to VMs
+
+#. The AFU under VF is accessible from applications in VM (using the
+ same driver inside the VF).
+
+Note that an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
Device enumeration
==================
This section introduces how applications enumerate the fpga device from
--
2.22.0
next prev parent reply other threads:[~2019-06-28 0:51 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-28 0:49 [PATCH 00/15] FPGA DFL updates Moritz Fischer
2019-06-28 0:49 ` [PATCH 01/15] fpga: dfl-fme-mgr: fix FME_PR_INTFC_ID register address Moritz Fischer
2019-06-28 0:49 ` [PATCH 02/15] fpga: dfl: fme: remove copy_to_user() in ioctl for PR Moritz Fischer
2019-06-28 0:49 ` [PATCH 03/15] fpga: dfl: fme: align PR buffer size per PR datawidth Moritz Fischer
2019-06-28 0:49 ` [PATCH 04/15] fpga: dfl: fme: support 512bit data width PR Moritz Fischer
2019-07-03 17:56 ` Greg KH
2019-07-03 23:44 ` Wu Hao
2019-06-28 0:49 ` Moritz Fischer [this message]
2019-07-03 17:59 ` [PATCH 05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces Greg KH
2019-07-03 23:38 ` Wu Hao
2019-07-04 5:37 ` Greg KH
2019-07-04 6:42 ` Wu Hao
2019-07-04 8:17 ` Greg KH
2019-07-04 8:19 ` Wu Hao
2019-06-28 0:49 ` [PATCH 06/15] fpga: dfl: fme: add DFL_FPGA_FME_PORT_RELEASE/ASSIGN ioctl support Moritz Fischer
2019-07-03 18:07 ` Greg KH
2019-07-03 23:30 ` Wu Hao
2019-07-04 5:39 ` Greg KH
2019-07-04 6:31 ` Wu Hao
2019-07-04 8:20 ` Greg KH
2019-07-04 8:58 ` Wu Hao
2019-07-04 11:04 ` Greg KH
2019-07-04 11:29 ` Wu Hao
2019-06-28 0:49 ` [PATCH 07/15] fpga: dfl: pci: enable SRIOV support Moritz Fischer
2019-06-28 0:49 ` [PATCH 08/15] fpga: dfl: afu: add AFU state related sysfs interfaces Moritz Fischer
2019-06-28 0:49 ` [PATCH 09/15] fpga: dfl: afu: add userclock " Moritz Fischer
2019-06-28 0:49 ` [PATCH 10/15] fpga: dfl: add id_table for dfl private feature driver Moritz Fischer
2019-06-28 0:49 ` [PATCH 11/15] fpga: dfl: afu: export __port_enable/disable function Moritz Fischer
2019-06-28 0:49 ` [PATCH 12/15] fpga: dfl: afu: add error reporting support Moritz Fischer
2019-06-28 0:49 ` [PATCH 13/15] fpga: dfl: afu: add STP (SignalTap) support Moritz Fischer
2019-06-28 0:49 ` [PATCH 14/15] fpga: dfl: fme: add capability sysfs interfaces Moritz Fischer
2019-06-28 0:49 ` [PATCH 15/15] fpga: dfl: fme: add global error reporting support Moritz Fischer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190628004951.6202-6-mdf@kernel.org \
--to=mdf@kernel.org \
--cc=atull@kernel.org \
--cc=gregkh@linuxfoundation.org \
--cc=hao.wu@intel.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-fpga@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=yilun.xu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).