linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Alan Tull <atull@kernel.org>
To: matthew.gerlach@linux.intel.com
Cc: Wu Hao <hao.wu@intel.com>,
	Moritz Fischer <moritz.fischer@ettus.com>,
	linux-fpga@vger.kernel.org,
	linux-kernel <linux-kernel@vger.kernel.org>,
	luwei.kang@intel.com, yi.z.zhang@intel.com,
	Enno Luebbers <enno.luebbers@intel.com>,
	Xiao Guangrong <guangrong.xiao@linux.intel.com>
Subject: Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
Date: Fri, 31 Mar 2017 13:38:06 -0500	[thread overview]
Message-ID: <CANk1AXTPM+9Xdahw2sy4kSJ-Dket3Z5eS83AvW4agwSvvoFSjQ@mail.gmail.com> (raw)
In-Reply-To: <alpine.DEB.2.20.1703310907270.2072@mgerlach-VirtualBox>

On Fri, Mar 31, 2017 at 1:24 PM,  <matthew.gerlach@linux.intel.com> wrote:
>
>
> On Thu, 30 Mar 2017, Wu Hao wrote:
>
>
> Hi Wu Hao,
>
> Great documentation. I'm looking forward to diving into the rest of the
> patches. Please see my comments inline.
>
> Matthew Gerlach
>
>
>> Add a document for Intel FPGA driver overview.
>>
>> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> Signed-off-by: Wu Hao <hao.wu@intel.com>
>> ---
>> Documentation/fpga/intel-fpga.txt | 259
>> ++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 259 insertions(+)
>> create mode 100644 Documentation/fpga/intel-fpga.txt
>>
>> diff --git a/Documentation/fpga/intel-fpga.txt
>> b/Documentation/fpga/intel-fpga.txt
>> new file mode 100644
>> index 0000000..9396cea
>> --- /dev/null
>> +++ b/Documentation/fpga/intel-fpga.txt
>> @@ -0,0 +1,259 @@
>>
>> +===============================================================================
>> +                    Intel FPGA driver Overview
>>
>> +-------------------------------------------------------------------------------
>> +                Enno Luebbers <enno.luebbers@intel.com>
>> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> +                Wu Hao <hao.wu@intel.com>
>> +
>> +The Intel FPGA driver provides interfaces for userspace applications to
>> +configure, enumerate, open, and access FPGA accelerators on platforms
>> equipped
>> +with Intel(R) FPGA solutions and enables system level management
>> functions such
>> +as FPGA reconfiguration, power management, and virtualization.
>> +
>
>
> From a Linux kernel perspective, I'm not sure this is the best name for
> this code.  The name gives me the impression that it is a driver for all
> Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
> PCIe bus.  The processor could be directely connected like the Arria10
> SOCFPGA.  Such a processor could certainly benefit from this accelerator
> usage model.  In an extreme case, couldn't a processor in the FPGA,
> running Linux, also benefit from this accelerator model?  Is this code a
> "FPGA Accelerator Framework"?
>
>> +HW Architecture
>> +===============
>> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
>> device.
>> +The FPGA device memory is organized using a predefined data structure
>> (Device
>> +Feature List). Features supported by the particular FPGA device are
>> exposed
>> +through these data structures, as illustrated below:
>> +
>> +  +-------------------------------+  +-------------+
>> +  |              PF               |  |     VF      |
>> +  +-------------------------------+  +-------------+
>> +      ^            ^         ^              ^
>> +      |            |         |              |
>> ++-----|------------|---------|--------------|-------+
>> +|     |            |         |              |       |
>> +|  +-----+     +-------+ +-------+      +-------+   |
>> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
>> +|  +-----+     +-------+ +-------+      +-------+   |
>> +|                  ^         ^              ^       |
>> +|                  |         |              |       |
>> +|              +-------+ +------+       +-------+   |
>> +|              |  AFU  | |  AFU |       |  AFU  |   |
>> +|              +-------+ +------+       +-------+   |
>> +|                                                   |
>> +|                 FPGA PCIe Device                  |
>> ++---------------------------------------------------+
>> +
>> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
>> can be
>> +used to assign individual accelerators to virtual machines .
>
>
> Does this HW Architecture require an Intel FPGA?  Couldn't any vendors FPGA
> be used as long as it presented itself the PCIe bus the same and contained
> an appropriate Device Feature List?
>
>> +
>> +FME (FPGA Management Engine)
>> +============================
>> +The FPGA Management Enging performs power and thermal management, error
>> +reporting, reconfiguration, performance reporting, and other
>> infrastructure
>> +functions. Each FPGA has one FME, which is always accessed through the
>> physical
>> +function (PF).
>> +
>> +User-space applications can acquire exclusive access to the FME using
>> open(),
>> +and release it using close().
>> +
>> +The following functions are exposed through ioctls:
>> +
>> +       Get driver API version (FPGA_GET_API_VERSION)
>> +       Check for extensions (FPGA_CHECK_EXTENSION)
>> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
>> +       Release port from PF (FPGA_FME_PORT_RELEASE)
>> +       Program bitstream (FPGA_FME_PORT_PR)
>> +
>> +More functions are exposed through sysfs
>> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
>> +
>> +       Read bitstream ID (bitstream_id)
>> +       Read bitstream metadata (bitstream_metadata)
>> +       Read number of ports (ports_num)
>> +       Read socket ID (socket_id)
>> +       Read performance counters (perf/)
>> +       Power management (power_mgmt/)
>> +       Thermal management (thermal_mgmt/)
>> +       Error reporting (errors/)
>> +
>> +PORT
>> +====
>> +A port represents the interface between the static FPGA fabric (the "blue
>> +bitstream") and a partially reconfigurable region containing an AFU (the
>> "green

Is this an fpga bridge but with added features?

>> +bitstream"). It controls the communication from SW to the accelerator and
>> +exposes features such as reset and debug.
>> +
>> +A PCIe device may have several ports and each port can be released from
>> PF by
>> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe
>> sriov
>> +sysfs interface.
>> +
>> +AFU
>> +===
>> +An AFU is attached to a port and exposes a 256k MMIO region to be used
>> for
>> +accelerator-specific control registers.
>> +
>> +User-space applications can acquire exclusive access to an AFU attached
>> to a
>> +port by using open() on the port device node, and release it using
>> close().
>> +
>> +The following functions are exposed through ioctls:
>> +
>> +       Get driver API version (FPGA_GET_API_VERSION)
>> +       Check for extensions (FPGA_CHECK_EXTENSION)
>> +       Get port info (FPGA_PORT_GET_INFO)
>> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
>> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
>> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
>> +       Reset AFU (FPGA_PORT_RESET)
>> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
>> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
>> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
>> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
>> +
>> +User-space applications can also mmap() accelerator MMIO regions.
>> +
>> +More functions are exposed through sysfs:
>> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
>> +
>> +       Read Accelerator GUID (afu_id)
>> +       Error reporting (errors/)
>> +
>> +Partial Reconfiguration
>> +=======================
>> +As mentioned above, accelerators can be reconfigured through partial
>> +reconfiguration of a green bitstream file (GBS). The green bitstream must
>> have
>> +been generated for the exact blue bitstream and targeted reconfigurable
>> region
>> +(port) of the FPGA; otherwise, the reconfiguration operation will fail
>> and
>> +possibly cause system instability. This compatibility can be checked by
>> +comparing the interface ID noted in the GBS header against the interface
>> ID
>> +exposed by the FME through sysfs (see above). This check is usually done
>> by
>> +user-space before calling the reconfiguration IOCTL.
>> +
>> +FPGA virtualization
>> +===================
>> +To enable accessing an accelerator from applications running in a VM, the
>> +respective AFU's port needs to be assigned to a VF using the following
>> steps:
>> +
>> + a) The PF owns all AFU ports by default. Any port that needs to be
>> reassigned
>> + to a VF must be released from PF firstly through the
>> FPGA_FME_PORT_RELEASE
>> + ioctl on the FME device.
>> +
>> + b) Once N ports are released from PF, then user can use below command to
>> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
>> +
>> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
>> +
>> + c) Pass through the VFs to VMs
>> +
>> + d) The AFU under VF is accessiable from applications in VM (using the
>> same
>> + driver inside the VF).
>> +
>> +Note the an FME can't be assigned to a VF, thus PR and other management
>> +functions are only available via the PF.
>> +
>> +
>> +Driver organization
>> +===================
>> +
>> +  +------------------+  +---------+   |             +---------+
>> +  | +-------+        |  |         |   |             |         |
>> +  | | FPGA  |  FME   |  |   AFU   |   |             |   AFU   |
>> +  | |Manager| Module |  |  Module |   |             |  Module |
>> +  | +-------+        |  |         |   |             |         |
>> +  +------------------+  +---------+   |             +---------+
>> +        +-----------------------+     |      +-----------------------+
>> +        | FPGA Container Device |     |      | FPGA Container Device |
>> +        +-----------------------+     |      +-----------------------+
>> +          +------------------+        |         +------------------+
>> +          | FPGA PCIE Module |        | Virtual | FPGA PCIE Module |
>> +          +------------------+   Host | Machine +------------------+
>> + ------------------------------------ | ------------------------------
>> +           +---------------+          |          +---------------+
>> +           | PCI PF Device |          |          | PCI VF Device |
>> +           +---------------+          |          +---------------+
>> +
>> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe
>> device
>> +driver is always loaded first once a FPGA PCIE PF or VF device is
>> detected. This
>> +driver plays an infrastructural role in the driver architecuture.  It:
>> +
>> +       a) creates FPGA container device as parent of the feature devices.
>> +       b) walks through the Device Feature List, which is implemented in
>> PCIE
>> +          device BAR memory, to discover feature devices and their sub
>> features
>> +          and create platform device for them under the container device.
>
>
> I really like the idea of creating platform devices for the sub features. It
> is in line with other FPGA use cases.  Platform devices are at the heart of
> device trees used by processors directly connected FPGAs and processors
> inside FPGAs.
>
>> +       c) supports SRIOV.
>> +       d) introduces the feature device infrastructure, which abstracts
>> +          operations for sub features and exposes common functions to
>> feature
>> +          device drivers.
>> +
>> +The FPGA Management Engine (FME) driver is a platform driver which is
>> loaded
>> +automatically after FME platform device creation from the PCIE driver. It
>> +provides the key features for FPGA management, including:
>> +
>> +       a) Power and thermal management, error reporting, performance
>> reporting
>> +          and other infrastructure functions. Users can access these
>> functions
>> +          via sysfs interfaces exposed by FME driver.
>> +       b) Paritial Reconfiguration. The FME driver registers a FPGA
>> Manager
>> +          during PR sub feature initialization; once it receives an
>> +          FPGA_FME_PORT_PR ioctl from user, it invokes the common
>> interface
>> +          function from FPGA Manager to complete the partial
>> reconfiguration of
>> +          the bitstream to the given port.
>> +       c) Port management for virtualization. The FME driver introduces
>> two
>> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
>> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the
>> port is
>> +          released from the PF, it can be assigned to the VF through the
>> SRIOV
>> +          interfaces provided by PCIE driver. (Refer to "FPGA
>> virtualization"
>> +          for more details).
>> +
>> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU)
>> driver
>> +is probed once the AFU platform device is created. The main function of
>> this
>> +module is to provide an interface for userspace applications to access
>> the
>> +individual accelerators, including basic reset control on port, AFU MMIO
>> region
>> +export, dma buffer mapping service, UMsg notification, and remote debug
>> +functions (see above).
>> +
>> +
>> +Device enumeration
>> +==================
>> +This section introduces how applications enumerate the fpga device from
>> +the sysfs hierarchy under /sys/class/fpga.
>> +
>> +In the example below, two Intel(R) FPGA devices are installed in the
>> host. Each
>> +fpga device has one FME and two ports (AFUs).
>> +
>> +For each FPGA device, a device director is created under
>> /sys/class/fpga/:
>> +
>> +       /sys/class/fpga/fpga.0
>> +       /sys/class/fpga/fpga.1
>> +
>> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's
>> name.
>> +Application can retrieve name information via the sysfs interface:
>> +
>> +       /sys/class/fpga/fpga.0/name
>> +
>> +Each node has one FME and two ports (AFUs) as child devices:
>> +
>> +       /sys/class/fpga/fpga.0/intel-fpga-fme.0
>> +       /sys/class/fpga/fpga.0/intel-fpga-port.0
>> +       /sys/class/fpga/fpga.0/intel-fpga-port.1
>> +
>> +       /sys/class/fpga/fpga.1/intel-fpga-fme.1
>> +       /sys/class/fpga/fpga.1/intel-fpga-port.2
>> +       /sys/class/fpga/fpga.1/intel-fpga-port.3
>> +
>> +In general, the FME/AFU sysfs interfaces are named as follows:
>> +
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
>> +
>> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering
>> all
>> +ports.
>> +
>> +The device nodes used for ioctl() or mmap() can be referenced through:
>> +
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
>> +
>> +
>> +Open discussions
>> +================
>> +The current FME driver does not provide user space access to the FME MMIO
>> +region, but exposes access through sysfs and ioctls. It also provides an
>> FPGA
>> +manger interface for partial reconfiguration (PR), but does not make use
>> of
>> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled
>> inside
>> +the FME, and fpga-region depends on device tree which is not used at all.
>> There
>> +are patches from Alan Tull to separate the device tree specific code and
>
>
> I am currently trying to use those patches in a different driver.  They've
> compiled cleanly in my out of tree pcie module driver against the 3.10
> kernel.
> I need to actually write the code to create and register the region, but
> Alan's platform driver code should be a good guide for me.  Just need to
> find the time.
>
>> +introduce a sysfs interface for PR. We plan to add fpga-regions support
>> in the
>> +driver once the related patches get merged. Then the FME driver should
>> create
>> +one fpga-region for each Port/AFU.
>
>
> Does the FME driver create the fpga-region, or is each region described as
> an entry in the Device Feature List and therefore created by the code that
> enumerates the Device Feature List?
>
>> --
>> 2.7.4
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

  reply	other threads:[~2017-03-31 18:38 UTC|newest]

Thread overview: 93+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
2017-03-30 12:08 ` [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Wu Hao
2017-03-31 18:24   ` matthew.gerlach
2017-03-31 18:38     ` Alan Tull [this message]
2017-04-01 11:16       ` Wu Hao
2017-04-02 14:41         ` Moritz Fischer
2017-04-03 20:44           ` Alan Tull
2017-04-04  5:24             ` Wu Hao
2017-04-04  5:06           ` Wu Hao
2017-04-11 18:02           ` Alan Tull
2017-04-12  3:22             ` Wu, Hao
2017-03-30 12:08 ` [PATCH 02/16] fpga: add FPGA device framework Wu Hao
2017-03-31  6:09   ` Greg KH
2017-03-31  7:48     ` Wu Hao
2017-03-31  9:03       ` Greg KH
2017-03-31 12:19         ` Wu Hao
2017-03-31 19:01       ` matthew.gerlach
2017-04-01 12:18         ` Wu Hao
2017-07-25 21:32           ` Alan Tull
2017-07-26  9:50             ` Wu Hao
2017-07-26 14:20               ` Alan Tull
2017-07-26 22:29                 ` Alan Tull
2017-07-27  4:54                   ` Wu Hao
2017-03-31  6:13   ` Greg KH
     [not found]     ` <82D7661F83C1A047AF7DC287873BF1E167C90F1B@SHSMSX101.ccr.corp.intel.com>
2017-03-31 13:31       ` Wu Hao
2017-03-31 14:10         ` Greg KH
2017-04-01 11:36           ` Wu Hao
2017-03-30 12:08 ` [PATCH 03/16] fpga: intel: add FPGA PCIe device driver Wu Hao
2017-04-04  2:10   ` Moritz Fischer
2017-04-05 13:14     ` Wu, Hao
2017-03-30 12:08 ` [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features Wu Hao
2017-04-03 21:44   ` Alan Tull
2017-04-05 11:58     ` Wu Hao
2017-04-11 20:21       ` Alan Tull
2017-04-13  4:12         ` Wu, Hao
2017-04-04  2:44   ` Moritz Fischer
2017-04-05 12:57     ` Wu Hao
2017-04-04 22:09   ` Alan Tull
2017-04-05 14:09     ` Wu Hao
2017-05-04 15:13   ` Li, Yi
2017-05-05  3:03     ` Wu Hao
2017-03-30 12:08 ` [PATCH 05/16] fpga: intel: pcie: add chardev support for feature devices Wu Hao
2017-03-30 12:08 ` [PATCH 06/16] fpga: intel: pcie: adds fpga_for_each_port callback for fme device Wu Hao
2017-03-30 12:08 ` [PATCH 07/16] fpga: intel: add feature device infrastructure Wu Hao
2017-03-30 12:08 ` [PATCH 08/16] fpga: intel: add FPGA Management Engine driver basic framework Wu Hao
2017-03-30 12:08 ` [PATCH 09/16] fpga: intel: fme: add header sub feature support Wu Hao
2017-03-30 12:08 ` [PATCH 10/16] fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao
2017-03-30 12:08 ` [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao
2017-03-30 20:30   ` Alan Tull
2017-03-31  4:11     ` Xiao Guangrong
2017-03-31  8:50       ` Wu Hao
2017-04-03 20:26         ` Alan Tull
2017-04-04  5:25           ` Wu Hao
2017-03-31 19:10   ` Alan Tull
2017-04-01 11:08     ` Wu Hao
2017-04-03 16:30       ` Alan Tull
2017-04-04  6:05         ` Wu Hao
2017-04-04 22:37           ` Alan Tull
2017-04-05 11:40             ` Wu, Hao
2017-04-05 15:26               ` Alan Tull
2017-04-05 15:39                 ` Alan Tull
2017-04-06 10:57                   ` Wu Hao
2017-04-06 19:27                     ` Alan Tull
2017-04-07  5:56                       ` Wu Hao
2017-03-31 23:45   ` kbuild test robot
2017-04-01  1:12   ` kbuild test robot
2017-04-03 21:24   ` Alan Tull
2017-04-03 22:49     ` matthew.gerlach
2017-04-04  6:48       ` Wu Hao
2017-04-04  6:28     ` Wu Hao
2017-03-30 12:08 ` [PATCH 12/16] fpga: intel: add FPGA Accelerated Function Unit driver basic framework Wu Hao
2017-03-30 12:08 ` [PATCH 13/16] fpga: intel: afu: add header sub feature support Wu Hao
2017-03-30 12:08 ` [PATCH 14/16] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao
2017-03-30 12:08 ` [PATCH 15/16] fpga: intel: afu: add user afu sub feature support Wu Hao
2017-03-30 12:08 ` [PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support Wu Hao
2017-04-01  0:00   ` kbuild test robot
2017-04-01  1:33   ` kbuild test robot
2017-03-30 17:17 ` [PATCH 00/16] Intel FPGA Device Drivers Moritz Fischer
2017-04-06 20:27 ` Jerome Glisse
2017-04-11 19:38   ` Luebbers, Enno
2017-04-12 13:29     ` Jerome Glisse
2017-04-12 14:46       ` Moritz Fischer
2017-04-12 15:37         ` Jerome Glisse
2017-04-14 19:48           ` Luebbers, Enno
2017-04-14 20:49             ` Jerome Glisse
2017-04-17 15:35               ` Alan Tull
2017-04-17 15:57                 ` Jerome Glisse
2017-04-17 16:22                   ` Alan Tull
2017-04-17 17:15                     ` Jerome Glisse
2017-04-18 13:36                   ` Alan Cox
2017-04-18 14:59                     ` Jerome Glisse
2017-04-25 20:02                       ` One Thousand Gnomes
2017-05-01 16:41                         ` Jerome Glisse

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CANk1AXTPM+9Xdahw2sy4kSJ-Dket3Z5eS83AvW4agwSvvoFSjQ@mail.gmail.com \
    --to=atull@kernel.org \
    --cc=enno.luebbers@intel.com \
    --cc=guangrong.xiao@linux.intel.com \
    --cc=hao.wu@intel.com \
    --cc=linux-fpga@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=luwei.kang@intel.com \
    --cc=matthew.gerlach@linux.intel.com \
    --cc=moritz.fischer@ettus.com \
    --cc=yi.z.zhang@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).