[PATCH v3 00/21] Intel FPGA Device Drivers

[PATCH v3 00/21] Intel FPGA Device Drivers

[Wu Hao]

Hi All,

Here is v3 patch-series adding drivers for Intel FPGA devices.

The Intel FPGA driver provides interfaces for userspace applications to
configure, enumerate, open, and access FPGA accelerators on platforms
equipped with Intel(R) PCIe based FPGA solutions and enables system
level management functions such as FPGA partial reconfiguration, power
management and virtualization. It also provides a common framework to
support FPGA devices which implement Device Feature List (DFL) and feature
devices under DFL for better code reuse.

This patch series only adds the basic functions for FPGA accelerators and
partial reconfiguration. Patches for more functions, e.g power management
and virtualization, will be submitted after this series gets reviewed.

Patch 1: add a document for Intel FPGA driver overview, including the HW
architecture, driver organization, device enumeration, virtualization and
opens.

Patch 2: add region_id for fpga_image_info data structure, which allows
driver to pass region id information to fpga-mgr for FPGA reconfiguration
function. (Used by Patch 13)

Patch 3: add a 'status' sysfs interface to fpga-mgr class, it reflects
the status of the fpga-mgr including reconfiguration errors. (Used by
Patch 14)

Patch 4-7: add FPGA device feature list support, it provides common
enumeration interfaces which creates container device (FPGA base region)
and all feature devices by walking through all the 'Device Feature Lists'
provided low level drivers.

Patch 8-9: implement Intel FPGA PCIe device driver. It locates all 'Device
Feature Lists' in PCIe device memory and invokes common interfaces from
above device feature list framework to finish the enumeration.

Patch 10-13: implement FPGA Management Engine (FME) driver. It's a
platform driver matching with the FME platform device created by above
device feature list framework during enumeration. Sysfs and device file
ioctls are exposed as user interfaces to allow partial reconfiguration
to Accelerated Function Units (AFUs) from user space applications.

Patch 14-16: implement FPGA manager/bridge/region platform drivers for
Intel FPGA Management Engine (FME). These platform drivers match with
platform devices created by above FME driver, they use the generic
fpga-mgr/bridge/region class infrastructure to implement FPGA partial
reconfiguration function.

Patch 17-21: implement FPGA Accelerated Function Unit (AFU) driver.
It's a platform driver matching with AFU platform device created by above
device feature list framework during enumeration. It provides user
interfaces to expose the AFU MMIO region, map/unmap dma buffer, and
control the port which AFU connects to.

Changes from v2:
- Split common enumeration code from pcie driver to a separated module
  which for device feature list support.
- Drop fpga-dev class and switch to use fpga base region as container.
- Update the intel-fpga.txt documentation for new driver organization.
- Rename feature device drivers for future code reuse.
- Rebase code due to fpga APIs changes
- replace bitfields with marco and shift.
- fix typos, checkpatch issue and other comments.

Changes from v1:

- Use GPLv2 license instead of Dual BSD/GPL.
- Move the code to drivers/fpga folder.
- Update the intel-fpga.txt documentation for new driver organization.
- Add documentation for new sysfs interfaces.
- Switch to use common fpga-region interface for partial reconfiguration
  (PR) function in FME. It creates fpga-region/fpga-mgr/fpga-bridge
  platform devices and leave the implementation to their platform drivers.
- Add platform drivers for FME fpga-mgr/bridge/region platform devices.
- Fix kbuild warnings, typos and other comments.

This patch series depends on the below patchset from Alan Tull.
[PATCH v2 0/5] fpga: don't use drvdata in common fpga code[1]

[1] https://marc.info/?l=linux-fpga&m=151077942606263&w=2

Kang Luwei (3):
  fpga: dfl: add FPGA Management Engine driver basic framework
  fpga: dfl: fme: add header sub feature support
  fpga: dfl: fme: add partial reconfiguration sub feature support

Wu Hao (15):
  docs: fpga: add a document for Intel FPGA driver overview
  fpga: mgr: add region_id to fpga_image_info
  fpga: mgr: add status for fpga-manager
  fpga: add device feature list support
  fpga: dfl: add chardev support for feature devices
  fpga: dfl: adds fpga_cdev_find_port
  fpga: intel-dfl-pci: add enumeration for feature devices
  fpga: dfl: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls
    support
  fpga: dfl: add fpga manager platform driver for FME
  fpga: dfl: add fpga bridge platform driver for FME
  fpga: dfl: add fpga region platform driver for FME
  fpga: dfl: add FPGA Accelerated Function Unit driver basic framework
  fpga: dfl: afu: add header sub feature support
  fpga: dfl: afu: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls
    support
  fpga: dfl: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

Xiao Guangrong (2):
  fpga: dfl: add feature device infrastructure
  fpga: dfl: afu: add user afu sub feature support

Zhang Yi (1):
  fpga: add Intel FPGA DFL PCIe device

 Documentation/ABI/testing/sysfs-class-fpga-manager |   10 +
 .../ABI/testing/sysfs-platform-fpga-dfl-afu        |   16 +
 .../ABI/testing/sysfs-platform-fpga-dfl-fme        |   21 +
 .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |    8 +
 Documentation/fpga/intel-fpga.txt                  |  261 +++++
 Documentation/ioctl/ioctl-number.txt               |    1 +
 drivers/fpga/Kconfig                               |   71 ++
 drivers/fpga/Makefile                              |   14 +
 drivers/fpga/dfl-afu-dma-region.c                  |  465 +++++++++
 drivers/fpga/dfl-afu-main.c                        |  478 +++++++++
 drivers/fpga/dfl-afu-region.c                      |  127 +++
 drivers/fpga/dfl-afu.h                             |   72 ++
 drivers/fpga/dfl-fme-main.c                        |  275 +++++
 drivers/fpga/dfl-fme-pr.c                          |  492 +++++++++
 drivers/fpga/dfl-fme.h                             |   57 ++
 drivers/fpga/fpga-dfl-fme-br.c                     |   87 ++
 drivers/fpga/fpga-dfl-fme-mgr.c                    |  318 ++++++
 drivers/fpga/fpga-dfl-fme-region.c                 |   92 ++
 drivers/fpga/fpga-dfl.c                            | 1068 ++++++++++++++++++++
 drivers/fpga/fpga-dfl.h                            |  511 ++++++++++
 drivers/fpga/fpga-mgr.c                            |   28 +
 drivers/fpga/intel-dfl-pci.c                       |  324 ++++++
 include/linux/fpga/fpga-mgr.h                      |   19 +
 include/uapi/linux/fpga-dfl.h                      |  178 ++++
 24 files changed, 4993 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
 create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme
 create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
 create mode 100644 Documentation/fpga/intel-fpga.txt
 create mode 100644 drivers/fpga/dfl-afu-dma-region.c
 create mode 100644 drivers/fpga/dfl-afu-main.c
 create mode 100644 drivers/fpga/dfl-afu-region.c
 create mode 100644 drivers/fpga/dfl-afu.h
 create mode 100644 drivers/fpga/dfl-fme-main.c
 create mode 100644 drivers/fpga/dfl-fme-pr.c
 create mode 100644 drivers/fpga/dfl-fme.h
 create mode 100644 drivers/fpga/fpga-dfl-fme-br.c
 create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
 create mode 100644 drivers/fpga/fpga-dfl-fme-region.c
 create mode 100644 drivers/fpga/fpga-dfl.c
 create mode 100644 drivers/fpga/fpga-dfl.h
 create mode 100644 drivers/fpga/intel-dfl-pci.c
 create mode 100644 include/uapi/linux/fpga-dfl.h

-- 
1.8.3.1

[PATCH v3 01/21] docs: fpga: add a document for Intel FPGA driver overview

[Wu Hao]

Add a document for Intel FPGA driver overview.

Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
    updated open discussion per current implementation.
    fixed some typos.
v3: use FPGA base region as container device instead of fpga-dev class.
    split common enumeration code from pcie driver to functions exposed by
    device feature list framework.
    update FME performance reporting which supports both integrated (iperf/)
    and discrete (dperf/) FPGA solutions.
---
 Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 261 insertions(+)
 create mode 100644 Documentation/fpga/intel-fpga.txt

diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
new file mode 100644
index 0000000..0754733
--- /dev/null
+++ b/Documentation/fpga/intel-fpga.txt
@@ -0,0 +1,261 @@
+===============================================================================
+                    Intel FPGA driver Overview
+-------------------------------------------------------------------------------
+                Enno Luebbers <enno.luebbers@intel.com>
+                Xiao Guangrong <guangrong.xiao@linux.intel.com>
+                Wu Hao <hao.wu@intel.com>
+
+The Intel FPGA driver provides interfaces for userspace applications to
+configure, enumerate, open, and access FPGA accelerators on platforms equipped
+with Intel(R) FPGA PCIe based solutions and enables system level management
+functions such as FPGA reconfiguration, power management, and virtualization.
+
+HW Architecture
+===============
+From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
+The FPGA device memory is organized using a predefined data structure (Device
+Feature List). Features supported by the particular FPGA device are exposed
+through these data structures, as illustrated below:
+
+  +-------------------------------+  +-------------+
+  |              PF               |  |     VF      |
+  +-------------------------------+  +-------------+
+      ^            ^         ^              ^
+      |            |         |              |
++-----|------------|---------|--------------|-------+
+|     |            |         |              |       |
+|  +-----+     +-------+ +-------+      +-------+   |
+|  | FME |     | Port0 | | Port1 |      | Port2 |   |
+|  +-----+     +-------+ +-------+      +-------+   |
+|                  ^         ^              ^       |
+|                  |         |              |       |
+|              +-------+ +------+       +-------+   |
+|              |  AFU  | |  AFU |       |  AFU  |   |
+|              +-------+ +------+       +-------+   |
+|                                                   |
+|                 FPGA PCIe Device                  |
++---------------------------------------------------+
+
+The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
+used to assign individual accelerators to virtual machines.
+
+FME (FPGA Management Engine)
+============================
+The FPGA Management Engine performs power and thermal management, error
+reporting, reconfiguration, performance reporting for integrated and discrete
+solution, and other infrastructure functions. Each FPGA has one FME, which is
+always accessed through the physical function (PF).
+
+User-space applications can acquire exclusive access to the FME using open(),
+and release it using close().
+
+The following functions are exposed through ioctls:
+
+	Get driver API version (FPGA_GET_API_VERSION)
+	Check for extensions (FPGA_CHECK_EXTENSION)
+	Assign port to PF (FPGA_FME_PORT_ASSIGN)
+	Release port from PF (FPGA_FME_PORT_RELEASE)
+	Program bitstream (FPGA_FME_PORT_PR)
+
+More functions are exposed through sysfs
+(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/):
+
+	Read bitstream ID (bitstream_id)
+	Read bitstream metadata (bitstream_metadata)
+	Read number of ports (ports_num)
+	Read socket ID (socket_id)
+	Read performance counters (iperf/ and dperf/)
+	Power management (power_mgmt/)
+	Thermal management (thermal_mgmt/)
+	Error reporting (errors/)
+
+PORT
+====
+A port represents the interface between the static FPGA fabric (the "blue
+bitstream") and a partially reconfigurable region containing an AFU (the "green
+bitstream"). It controls the communication from SW to the accelerator and
+exposes features such as reset and debug.
+
+A PCIe device may have several ports and each port can be released from PF by
+FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
+sysfs interface.
+
+AFU
+===
+An AFU is attached to a port and exposes a 256k MMIO region to be used for
+accelerator-specific control registers.
+
+User-space applications can acquire exclusive access to an AFU attached to a
+port by using open() on the port device node, and release it using close().
+
+The following functions are exposed through ioctls:
+
+	Get driver API version (FPGA_GET_API_VERSION)
+	Check for extensions (FPGA_CHECK_EXTENSION)
+	Get port info (FPGA_PORT_GET_INFO)
+	Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
+	Map DMA buffer (FPGA_PORT_DMA_MAP)
+	Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
+	Reset AFU (FPGA_PORT_RESET)
+	Enable UMsg (FPGA_PORT_UMSG_ENABLE)
+	Disable UMsg (FPGA_PORT_UMSG_DISABLE)
+	Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
+	Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
+
+User-space applications can also mmap() accelerator MMIO regions.
+
+More functions are exposed through sysfs:
+(/sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/):
+
+	Read Accelerator GUID (afu_id)
+	Error reporting (errors/)
+
+Partial Reconfiguration
+=======================
+As mentioned above, accelerators can be reconfigured through partial
+reconfiguration of a green bitstream file (GBS). The green bitstream must have
+been generated for the exact blue bitstream and targeted reconfigurable region
+(port) of the FPGA; otherwise, the reconfiguration operation will fail and
+possibly cause system instability. This compatibility can be checked by
+comparing the interface ID noted in the GBS header against the interface ID
+exposed by the FME through sysfs (see above). This check is usually done by
+user-space before calling the reconfiguration IOCTL.
+
+FPGA virtualization
+===================
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+ a) The PF owns all AFU ports by default. Any port that needs to be reassigned
+ to a VF must first be released through the FPGA_FME_PORT_RELEASE ioctl on the
+ FME device.
+
+ b) Once N ports are released from PF, then user can use command below to
+ enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+ echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+ c) Pass through the VFs to VMs
+
+ d) The AFU under VF is accessible from applications in VM (using the same
+ driver inside the VF).
+
+Note that an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+
+Driver organization
+===================
+
+  +-------++------++------+             |
+  | FME   || FME  || FME  |             |
+  | FPGA  || FPGA || FPGA |             |
+  |Manager||Bridge||Region|             |
+  +-------++------++------+             |
+  +-----------------------+  +--------+ |             +--------+
+  |          FME          |  |  AFU   | |             |  AFU   |
+  |         Module        |  | Module | |             | Module |
+  +-----------------------+  +--------+ |             +--------+
+        +-----------------------+       |       +-----------------------+
+        | FPGA Container Device |       |       | FPGA Container Device |
+        |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
+        +-----------------------+       |       +-----------------------+
+          +------------------+          |         +------------------+
+          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
+          +------------------+   Host   | Machine +------------------+
+ -------------------------------------- | ------------------------------
+           +---------------+            |          +---------------+
+           | PCI PF Device |            |          | PCI VF Device |
+           +---------------+            |          +---------------+
+
+The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
+driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
+driver plays an infrastructural role in the driver architecture.  It:
+
+	a) locates the Device Feature Lists in PCIE device BAR memory, handles
+	   them and related resources to common interfaces from DFL framework
+	   for enumeration.
+	b) supports SRIOV.
+
+The feature device infrastructure provides common interfaces to create container
+device (FPGA base region), discover feature devices and their sub features from
+the given Device Feature Lists, and create platform devices for feature devices
+with related resources under the container device. It also abstracts operations
+for sub features and exposes common interfaces to feature device drivers.
+
+The FPGA Management Engine (FME) driver is a platform driver which is loaded
+automatically after FME platform device creation from the PCIE driver. It
+provides the key features for FPGA management, including:
+
+	a) Power and thermal management, error reporting, performance reporting
+	   and other infrastructure functions. Users can access these functions
+	   via sysfs interfaces exposed by FME driver.
+	b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
+	   bridges and FPGA regions during PR sub feature initialization; Once
+	   it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the
+	   common interface function from FPGA Region to complete the partial
+	   reconfiguration of the bitstream to the given port.
+	c) Port management for virtualization. The FME driver introduces two
+	   ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
+	   FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
+	   released from the PF, it can be assigned to the VF through the SRIOV
+	   interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
+	   for more details).
+
+Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
+is probed once the AFU platform device is created. The main function of this
+module is to provide an interface for userspace applications to access the
+individual accelerators, including basic reset control on port, AFU MMIO region
+export, dma buffer mapping service, UMsg notification, and remote debug
+functions (see above).
+
+
+Device enumeration
+==================
+This section introduces how applications enumerate the fpga device from
+the sysfs hierarchy under /sys/class/fpga_region.
+
+In the example below, two Intel(R) FPGA devices are installed in the host. Each
+fpga device has one FME and two ports (AFUs).
+
+FPGA regions are created under /sys/class/fpga_region/
+
+	/sys/class/fpga_region/region0
+	/sys/class/fpga_region/region1
+	/sys/class/fpga_region/region2
+	...
+
+Application needs to search each regionX folder, if feature device is found,
+(e.g "fpga-dfl-port.n" or "fpga-dfl-fme.m" is found), then it's the base
+fpga region which represents the FPGA device.
+
+Each base region has one FME and two ports (AFUs) as child devices:
+
+	/sys/class/fpga_region/region0/fpga-dfl-fme.0
+	/sys/class/fpga_region/region0/fpga-dfl-port.0
+	/sys/class/fpga_region/region0/fpga-dfl-port.1
+	...
+
+	/sys/class/fpga_region/region3/fpga-dfl-fme.1
+	/sys/class/fpga_region/region3/fpga-dfl-port.2
+	/sys/class/fpga_region/region3/fpga-dfl-port.3
+	...
+
+In general, the FME/AFU sysfs interfaces are named as follows:
+
+	/sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/
+	/sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/
+
+with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
+ports.
+
+The device nodes used for ioctl() or mmap() can be referenced through:
+
+	/sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/dev
+	/sys/class/fpga_region/<regionX>/<fpga-dfl-port.n>/dev
+
+Open discussion
+===============
+FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
+user now. In the future, if unified user interfaces for reconfiguration are
+added, FME driver should switch to them from ioctl interface.
-- 
1.8.3.1

Re: [PATCH v3 01/21] docs: fpga: add a document for Intel FPGA driver overview

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> Add a document for Intel FPGA driver overview.
>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
>     updated open discussion per current implementation.
>     fixed some typos.
> v3: use FPGA base region as container device instead of fpga-dev class.
>     split common enumeration code from pcie driver to functions exposed by
>     device feature list framework.
>     update FME performance reporting which supports both integrated (iperf/)
>     and discrete (dperf/) FPGA solutions.
> ---
>  Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 261 insertions(+)
>  create mode 100644 Documentation/fpga/intel-fpga.txt
>
> diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> new file mode 100644
> index 0000000..0754733
> --- /dev/null
> +++ b/Documentation/fpga/intel-fpga.txt
> @@ -0,0 +1,261 @@
> +===============================================================================
> +                    Intel FPGA driver Overview

This doesn't look Intel specific to me.  This could all be 'DFL FPGA Framework'

> +-------------------------------------------------------------------------------
> +                Enno Luebbers <enno.luebbers@intel.com>
> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> +                Wu Hao <hao.wu@intel.com>
> +
> +The Intel FPGA driver provides interfaces for userspace applications to
> +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> +with Intel(R) FPGA PCIe based solutions and enables system level management
> +functions such as FPGA reconfiguration, power management, and virtualization.
> +
> +HW Architecture
> +===============
> +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> +The FPGA device memory is organized using a predefined data structure (Device
> +Feature List). Features supported by the particular FPGA device are exposed
> +through these data structures, as illustrated below:
> +
> +  +-------------------------------+  +-------------+
> +  |              PF               |  |     VF      |
> +  +-------------------------------+  +-------------+
> +      ^            ^         ^              ^
> +      |            |         |              |
> ++-----|------------|---------|--------------|-------+
> +|     |            |         |              |       |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|                  ^         ^              ^       |
> +|                  |         |              |       |
> +|              +-------+ +------+       +-------+   |
> +|              |  AFU  | |  AFU |       |  AFU  |   |
> +|              +-------+ +------+       +-------+   |
> +|                                                   |
> +|                 FPGA PCIe Device                  |
> ++---------------------------------------------------+
> +
> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> +used to assign individual accelerators to virtual machines.
> +
> +FME (FPGA Management Engine)
> +============================
> +The FPGA Management Engine performs power and thermal management, error
> +reporting, reconfiguration, performance reporting for integrated and discrete
> +solution, and other infrastructure functions. Each FPGA has one FME, which is
> +always accessed through the physical function (PF).
> +
> +User-space applications can acquire exclusive access to the FME using open(),
> +and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +       Get driver API version (FPGA_GET_API_VERSION)
> +       Check for extensions (FPGA_CHECK_EXTENSION)
> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> +       Release port from PF (FPGA_FME_PORT_RELEASE)
> +       Program bitstream (FPGA_FME_PORT_PR)
> +
> +More functions are exposed through sysfs
> +(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/):

I see that /sys/class/fpga/* has changed to /sys/class/fpga_region/*
now as requested (thanks!).  It looks like it ended up being pretty
straightforward (so far, just diffing this doc with the previous v2).

> +
> +       Read bitstream ID (bitstream_id)
> +       Read bitstream metadata (bitstream_metadata)
> +       Read number of ports (ports_num)
> +       Read socket ID (socket_id)
> +       Read performance counters (iperf/ and dperf/)
> +       Power management (power_mgmt/)
> +       Thermal management (thermal_mgmt/)
> +       Error reporting (errors/)
> +
> +PORT
> +====
> +A port represents the interface between the static FPGA fabric (the "blue
> +bitstream") and a partially reconfigurable region containing an AFU (the "green
> +bitstream"). It controls the communication from SW to the accelerator and
> +exposes features such as reset and debug.
> +
> +A PCIe device may have several ports and each port can be released from PF by
> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> +sysfs interface.
> +
> +AFU
> +===
> +An AFU is attached to a port and exposes a 256k MMIO region to be used for
> +accelerator-specific control registers.
> +
> +User-space applications can acquire exclusive access to an AFU attached to a
> +port by using open() on the port device node, and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +       Get driver API version (FPGA_GET_API_VERSION)
> +       Check for extensions (FPGA_CHECK_EXTENSION)
> +       Get port info (FPGA_PORT_GET_INFO)
> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> +       Reset AFU (FPGA_PORT_RESET)
> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> +
> +User-space applications can also mmap() accelerator MMIO regions.
> +
> +More functions are exposed through sysfs:
> +(/sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/):
> +
> +       Read Accelerator GUID (afu_id)
> +       Error reporting (errors/)
> +
> +Partial Reconfiguration
> +=======================
> +As mentioned above, accelerators can be reconfigured through partial
> +reconfiguration of a green bitstream file (GBS). The green bitstream must have
> +been generated for the exact blue bitstream and targeted reconfigurable region
> +(port) of the FPGA; otherwise, the reconfiguration operation will fail and
> +possibly cause system instability. This compatibility can be checked by
> +comparing the interface ID noted in the GBS header against the interface ID
> +exposed by the FME through sysfs (see above). This check is usually done by
> +user-space before calling the reconfiguration IOCTL.
> +
> +FPGA virtualization
> +===================
> +To enable accessing an accelerator from applications running in a VM, the
> +respective AFU's port needs to be assigned to a VF using the following steps:
> +
> + a) The PF owns all AFU ports by default. Any port that needs to be reassigned
> + to a VF must first be released through the FPGA_FME_PORT_RELEASE ioctl on the
> + FME device.
> +
> + b) Once N ports are released from PF, then user can use command below to
> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> +
> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> +
> + c) Pass through the VFs to VMs
> +
> + d) The AFU under VF is accessible from applications in VM (using the same
> + driver inside the VF).
> +
> +Note that an FME can't be assigned to a VF, thus PR and other management
> +functions are only available via the PF.
> +
> +
> +Driver organization
> +===================
> +
> +  +-------++------++------+             |
> +  | FME   || FME  || FME  |             |
> +  | FPGA  || FPGA || FPGA |             |
> +  |Manager||Bridge||Region|             |
> +  +-------++------++------+             |
> +  +-----------------------+  +--------+ |             +--------+
> +  |          FME          |  |  AFU   | |             |  AFU   |
> +  |         Module        |  | Module | |             | Module |
> +  +-----------------------+  +--------+ |             +--------+
> +        +-----------------------+       |       +-----------------------+
> +        | FPGA Container Device |       |       | FPGA Container Device |
> +        |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
> +        +-----------------------+       |       +-----------------------+
> +          +------------------+          |         +------------------+
> +          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
> +          +------------------+   Host   | Machine +------------------+
> + -------------------------------------- | ------------------------------
> +           +---------------+            |          +---------------+
> +           | PCI PF Device |            |          | PCI VF Device |
> +           +---------------+            |          +---------------+
> +
> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
> +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
> +driver plays an infrastructural role in the driver architecture.  It:
> +
> +       a) locates the Device Feature Lists in PCIE device BAR memory, handles
> +          them and related resources to common interfaces from DFL framework
> +          for enumeration.
> +       b) supports SRIOV.
> +
> +The feature device infrastructure provides common interfaces to create container
> +device (FPGA base region), discover feature devices and their sub features from
> +the given Device Feature Lists, and create platform devices for feature devices
> +with related resources under the container device. It also abstracts operations
> +for sub features and exposes common interfaces to feature device drivers.
> +
> +The FPGA Management Engine (FME) driver is a platform driver which is loaded
> +automatically after FME platform device creation from the PCIE driver. It
> +provides the key features for FPGA management, including:
> +
> +       a) Power and thermal management, error reporting, performance reporting
> +          and other infrastructure functions. Users can access these functions
> +          via sysfs interfaces exposed by FME driver.
> +       b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
> +          bridges and FPGA regions during PR sub feature initialization; Once
> +          it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the
> +          common interface function from FPGA Region to complete the partial
> +          reconfiguration of the bitstream to the given port.
> +       c) Port management for virtualization. The FME driver introduces two
> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
> +          released from the PF, it can be assigned to the VF through the SRIOV
> +          interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
> +          for more details).
> +
> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
> +is probed once the AFU platform device is created. The main function of this
> +module is to provide an interface for userspace applications to access the
> +individual accelerators, including basic reset control on port, AFU MMIO region
> +export, dma buffer mapping service, UMsg notification, and remote debug
> +functions (see above).
> +
> +
> +Device enumeration
> +==================
> +This section introduces how applications enumerate the fpga device from
> +the sysfs hierarchy under /sys/class/fpga_region.
> +
> +In the example below, two Intel(R) FPGA devices are installed in the host. Each
> +fpga device has one FME and two ports (AFUs).
> +
> +FPGA regions are created under /sys/class/fpga_region/
> +
> +       /sys/class/fpga_region/region0
> +       /sys/class/fpga_region/region1
> +       /sys/class/fpga_region/region2
> +       ...
> +
> +Application needs to search each regionX folder, if feature device is found,
> +(e.g "fpga-dfl-port.n" or "fpga-dfl-fme.m" is found), then it's the base
> +fpga region which represents the FPGA device.
> +
> +Each base region has one FME and two ports (AFUs) as child devices:
> +
> +       /sys/class/fpga_region/region0/fpga-dfl-fme.0
> +       /sys/class/fpga_region/region0/fpga-dfl-port.0
> +       /sys/class/fpga_region/region0/fpga-dfl-port.1
> +       ...
> +
> +       /sys/class/fpga_region/region3/fpga-dfl-fme.1
> +       /sys/class/fpga_region/region3/fpga-dfl-port.2
> +       /sys/class/fpga_region/region3/fpga-dfl-port.3
> +       ...
> +
> +In general, the FME/AFU sysfs interfaces are named as follows:
> +
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/
> +
> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
> +ports.
> +
> +The device nodes used for ioctl() or mmap() can be referenced through:
> +
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/dev
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-port.n>/dev
> +
> +Open discussion
> +===============
> +FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
> +user now. In the future, if unified user interfaces for reconfiguration are
> +added, FME driver should switch to them from ioctl interface.
> --
> 1.8.3.1
>

Re: [PATCH v3 01/21] docs: fpga: add a document for Intel FPGA driver overview

[Wu Hao]

On Mon, Dec 04, 2017 at 01:55:37PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > Add a document for Intel FPGA driver overview.
> >
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ----
> > v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
> >     updated open discussion per current implementation.
> >     fixed some typos.
> > v3: use FPGA base region as container device instead of fpga-dev class.
> >     split common enumeration code from pcie driver to functions exposed by
> >     device feature list framework.
> >     update FME performance reporting which supports both integrated (iperf/)
> >     and discrete (dperf/) FPGA solutions.
> > ---
> >  Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 261 insertions(+)
> >  create mode 100644 Documentation/fpga/intel-fpga.txt
> >
> > diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> > new file mode 100644
> > index 0000000..0754733
> > --- /dev/null
> > +++ b/Documentation/fpga/intel-fpga.txt
> > @@ -0,0 +1,261 @@
> > +===============================================================================
> > +                    Intel FPGA driver Overview
> 
> This doesn't look Intel specific to me.  This could all be 'DFL FPGA Framework'

Sure, will rename this doc to dfl-fpga.txt in the next version as we plan to rename
the pcie driver to dfl-pci per your comments on the patch #8, there is no reason to
keep it in this doc as all drivers will be dfl-* in the next version. :)

> 
> > +-------------------------------------------------------------------------------
> > +                Enno Luebbers <enno.luebbers@intel.com>
> > +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > +                Wu Hao <hao.wu@intel.com>
> > +
> > +The Intel FPGA driver provides interfaces for userspace applications to
> > +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> > +with Intel(R) FPGA PCIe based solutions and enables system level management
> > +functions such as FPGA reconfiguration, power management, and virtualization.
> > +
> > +HW Architecture
> > +===============
> > +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> > +The FPGA device memory is organized using a predefined data structure (Device
> > +Feature List). Features supported by the particular FPGA device are exposed
> > +through these data structures, as illustrated below:
> > +
> > +  +-------------------------------+  +-------------+
> > +  |              PF               |  |     VF      |
> > +  +-------------------------------+  +-------------+
> > +      ^            ^         ^              ^
> > +      |            |         |              |
> > ++-----|------------|---------|--------------|-------+
> > +|     |            |         |              |       |
> > +|  +-----+     +-------+ +-------+      +-------+   |
> > +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> > +|  +-----+     +-------+ +-------+      +-------+   |
> > +|                  ^         ^              ^       |
> > +|                  |         |              |       |
> > +|              +-------+ +------+       +-------+   |
> > +|              |  AFU  | |  AFU |       |  AFU  |   |
> > +|              +-------+ +------+       +-------+   |
> > +|                                                   |
> > +|                 FPGA PCIe Device                  |
> > ++---------------------------------------------------+
> > +
> > +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> > +used to assign individual accelerators to virtual machines.
> > +
> > +FME (FPGA Management Engine)
> > +============================
> > +The FPGA Management Engine performs power and thermal management, error
> > +reporting, reconfiguration, performance reporting for integrated and discrete
> > +solution, and other infrastructure functions. Each FPGA has one FME, which is
> > +always accessed through the physical function (PF).
> > +
> > +User-space applications can acquire exclusive access to the FME using open(),
> > +and release it using close().
> > +
> > +The following functions are exposed through ioctls:
> > +
> > +       Get driver API version (FPGA_GET_API_VERSION)
> > +       Check for extensions (FPGA_CHECK_EXTENSION)
> > +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> > +       Release port from PF (FPGA_FME_PORT_RELEASE)
> > +       Program bitstream (FPGA_FME_PORT_PR)
> > +
> > +More functions are exposed through sysfs
> > +(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/):
> 
> I see that /sys/class/fpga/* has changed to /sys/class/fpga_region/*
> now as requested (thanks!).  It looks like it ended up being pretty
> straightforward (so far, just diffing this doc with the previous v2).

Thanks for the suggestion on using fpga base region. :)

Hao

RE: [PATCH v3 01/21] docs: fpga: add a document for Intel FPGA driver overview

[David Laight]

From: Alan Tull
> Sent: 04 December 2017 19:56
> > +===============================================================================
> > +                    Intel FPGA driver Overview
> 
> This doesn't look Intel specific to me.  This could all be 'DFL FPGA Framework'

I've not read the details, but I susect that it is 'an Intel DFL Framework'.
(It might not even be 'the Intel DFL Framework').

	David

Re: [PATCH v3 01/21] docs: fpga: add a document for Intel FPGA driver overview

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> +
> +PORT
> +====
> +A port represents the interface between the static FPGA fabric (the "blue
> +bitstream") and a partially reconfigurable region containing an AFU (the "green
> +bitstream"). It controls the communication from SW to the accelerator and
> +exposes features such as reset and debug.

Hi Hao,

If I remember correctly, reset means that the accelerator gets reset
and this is something that is desirable to do between jobs.  I've
asked for some documentation about the port reset function, partly
because the idea of being able to reset hardware from userspace
somehow scares me.  So please find a good logical place to explain
what a port reset does and how it is safe for userspace to request it
at some arbitrary time and how it won't crash the kernel.  We
discussed this in v2, I grepped v3 for it, maybe I missed it, but I
don't see it in v3.  My understanding is that disabling and reenabling
the port bridge causes the accelerator in its FPGA region to get
reset.

Alan

> +
> +A PCIe device may have several ports and each port can be released from PF by
> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> +sysfs interface.
> +

Re: [PATCH v3 01/21] docs: fpga: add a document for Intel FPGA driver overview

[Wu Hao]

On Wed, Dec 20, 2017 at 04:31:15PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > +
> > +PORT
> > +====
> > +A port represents the interface between the static FPGA fabric (the "blue
> > +bitstream") and a partially reconfigurable region containing an AFU (the "green
> > +bitstream"). It controls the communication from SW to the accelerator and
> > +exposes features such as reset and debug.
> 
> Hi Hao,
> 
> If I remember correctly, reset means that the accelerator gets reset
> and this is something that is desirable to do between jobs.  I've
> asked for some documentation about the port reset function, partly
> because the idea of being able to reset hardware from userspace
> somehow scares me.  So please find a good logical place to explain
> what a port reset does and how it is safe for userspace to request it
> at some arbitrary time and how it won't crash the kernel.  We
> discussed this in v2, I grepped v3 for it, maybe I missed it, but I
> don't see it in v3.  My understanding is that disabling and reenabling
> the port bridge causes the accelerator in its FPGA region to get
> reset.

Hi Alan

Yes, that's correct.

Some descriptions are added in Patch#18[1] when introduced the reset ioctl.
I will add some descriptions in the doc as well.

[1]https://marc.info/?l=linux-fpga&m=151176566714744&w=2

@@ -50,6 +53,20 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for AFU file descriptor */
+
+/**
+ * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
+ *
+ * Reset the FPGA Port and its AFU. No parameters are supported.
+ * Userspace can do Port reset at any time, e.g during DMA or PR. But
+ * it should never cause any system level issue, only functional failure
+ * (e.g DMA or PR operation failure) and be recoverable from the failure.
+ * Return: 0 on success, -errno of failure
+ */
+
+#define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
+

Thanks
Hao

> 
> Alan
> 
> > +
> > +A PCIe device may have several ports and each port can be released from PF by
> > +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> > +sysfs interface.
> > +
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 02/21] fpga: mgr: add region_id to fpga_image_info

[Wu Hao]

This patch adds region_id to fpga_image_info data structure, it
allows driver to pass region id information to fpga-mgr via
fpga_image_info for fpga reconfiguration function.

Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: add one line comment for region_id
---
 include/linux/fpga/fpga-mgr.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
index 1ad4822..6e98b66 100644
--- a/include/linux/fpga/fpga-mgr.h
+++ b/include/linux/fpga/fpga-mgr.h
@@ -88,6 +88,7 @@ enum fpga_mgr_states {
  * @sgt: scatter/gather table containing FPGA image
  * @buf: contiguous buffer containing FPGA image
  * @count: size of buf
+ * @region_id: id of target region
  * @dev: device that owns this
  * @overlay: Device Tree overlay
  */
@@ -100,6 +101,7 @@ struct fpga_image_info {
 	struct sg_table *sgt;
 	const char *buf;
 	size_t count;
+	int region_id;
 	struct device *dev;
 #ifdef CONFIG_OF
 	struct device_node *overlay;
-- 
1.8.3.1

Re: [PATCH v3 02/21] fpga: mgr: add region_id to fpga_image_info

[Moritz Fischer]

Hi Hao,

On Mon, Nov 27, 2017 at 02:42:09PM +0800, Wu Hao wrote:
> This patch adds region_id to fpga_image_info data structure, it
> allows driver to pass region id information to fpga-mgr via
> fpga_image_info for fpga reconfiguration function.
> 
> Signed-off-by: Wu Hao <hao.wu@intel.com>
Acked-by: Moritz Fischer <mdf@kernel.org>
> ----
> v3: add one line comment for region_id
> ---
>  include/linux/fpga/fpga-mgr.h | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
> index 1ad4822..6e98b66 100644
> --- a/include/linux/fpga/fpga-mgr.h
> +++ b/include/linux/fpga/fpga-mgr.h
> @@ -88,6 +88,7 @@ enum fpga_mgr_states {
>   * @sgt: scatter/gather table containing FPGA image
>   * @buf: contiguous buffer containing FPGA image
>   * @count: size of buf
> + * @region_id: id of target region
>   * @dev: device that owns this
>   * @overlay: Device Tree overlay
>   */
> @@ -100,6 +101,7 @@ struct fpga_image_info {
>  	struct sg_table *sgt;
>  	const char *buf;
>  	size_t count;
> +	int region_id;
>  	struct device *dev;
>  #ifdef CONFIG_OF
>  	struct device_node *overlay;
> -- 
> 1.8.3.1
> 

Moritz

Re: [PATCH v3 02/21] fpga: mgr: add region_id to fpga_image_info

[Alan Tull]

On Wed, Nov 29, 2017 at 12:11 AM, Moritz Fischer <mdf@kernel.org> wrote:
> Hi Hao,
>
> On Mon, Nov 27, 2017 at 02:42:09PM +0800, Wu Hao wrote:
>> This patch adds region_id to fpga_image_info data structure, it
>> allows driver to pass region id information to fpga-mgr via
>> fpga_image_info for fpga reconfiguration function.
>>
>> Signed-off-by: Wu Hao <hao.wu@intel.com>
> Acked-by: Moritz Fischer <mdf@kernel.org>

Acked-by: Alan Tull <atull@kernel.org>

>> ----
>> v3: add one line comment for region_id
>> ---
>>  include/linux/fpga/fpga-mgr.h | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
>> index 1ad4822..6e98b66 100644
>> --- a/include/linux/fpga/fpga-mgr.h
>> +++ b/include/linux/fpga/fpga-mgr.h
>> @@ -88,6 +88,7 @@ enum fpga_mgr_states {
>>   * @sgt: scatter/gather table containing FPGA image
>>   * @buf: contiguous buffer containing FPGA image
>>   * @count: size of buf
>> + * @region_id: id of target region
>>   * @dev: device that owns this
>>   * @overlay: Device Tree overlay
>>   */
>> @@ -100,6 +101,7 @@ struct fpga_image_info {
>>       struct sg_table *sgt;
>>       const char *buf;
>>       size_t count;
>> +     int region_id;
>>       struct device *dev;
>>  #ifdef CONFIG_OF
>>       struct device_node *overlay;
>> --
>> 1.8.3.1
>>
>
> Moritz

Re: [PATCH v3 02/21] fpga: mgr: add region_id to fpga_image_info

[Wu Hao]

On Mon, Dec 04, 2017 at 02:26:14PM -0600, Alan Tull wrote:
> On Wed, Nov 29, 2017 at 12:11 AM, Moritz Fischer <mdf@kernel.org> wrote:
> > Hi Hao,
> >
> > On Mon, Nov 27, 2017 at 02:42:09PM +0800, Wu Hao wrote:
> >> This patch adds region_id to fpga_image_info data structure, it
> >> allows driver to pass region id information to fpga-mgr via
> >> fpga_image_info for fpga reconfiguration function.
> >>
> >> Signed-off-by: Wu Hao <hao.wu@intel.com>
> > Acked-by: Moritz Fischer <mdf@kernel.org>
> 
> Acked-by: Alan Tull <atull@kernel.org>

Hi Alan / Moritz

Thanks for the review. :)

Hao

Re: [PATCH v3 02/21] fpga: mgr: add region_id to fpga_image_info

[Alan Tull]

On Mon, Dec 4, 2017 at 9:36 PM, Wu Hao <hao.wu@intel.com> wrote:
> On Mon, Dec 04, 2017 at 02:26:14PM -0600, Alan Tull wrote:
>> On Wed, Nov 29, 2017 at 12:11 AM, Moritz Fischer <mdf@kernel.org> wrote:
>> > Hi Hao,
>> >
>> > On Mon, Nov 27, 2017 at 02:42:09PM +0800, Wu Hao wrote:
>> >> This patch adds region_id to fpga_image_info data structure, it
>> >> allows driver to pass region id information to fpga-mgr via
>> >> fpga_image_info for fpga reconfiguration function.
>> >>
>> >> Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > Acked-by: Moritz Fischer <mdf@kernel.org>
>>
>> Acked-by: Alan Tull <atull@kernel.org>
>
> Hi Alan / Moritz
>
> Thanks for the review. :)

Just to be clear, when you post the next version, keep the Acks that
have been added.

Thanks,
Alan

>
> Hao

Re: [PATCH v3 02/21] fpga: mgr: add region_id to fpga_image_info

[Wu Hao]

On Wed, Jan 31, 2018 at 09:35:20AM -0600, Alan Tull wrote:
> On Mon, Dec 4, 2017 at 9:36 PM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Dec 04, 2017 at 02:26:14PM -0600, Alan Tull wrote:
> >> On Wed, Nov 29, 2017 at 12:11 AM, Moritz Fischer <mdf@kernel.org> wrote:
> >> > Hi Hao,
> >> >
> >> > On Mon, Nov 27, 2017 at 02:42:09PM +0800, Wu Hao wrote:
> >> >> This patch adds region_id to fpga_image_info data structure, it
> >> >> allows driver to pass region id information to fpga-mgr via
> >> >> fpga_image_info for fpga reconfiguration function.
> >> >>
> >> >> Signed-off-by: Wu Hao <hao.wu@intel.com>
> >> > Acked-by: Moritz Fischer <mdf@kernel.org>
> >>
> >> Acked-by: Alan Tull <atull@kernel.org>
> >
> > Hi Alan / Moritz
> >
> > Thanks for the review. :)
> 
> Just to be clear, when you post the next version, keep the Acks that
> have been added.

Sure, I will keep the Acks in the next version.

Thanks
Hao

[PATCH v3 03/21] fpga: mgr: add status for fpga-manager

[Wu Hao]

This patch adds status to fpga-manager data structure, to allow
driver to store full/partial reconfiguration errors and other
status information, and adds one status callback to fpga_manager_ops
to allow fpga_manager to collect latest status when failures are
detected.

The following sysfs file is created:
* /sys/class/fpga_manager/<fpga>/status
  Return status of fpga manager, including reconfiguration errors.

Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: add one line description for status
    add status callback function to fpga_manager_ops
    update fpga-mgr status if any failure or during initialization
    s/INCOMPATIBLE_BS_ERR/INCOMPATIBLE_IMAGE_ERR/
---
 Documentation/ABI/testing/sysfs-class-fpga-manager | 10 ++++++++
 drivers/fpga/fpga-mgr.c                            | 28 ++++++++++++++++++++++
 include/linux/fpga/fpga-mgr.h                      | 17 +++++++++++++
 3 files changed, 55 insertions(+)

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-manager b/Documentation/ABI/testing/sysfs-class-fpga-manager
index 23056c5..01db14d 100644
--- a/Documentation/ABI/testing/sysfs-class-fpga-manager
+++ b/Documentation/ABI/testing/sysfs-class-fpga-manager
@@ -35,3 +35,13 @@ Description:	Read fpga manager state as a string.
 		* write complete	= Doing post programming steps
 		* write complete error	= Error while doing post programming
 		* operating		= FPGA is programmed and operating
+
+What:		/sys/class/fpga_manager/<fpga>/status
+Date:		November 2017
+KernelVersion:	4.15
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read fpga manager status as a string.
+		If FPGA programming operation fails, it could be due to crc
+		error or incompatible bitstream image. The intent of this
+		interface is to provide more detailed information for FPGA
+		programming errors to userspace.
diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c
index 1fd5494..8b583ba 100644
--- a/drivers/fpga/fpga-mgr.c
+++ b/drivers/fpga/fpga-mgr.c
@@ -88,6 +88,7 @@ static int fpga_mgr_write_init_buf(struct fpga_manager *mgr,
 	if (ret) {
 		dev_err(&mgr->dev, "Error preparing FPGA for writing\n");
 		mgr->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
+		fpga_mgr_update_status(mgr);
 		return ret;
 	}
 
@@ -148,6 +149,7 @@ static int fpga_mgr_write_complete(struct fpga_manager *mgr,
 	if (ret) {
 		dev_err(&mgr->dev, "Error after writing image data to FPGA\n");
 		mgr->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
+		fpga_mgr_update_status(mgr);
 		return ret;
 	}
 	mgr->state = FPGA_MGR_STATE_OPERATING;
@@ -225,6 +227,7 @@ static int fpga_mgr_buf_load_mapped(struct fpga_manager *mgr,
 	if (ret) {
 		dev_err(&mgr->dev, "Error while writing image data to FPGA\n");
 		mgr->state = FPGA_MGR_STATE_WRITE_ERR;
+		fpga_mgr_update_status(mgr);
 		return ret;
 	}
 
@@ -397,12 +400,36 @@ static ssize_t state_show(struct device *dev,
 	return sprintf(buf, "%s\n", state_str[mgr->state]);
 }
 
+static ssize_t status_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct fpga_manager *mgr = to_fpga_manager(dev);
+	int len = 0;
+
+	if (mgr->status & FPGA_MGR_STATUS_OPERATION_ERR)
+		len += sprintf(buf + len, "reconfig operation error\n");
+	if (mgr->status & FPGA_MGR_STATUS_CRC_ERR)
+		len += sprintf(buf + len, "reconfig CRC error\n");
+	if (mgr->status & FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR)
+		len += sprintf(buf + len, "reconfig incompatible image\n");
+	if (mgr->status & FPGA_MGR_STATUS_IP_PROTOCOL_ERR)
+		len += sprintf(buf + len, "reconfig IP protocol error\n");
+	if (mgr->status & FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR)
+		len += sprintf(buf + len, "reconfig fifo overflow error\n");
+	if (mgr->status & FPGA_MGR_STATUS_SECURE_LOAD_ERR)
+		len += sprintf(buf + len, "reconfig secure load error\n");
+
+	return len;
+}
+
 static DEVICE_ATTR_RO(name);
 static DEVICE_ATTR_RO(state);
+static DEVICE_ATTR_RO(status);
 
 static struct attribute *fpga_mgr_attrs[] = {
 	&dev_attr_name.attr,
 	&dev_attr_state.attr,
+	&dev_attr_status.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(fpga_mgr);
@@ -561,6 +588,7 @@ int fpga_mgr_register(struct fpga_manager *mgr)
 	 * by bootloader or EEPROM.
 	 */
 	mgr->state = mgr->mops->state(mgr);
+	fpga_mgr_update_status(mgr);
 
 	device_initialize(&mgr->dev);
 	mgr->dev.class = fpga_mgr_class;
diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
index 6e98b66..d4faf3d 100644
--- a/include/linux/fpga/fpga-mgr.h
+++ b/include/linux/fpga/fpga-mgr.h
@@ -112,6 +112,7 @@ struct fpga_image_info {
  * struct fpga_manager_ops - ops for low level fpga manager drivers
  * @initial_header_size: Maximum number of bytes that should be passed into write_init
  * @state: returns an enum value of the FPGA's state
+ * @status: returns status of the FPGA, including reconfiguration error code
  * @write_init: prepare the FPGA to receive confuration data
  * @write: write count bytes of configuration data to the FPGA
  * @write_sg: write the scatter list of configuration data to the FPGA
@@ -126,6 +127,7 @@ struct fpga_image_info {
 struct fpga_manager_ops {
 	size_t initial_header_size;
 	enum fpga_mgr_states (*state)(struct fpga_manager *mgr);
+	u64 (*status)(struct fpga_manager *mgr);
 	int (*write_init)(struct fpga_manager *mgr,
 			  struct fpga_image_info *info,
 			  const char *buf, size_t count);
@@ -137,6 +139,14 @@ struct fpga_manager_ops {
 	const struct attribute_group **groups;
 };
 
+/* FPGA manager status: Partial/Full Reconfiguration errors */
+#define FPGA_MGR_STATUS_OPERATION_ERR		BIT(0)
+#define FPGA_MGR_STATUS_CRC_ERR			BIT(1)
+#define FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR	BIT(2)
+#define FPGA_MGR_STATUS_IP_PROTOCOL_ERR		BIT(3)
+#define FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR	BIT(4)
+#define FPGA_MGR_STATUS_SECURE_LOAD_ERR		BIT(5)
+
 /**
  * struct fpga_manager - fpga manager structure
  * @name: name of low level fpga manager
@@ -144,6 +154,7 @@ struct fpga_manager_ops {
  * @dev: fpga manager device
  * @ref_mutex: only allows one reference to fpga manager
  * @state: state of fpga manager
+ * @status: status of fpga manager, including reconfiguration error.
  * @mops: pointer to struct of fpga manager ops
  * @priv: low level driver private date
  */
@@ -153,6 +164,7 @@ struct fpga_manager {
 	struct device dev;
 	struct mutex ref_mutex;
 	enum fpga_mgr_states state;
+	u64 status;
 	const struct fpga_manager_ops *mops;
 	void *priv;
 };
@@ -177,4 +189,9 @@ struct fpga_manager {
 int fpga_mgr_register(struct fpga_manager *mgr);
 void fpga_mgr_unregister(struct fpga_manager *mgr);
 
+static inline void fpga_mgr_update_status(struct fpga_manager *mgr)
+{
+	if (mgr->mops->status)
+		mgr->status = mgr->mops->status(mgr);
+}
 #endif /*_LINUX_FPGA_MGR_H */
-- 
1.8.3.1

Re: [PATCH v3 03/21] fpga: mgr: add status for fpga-manager

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

mgr->status isn't used anywhere except in status_show.  So we don't
need to add status to the fpga_manager struct.  Also don't need the
inline function to update it.  Just read the status in status_show, if
that ops exists.  If mops->status is NULL, return an error, as below.
This will simplify things and make sure that status_show gets status
that is not stale at all.

> This patch adds status to fpga-manager data structure, to allow
> driver to store full/partial reconfiguration errors and other
> status information, and adds one status callback to fpga_manager_ops
> to allow fpga_manager to collect latest status when failures are
> detected.
>
> The following sysfs file is created:
> * /sys/class/fpga_manager/<fpga>/status
>   Return status of fpga manager, including reconfiguration errors.
>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v3: add one line description for status
>     add status callback function to fpga_manager_ops
>     update fpga-mgr status if any failure or during initialization
>     s/INCOMPATIBLE_BS_ERR/INCOMPATIBLE_IMAGE_ERR/
> ---
>  Documentation/ABI/testing/sysfs-class-fpga-manager | 10 ++++++++
>  drivers/fpga/fpga-mgr.c                            | 28 ++++++++++++++++++++++
>  include/linux/fpga/fpga-mgr.h                      | 17 +++++++++++++
>  3 files changed, 55 insertions(+)
>
> diff --git a/Documentation/ABI/testing/sysfs-class-fpga-manager b/Documentation/ABI/testing/sysfs-class-fpga-manager
> index 23056c5..01db14d 100644
> --- a/Documentation/ABI/testing/sysfs-class-fpga-manager
> +++ b/Documentation/ABI/testing/sysfs-class-fpga-manager
> @@ -35,3 +35,13 @@ Description: Read fpga manager state as a string.
>                 * write complete        = Doing post programming steps
>                 * write complete error  = Error while doing post programming
>                 * operating             = FPGA is programmed and operating
> +
> +What:          /sys/class/fpga_manager/<fpga>/status
> +Date:          November 2017
> +KernelVersion: 4.15
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read fpga manager status as a string.
> +               If FPGA programming operation fails, it could be due to crc
> +               error or incompatible bitstream image. The intent of this
> +               interface is to provide more detailed information for FPGA
> +               programming errors to userspace.
> diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c
> index 1fd5494..8b583ba 100644
> --- a/drivers/fpga/fpga-mgr.c
> +++ b/drivers/fpga/fpga-mgr.c
> @@ -88,6 +88,7 @@ static int fpga_mgr_write_init_buf(struct fpga_manager *mgr,
>         if (ret) {
>                 dev_err(&mgr->dev, "Error preparing FPGA for writing\n");
>                 mgr->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> +               fpga_mgr_update_status(mgr);

Remove these fpga_mgr_update_status

>                 return ret;
>         }
>
> @@ -148,6 +149,7 @@ static int fpga_mgr_write_complete(struct fpga_manager *mgr,
>         if (ret) {
>                 dev_err(&mgr->dev, "Error after writing image data to FPGA\n");
>                 mgr->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
> +               fpga_mgr_update_status(mgr);
>                 return ret;
>         }
>         mgr->state = FPGA_MGR_STATE_OPERATING;
> @@ -225,6 +227,7 @@ static int fpga_mgr_buf_load_mapped(struct fpga_manager *mgr,
>         if (ret) {
>                 dev_err(&mgr->dev, "Error while writing image data to FPGA\n");
>                 mgr->state = FPGA_MGR_STATE_WRITE_ERR;
> +               fpga_mgr_update_status(mgr);
>                 return ret;
>         }
>
> @@ -397,12 +400,36 @@ static ssize_t state_show(struct device *dev,
>         return sprintf(buf, "%s\n", state_str[mgr->state]);
>  }
>
> +static ssize_t status_show(struct device *dev,
> +                          struct device_attribute *attr, char *buf)
> +{
> +       struct fpga_manager *mgr = to_fpga_manager(dev);
> +       int len = 0;

u64 status;

if (!mgr->mops->status)
        return -ENOENT;

status = mgr->mops->status(mgr);

> +
> +       if (mgr->status & FPGA_MGR_STATUS_OPERATION_ERR)

if (status & ...)

> +               len += sprintf(buf + len, "reconfig operation error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_CRC_ERR)
> +               len += sprintf(buf + len, "reconfig CRC error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR)
> +               len += sprintf(buf + len, "reconfig incompatible image\n");
> +       if (mgr->status & FPGA_MGR_STATUS_IP_PROTOCOL_ERR)
> +               len += sprintf(buf + len, "reconfig IP protocol error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR)
> +               len += sprintf(buf + len, "reconfig fifo overflow error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_SECURE_LOAD_ERR)
> +               len += sprintf(buf + len, "reconfig secure load error\n");
> +
> +       return len;
> +}
> +
>  static DEVICE_ATTR_RO(name);
>  static DEVICE_ATTR_RO(state);
> +static DEVICE_ATTR_RO(status);
>
>  static struct attribute *fpga_mgr_attrs[] = {
>         &dev_attr_name.attr,
>         &dev_attr_state.attr,
> +       &dev_attr_status.attr,
>         NULL,
>  };
>  ATTRIBUTE_GROUPS(fpga_mgr);
> @@ -561,6 +588,7 @@ int fpga_mgr_register(struct fpga_manager *mgr)
>          * by bootloader or EEPROM.
>          */
>         mgr->state = mgr->mops->state(mgr);
> +       fpga_mgr_update_status(mgr);
>
>         device_initialize(&mgr->dev);
>         mgr->dev.class = fpga_mgr_class;
> diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
> index 6e98b66..d4faf3d 100644
> --- a/include/linux/fpga/fpga-mgr.h
> +++ b/include/linux/fpga/fpga-mgr.h
> @@ -112,6 +112,7 @@ struct fpga_image_info {
>   * struct fpga_manager_ops - ops for low level fpga manager drivers
>   * @initial_header_size: Maximum number of bytes that should be passed into write_init
>   * @state: returns an enum value of the FPGA's state
> + * @status: returns status of the FPGA, including reconfiguration error code
>   * @write_init: prepare the FPGA to receive confuration data
>   * @write: write count bytes of configuration data to the FPGA
>   * @write_sg: write the scatter list of configuration data to the FPGA
> @@ -126,6 +127,7 @@ struct fpga_image_info {
>  struct fpga_manager_ops {
>         size_t initial_header_size;
>         enum fpga_mgr_states (*state)(struct fpga_manager *mgr);
> +       u64 (*status)(struct fpga_manager *mgr);
>         int (*write_init)(struct fpga_manager *mgr,
>                           struct fpga_image_info *info,
>                           const char *buf, size_t count);
> @@ -137,6 +139,14 @@ struct fpga_manager_ops {
>         const struct attribute_group **groups;
>  };
>
> +/* FPGA manager status: Partial/Full Reconfiguration errors */
> +#define FPGA_MGR_STATUS_OPERATION_ERR          BIT(0)
> +#define FPGA_MGR_STATUS_CRC_ERR                        BIT(1)
> +#define FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR BIT(2)
> +#define FPGA_MGR_STATUS_IP_PROTOCOL_ERR                BIT(3)
> +#define FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR      BIT(4)
> +#define FPGA_MGR_STATUS_SECURE_LOAD_ERR                BIT(5)
> +
>  /**
>   * struct fpga_manager - fpga manager structure
>   * @name: name of low level fpga manager
> @@ -144,6 +154,7 @@ struct fpga_manager_ops {
>   * @dev: fpga manager device
>   * @ref_mutex: only allows one reference to fpga manager
>   * @state: state of fpga manager
> + * @status: status of fpga manager, including reconfiguration error.
>   * @mops: pointer to struct of fpga manager ops
>   * @priv: low level driver private date
>   */
> @@ -153,6 +164,7 @@ struct fpga_manager {
>         struct device dev;
>         struct mutex ref_mutex;
>         enum fpga_mgr_states state;
> +       u64 status;

Remove.

>         const struct fpga_manager_ops *mops;
>         void *priv;
>  };
> @@ -177,4 +189,9 @@ struct fpga_manager {
>  int fpga_mgr_register(struct fpga_manager *mgr);
>  void fpga_mgr_unregister(struct fpga_manager *mgr);
>
> +static inline void fpga_mgr_update_status(struct fpga_manager *mgr)
> +{
> +       if (mgr->mops->status)
> +               mgr->status = mgr->mops->status(mgr);
> +}

Remove fpga_mgr_update_status, not needed.

Thanks,
Alan

Re: [PATCH v3 03/21] fpga: mgr: add status for fpga-manager

[Wu Hao]

On Mon, Dec 04, 2017 at 02:55:42PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> mgr->status isn't used anywhere except in status_show.  So we don't
> need to add status to the fpga_manager struct.  Also don't need the
> inline function to update it.  Just read the status in status_show, if
> that ops exists.  If mops->status is NULL, return an error, as below.
> This will simplify things and make sure that status_show gets status
> that is not stale at all.

Hi Alan

You're right, this is much simpler. Will follow this to fix it in the
next version. Thanks!

Hao

Re: [PATCH v3 03/21] fpga: mgr: add status for fpga-manager

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> This patch adds status to fpga-manager data structure, to allow
> driver to store full/partial reconfiguration errors and other
> status information, and adds one status callback to fpga_manager_ops
> to allow fpga_manager to collect latest status when failures are
> detected.
>
> The following sysfs file is created:
> * /sys/class/fpga_manager/<fpga>/status
>   Return status of fpga manager, including reconfiguration errors.
>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v3: add one line description for status
>     add status callback function to fpga_manager_ops
>     update fpga-mgr status if any failure or during initialization
>     s/INCOMPATIBLE_BS_ERR/INCOMPATIBLE_IMAGE_ERR/
> ---
>  Documentation/ABI/testing/sysfs-class-fpga-manager | 10 ++++++++
>  drivers/fpga/fpga-mgr.c                            | 28 ++++++++++++++++++++++
>  include/linux/fpga/fpga-mgr.h                      | 17 +++++++++++++
>  3 files changed, 55 insertions(+)
>
> diff --git a/Documentation/ABI/testing/sysfs-class-fpga-manager b/Documentation/ABI/testing/sysfs-class-fpga-manager
> index 23056c5..01db14d 100644
> --- a/Documentation/ABI/testing/sysfs-class-fpga-manager
> +++ b/Documentation/ABI/testing/sysfs-class-fpga-manager
> @@ -35,3 +35,13 @@ Description: Read fpga manager state as a string.
>                 * write complete        = Doing post programming steps
>                 * write complete error  = Error while doing post programming
>                 * operating             = FPGA is programmed and operating
> +
> +What:          /sys/class/fpga_manager/<fpga>/status
> +Date:          November 2017
> +KernelVersion: 4.15
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read fpga manager status as a string.
> +               If FPGA programming operation fails, it could be due to crc
> +               error or incompatible bitstream image. The intent of this
> +               interface is to provide more detailed information for FPGA
> +               programming errors to userspace.

Hi Hao,

Please also add a list of status strings that can be given, such as
"reconfig operation error"  I understand that this list may grow
somewhat in the future.

Alan

> diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c
> index 1fd5494..8b583ba 100644
> --- a/drivers/fpga/fpga-mgr.c
> +++ b/drivers/fpga/fpga-mgr.c
> @@ -88,6 +88,7 @@ static int fpga_mgr_write_init_buf(struct fpga_manager *mgr,
>         if (ret) {
>                 dev_err(&mgr->dev, "Error preparing FPGA for writing\n");
>                 mgr->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> +               fpga_mgr_update_status(mgr);
>                 return ret;
>         }
>
> @@ -148,6 +149,7 @@ static int fpga_mgr_write_complete(struct fpga_manager *mgr,
>         if (ret) {
>                 dev_err(&mgr->dev, "Error after writing image data to FPGA\n");
>                 mgr->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
> +               fpga_mgr_update_status(mgr);
>                 return ret;
>         }
>         mgr->state = FPGA_MGR_STATE_OPERATING;
> @@ -225,6 +227,7 @@ static int fpga_mgr_buf_load_mapped(struct fpga_manager *mgr,
>         if (ret) {
>                 dev_err(&mgr->dev, "Error while writing image data to FPGA\n");
>                 mgr->state = FPGA_MGR_STATE_WRITE_ERR;
> +               fpga_mgr_update_status(mgr);
>                 return ret;
>         }
>
> @@ -397,12 +400,36 @@ static ssize_t state_show(struct device *dev,
>         return sprintf(buf, "%s\n", state_str[mgr->state]);
>  }
>
> +static ssize_t status_show(struct device *dev,
> +                          struct device_attribute *attr, char *buf)
> +{
> +       struct fpga_manager *mgr = to_fpga_manager(dev);
> +       int len = 0;
> +
> +       if (mgr->status & FPGA_MGR_STATUS_OPERATION_ERR)
> +               len += sprintf(buf + len, "reconfig operation error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_CRC_ERR)
> +               len += sprintf(buf + len, "reconfig CRC error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR)
> +               len += sprintf(buf + len, "reconfig incompatible image\n");
> +       if (mgr->status & FPGA_MGR_STATUS_IP_PROTOCOL_ERR)
> +               len += sprintf(buf + len, "reconfig IP protocol error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR)
> +               len += sprintf(buf + len, "reconfig fifo overflow error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_SECURE_LOAD_ERR)
> +               len += sprintf(buf + len, "reconfig secure load error\n");
> +
> +       return len;
> +}

Re: [PATCH v3 03/21] fpga: mgr: add status for fpga-manager

[Wu Hao]

On Tue, Dec 12, 2017 at 12:18:06PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > This patch adds status to fpga-manager data structure, to allow
> > driver to store full/partial reconfiguration errors and other
> > status information, and adds one status callback to fpga_manager_ops
> > to allow fpga_manager to collect latest status when failures are
> > detected.
> >
> > The following sysfs file is created:
> > * /sys/class/fpga_manager/<fpga>/status
> >   Return status of fpga manager, including reconfiguration errors.
> >
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ----
> > v3: add one line description for status
> >     add status callback function to fpga_manager_ops
> >     update fpga-mgr status if any failure or during initialization
> >     s/INCOMPATIBLE_BS_ERR/INCOMPATIBLE_IMAGE_ERR/
> > ---
> >  Documentation/ABI/testing/sysfs-class-fpga-manager | 10 ++++++++
> >  drivers/fpga/fpga-mgr.c                            | 28 ++++++++++++++++++++++
> >  include/linux/fpga/fpga-mgr.h                      | 17 +++++++++++++
> >  3 files changed, 55 insertions(+)
> >
> > diff --git a/Documentation/ABI/testing/sysfs-class-fpga-manager b/Documentation/ABI/testing/sysfs-class-fpga-manager
> > index 23056c5..01db14d 100644
> > --- a/Documentation/ABI/testing/sysfs-class-fpga-manager
> > +++ b/Documentation/ABI/testing/sysfs-class-fpga-manager
> > @@ -35,3 +35,13 @@ Description: Read fpga manager state as a string.
> >                 * write complete        = Doing post programming steps
> >                 * write complete error  = Error while doing post programming
> >                 * operating             = FPGA is programmed and operating
> > +
> > +What:          /sys/class/fpga_manager/<fpga>/status
> > +Date:          November 2017
> > +KernelVersion: 4.15
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read fpga manager status as a string.
> > +               If FPGA programming operation fails, it could be due to crc
> > +               error or incompatible bitstream image. The intent of this
> > +               interface is to provide more detailed information for FPGA
> > +               programming errors to userspace.
> 
> Hi Hao,
> 
> Please also add a list of status strings that can be given, such as
> "reconfig operation error"  I understand that this list may grow
> somewhat in the future.

Sure, will add it in the next version.

Thanks
Hao

> 
> Alan
> 
> > diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c
> > index 1fd5494..8b583ba 100644
> > --- a/drivers/fpga/fpga-mgr.c
> > +++ b/drivers/fpga/fpga-mgr.c
> > @@ -88,6 +88,7 @@ static int fpga_mgr_write_init_buf(struct fpga_manager *mgr,
> >         if (ret) {
> >                 dev_err(&mgr->dev, "Error preparing FPGA for writing\n");
> >                 mgr->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> > +               fpga_mgr_update_status(mgr);
> >                 return ret;
> >         }
> >
> > @@ -148,6 +149,7 @@ static int fpga_mgr_write_complete(struct fpga_manager *mgr,
> >         if (ret) {
> >                 dev_err(&mgr->dev, "Error after writing image data to FPGA\n");
> >                 mgr->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
> > +               fpga_mgr_update_status(mgr);
> >                 return ret;
> >         }
> >         mgr->state = FPGA_MGR_STATE_OPERATING;
> > @@ -225,6 +227,7 @@ static int fpga_mgr_buf_load_mapped(struct fpga_manager *mgr,
> >         if (ret) {
> >                 dev_err(&mgr->dev, "Error while writing image data to FPGA\n");
> >                 mgr->state = FPGA_MGR_STATE_WRITE_ERR;
> > +               fpga_mgr_update_status(mgr);
> >                 return ret;
> >         }
> >
> > @@ -397,12 +400,36 @@ static ssize_t state_show(struct device *dev,
> >         return sprintf(buf, "%s\n", state_str[mgr->state]);
> >  }
> >
> > +static ssize_t status_show(struct device *dev,
> > +                          struct device_attribute *attr, char *buf)
> > +{
> > +       struct fpga_manager *mgr = to_fpga_manager(dev);
> > +       int len = 0;
> > +
> > +       if (mgr->status & FPGA_MGR_STATUS_OPERATION_ERR)
> > +               len += sprintf(buf + len, "reconfig operation error\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_CRC_ERR)
> > +               len += sprintf(buf + len, "reconfig CRC error\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR)
> > +               len += sprintf(buf + len, "reconfig incompatible image\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_IP_PROTOCOL_ERR)
> > +               len += sprintf(buf + len, "reconfig IP protocol error\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR)
> > +               len += sprintf(buf + len, "reconfig fifo overflow error\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_SECURE_LOAD_ERR)
> > +               len += sprintf(buf + len, "reconfig secure load error\n");
> > +
> > +       return len;
> > +}

[PATCH v3 04/21] fpga: add device feature list support

[Wu Hao]

Device Feature List (DFL) defines a feature list structure that creates
a link list of feature headers within the MMIO space to provide an
extensible way of adding features. This patch introduces a kernel module
to provide basic infrastructure to support FPGA devices which implement
the Device Feature List.

Usually there will be different features and their sub features linked into
the DFL. This code provides common APIs for feature enumeration, it creates
a container device (FPGA base region), walks through the DFLs and creates
platform devices for feature devices (Currently it only supports two
different feature devices, FPGA Management Engine (FME) and Port which
the Accelerator Function Unit (AFU) connected to). In order to enumerate
the DFLs, the common APIs required low level driver to provide necessary
enumeration information (e.g address for each device feature list for
given device) and fill it to the fpga_enum_info data structure. Please
refer to below description for APIs added for enumeration.

Functions for enumeration information preparation:
 *fpga_enum_info_alloc
   allocate enumeration information data structure.

 *fpga_enum_info_add_dfl
   add a device feature list to fpga_enum_info data structure.

 *fpga_enum_info_free
   free fpga_enum_info data structure and related resources.

Functions for feature device enumeration:
 *fpga_enumerate_feature_devs
   enumerate feature devices and return container device.

 *fpga_remove_feature_devs
   remove feature devices under given container device.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: split from another patch.
    separate dfl enumeration code from original pcie driver.
    provide common data structures and APIs for enumeration.
    update device feature list parsing process according to latest hw.
    add dperf/iperf/hssi sub feature placeholder according to latest hw.
    remove build_info_add_sub_feature and other small functions.
    replace *_feature_num function with macro.
    remove writeq/readq.
---
 drivers/fpga/Kconfig    |  16 +
 drivers/fpga/Makefile   |   3 +
 drivers/fpga/fpga-dfl.c | 884 ++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/fpga-dfl.h | 365 ++++++++++++++++++++
 4 files changed, 1268 insertions(+)
 create mode 100644 drivers/fpga/fpga-dfl.c
 create mode 100644 drivers/fpga/fpga-dfl.h

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index f47ef84..01ad31f 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -124,4 +124,20 @@ config OF_FPGA_REGION
 	  Support for loading FPGA images by applying a Device Tree
 	  overlay.
 
+config FPGA_DFL
+	tristate "FPGA Device Feature List (DFL) support"
+	select FPGA_BRIDGE
+	select FPGA_REGION
+	help
+	  Device Feature List (DFL) defines a feature list structure that
+	  creates a link list of feature headers within the MMIO space
+	  to provide an extensible way of adding features for FPGA.
+	  Driver can walk through the feature headers to enumerate feature
+	  devices (e.g FPGA Management Engine, Port and Accelerator
+	  Function Unit) and their private features for target FPGA devices.
+
+	  Select this option to enable common support for Field-Programmable
+	  Gate Array (FPGA) solutions which implement Device Feature List.
+	  It provides enumeration APIs, and feature device infrastructure.
+
 endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 3cb276a..447ba2b 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -27,3 +27,6 @@ obj-$(CONFIG_XILINX_PR_DECOUPLER)	+= xilinx-pr-decoupler.o
 # High Level Interfaces
 obj-$(CONFIG_FPGA_REGION)		+= fpga-region.o
 obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
+
+# FPGA Device Feature List Support
+obj-$(CONFIG_FPGA_DFL)			+= fpga-dfl.o
diff --git a/drivers/fpga/fpga-dfl.c b/drivers/fpga/fpga-dfl.c
new file mode 100644
index 0000000..6609828
--- /dev/null
+++ b/drivers/fpga/fpga-dfl.c
@@ -0,0 +1,884 @@
+/*
+ * Driver for FPGA Device Feature List (DFL) Support
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+#include <linux/module.h>
+
+#include "fpga-dfl.h"
+
+static DEFINE_MUTEX(fpga_id_mutex);
+
+enum fpga_id_type {
+	FME_ID,		/* fme id allocation and mapping */
+	PORT_ID,	/* port id allocation and mapping */
+	FPGA_ID_MAX,
+};
+
+/* it is protected by fpga_id_mutex */
+static struct idr fpga_ids[FPGA_ID_MAX];
+
+static void fpga_ids_init(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+		idr_init(fpga_ids + i);
+}
+
+static void fpga_ids_destroy(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+		idr_destroy(fpga_ids + i);
+}
+
+static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
+{
+	int id;
+
+	WARN_ON(type >= FPGA_ID_MAX);
+	mutex_lock(&fpga_id_mutex);
+	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
+	mutex_unlock(&fpga_id_mutex);
+
+	return id;
+}
+
+static void free_fpga_id(enum fpga_id_type type, int id)
+{
+	WARN_ON(type >= FPGA_ID_MAX);
+	mutex_lock(&fpga_id_mutex);
+	idr_remove(fpga_ids + type, id);
+	mutex_unlock(&fpga_id_mutex);
+}
+
+static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
+{
+	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
+		return FME_ID;
+
+	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
+		return PORT_ID;
+
+	WARN_ON(1);
+
+	return FPGA_ID_MAX;
+}
+
+/**
+ * build_feature_devs_info - info collected during feature dev build.
+ *
+ * @dev: device to enumerate.
+ * @cdev: the container device for all feature devices.
+ * @feature_dev: current feature device.
+ */
+struct build_feature_devs_info {
+	struct device *dev;
+	struct fpga_cdev *cdev;
+	struct platform_device *feature_dev;
+};
+
+static void fpga_cdev_add_port_dev(struct fpga_cdev *cdev,
+				   struct platform_device *port_pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&port_pdev->dev);
+
+	mutex_lock(&cdev->lock);
+	list_add(&pdata->node, &cdev->port_dev_list);
+	get_device(&pdata->dev->dev);
+	mutex_unlock(&cdev->lock);
+}
+
+/*
+ * register current feature device, it is called when we need to switch to
+ * another feature parsing or we have parsed all features on given device
+ * feature list.
+ */
+static int build_info_commit_dev(struct build_feature_devs_info *binfo)
+{
+	int ret;
+
+	if (!binfo->feature_dev)
+		return 0;
+
+	ret = platform_device_add(binfo->feature_dev);
+	if (!ret) {
+		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
+			fpga_cdev_add_port_dev(binfo->cdev, binfo->feature_dev);
+		else
+			binfo->cdev->fme_dev =
+					get_device(&binfo->feature_dev->dev);
+		/*
+		 * reset it to avoid build_info_free() freeing their resource.
+		 *
+		 * The resource of successfully registered feature devices
+		 * will be freed by platform_device_unregister(). See the
+		 * comments in build_info_create_dev().
+		 */
+		binfo->feature_dev = NULL;
+	}
+
+	return ret;
+}
+
+static int
+build_info_create_dev(struct build_feature_devs_info *binfo,
+		      enum fpga_id_type type, int feature_nr, const char *name)
+{
+	struct platform_device *fdev;
+	struct resource *res;
+	struct feature_platform_data *pdata;
+	int ret;
+
+	/* we will create a new device, commit current device first */
+	ret = build_info_commit_dev(binfo);
+	if (ret)
+		return ret;
+
+	/*
+	 * we use -ENODEV as the initialization indicator which indicates
+	 * whether the id need to be reclaimed
+	 */
+	fdev = platform_device_alloc(name, -ENODEV);
+	if (!fdev)
+		return -ENOMEM;
+
+	binfo->feature_dev = fdev;
+
+	fdev->id = alloc_fpga_id(type, &fdev->dev);
+	if (fdev->id < 0)
+		return fdev->id;
+
+	fdev->dev.parent = &binfo->cdev->region.dev;
+
+	/*
+	 * we do not need to care for the memory which is associated with
+	 * the platform device. After calling platform_device_unregister(),
+	 * it will be automatically freed by device's release() callback,
+	 * platform_device_release().
+	 */
+	pdata = kzalloc(feature_platform_data_size(feature_nr), GFP_KERNEL);
+	if (pdata) {
+		pdata->dev = fdev;
+		pdata->num = feature_nr;
+		mutex_init(&pdata->lock);
+	} else {
+		return -ENOMEM;
+	}
+
+	/*
+	 * the count should be initialized to 0 to make sure
+	 *__fpga_port_enable() following __fpga_port_disable()
+	 * works properly for port device.
+	 * and it should always be 0 for fme device.
+	 */
+	WARN_ON(pdata->disable_count);
+
+	fdev->dev.platform_data = pdata;
+	fdev->num_resources = feature_nr;
+	fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
+	if (!fdev->resource)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static void build_info_free(struct build_feature_devs_info *binfo)
+{
+	/*
+	 * it is a valid id, free it. See comments in
+	 * build_info_create_dev()
+	 */
+	if (binfo->feature_dev && binfo->feature_dev->id >= 0)
+		free_fpga_id(feature_dev_id_type(binfo->feature_dev),
+			     binfo->feature_dev->id);
+
+	platform_device_put(binfo->feature_dev);
+
+	devm_kfree(binfo->dev, binfo);
+}
+
+/*
+ * UAFU GUID is dynamic as it can be changed after FME downloads different
+ * Green Bitstream to the port, so we treat the unknown GUIDs which are
+ * attached on port's feature list as UAFU.
+ */
+static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
+{
+	if (!binfo->feature_dev ||
+	    feature_dev_id_type(binfo->feature_dev) != PORT_ID)
+		return false;
+
+	return true;
+}
+
+struct feature_info {
+	const char *name;
+	resource_size_t resource_size;
+	int feature_index;
+};
+
+/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
+static struct feature_info fme_features[] = {
+	{
+		.name = FME_FEATURE_HEADER,
+		.resource_size = FME_HDR_SIZE,
+		.feature_index = FME_FEATURE_ID_HEADER,
+	},
+	{
+		.name = FME_FEATURE_THERMAL_MGMT,
+		.resource_size = FME_THERMAL_SIZE,
+		.feature_index = FME_FEATURE_ID_THERMAL_MGMT,
+	},
+	{
+		.name = FME_FEATURE_POWER_MGMT,
+		.resource_size = FME_POWER_SIZE,
+		.feature_index = FME_FEATURE_ID_POWER_MGMT,
+	},
+	{
+		.name = FME_FEATURE_GLOBAL_IPERF,
+		.resource_size = FME_IPERF_SIZE,
+		.feature_index = FME_FEATURE_ID_GLOBAL_IPERF,
+	},
+	{
+		.name = FME_FEATURE_GLOBAL_ERR,
+		.resource_size = FME_ERR_SIZE,
+		.feature_index = FME_FEATURE_ID_GLOBAL_ERR,
+	},
+	{
+		.name = FME_FEATURE_PR_MGMT,
+		.resource_size = FME_PR_SIZE,
+		.feature_index = FME_FEATURE_ID_PR_MGMT,
+	},
+	{
+		.name = FME_FEATURE_HSSI,
+		.resource_size = FME_HSSI_SIZE,
+		.feature_index = FME_FEATURE_ID_HSSI,
+	},
+	{
+		.name = FME_FEATURE_GLOBAL_DPERF,
+		.resource_size = FME_DPERF_SIZE,
+		.feature_index = FME_FEATURE_ID_GLOBAL_DPERF,
+	},
+};
+
+/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
+static struct feature_info port_features[] = {
+	{
+		.name = PORT_FEATURE_HEADER,
+		.resource_size = PORT_HDR_SIZE,
+		.feature_index = PORT_FEATURE_ID_HEADER,
+	},
+	{
+		.name = PORT_FEATURE_ERR,
+		.resource_size = PORT_ERR_SIZE,
+		.feature_index = PORT_FEATURE_ID_ERROR,
+	},
+	{
+		.name = PORT_FEATURE_UMSG,
+		.resource_size = PORT_UMSG_SIZE,
+		.feature_index = PORT_FEATURE_ID_UMSG,
+	},
+	{
+		/* This feature isn't available for now */
+		.name = PORT_FEATURE_PR,
+		.resource_size = DFH_SIZE,
+		.feature_index = PORT_FEATURE_ID_PR,
+	},
+	{
+		.name = PORT_FEATURE_STP,
+		.resource_size = PORT_STP_SIZE,
+		.feature_index = PORT_FEATURE_ID_STP,
+	},
+	{
+		/*
+		 * For User AFU feature, its region size is not fixed, but
+		 * reported by register PortCapability.mmio_size. Resource
+		 * size of UAFU will be set while parse port device.
+		 */
+		.name = PORT_FEATURE_UAFU,
+		.resource_size = 0,
+		.feature_index = PORT_FEATURE_ID_UAFU,
+	},
+};
+
+static int
+create_feature_instance(struct build_feature_devs_info *binfo,
+			struct fpga_enum_dfl *dfl, resource_size_t ofst,
+			struct feature_info *finfo)
+{
+	int index = finfo->feature_index;
+	struct platform_device *fdev = binfo->feature_dev;
+	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
+	struct resource *res = &fdev->resource[index];
+
+	if ((dfl->len - ofst < finfo->resource_size) || pdata->num < index)
+		return -EINVAL;
+
+	res->start = dfl->start + ofst;
+	res->end = res->start + finfo->resource_size - 1;
+	res->flags = IORESOURCE_MEM;
+	res->name = finfo->name;
+
+	pdata->features[index].name = finfo->name;
+	pdata->features[index].resource_index = index;
+	pdata->features[index].ioaddr = dfl->ioaddr + ofst;
+
+	return 0;
+}
+
+static int parse_feature_fme(struct build_feature_devs_info *binfo,
+			     struct fpga_enum_dfl *dfl,
+			     resource_size_t ofst)
+{
+	int ret;
+
+	ret = build_info_create_dev(binfo, FME_ID, FME_FEATURE_NUM,
+				    FPGA_FEATURE_DEV_FME);
+	if (ret)
+		return ret;
+
+	return create_feature_instance(binfo, dfl, ofst,
+				       &fme_features[FME_FEATURE_ID_HEADER]);
+}
+
+static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
+				     struct fpga_enum_dfl *dfl,
+				     resource_size_t ofst)
+{
+	u64 v;
+	int id;
+
+	v = readq(dfl->ioaddr + ofst + DFH);
+	id = FIELD_GET(DFH_ID, v);
+
+	if (id >= ARRAY_SIZE(fme_features)) {
+		dev_info(binfo->dev, "FME feature id %x is not supported yet.\n",
+			 id);
+		return 0;
+	}
+
+	return create_feature_instance(binfo, dfl, ofst, &fme_features[id]);
+}
+
+static int parse_feature_port(struct build_feature_devs_info *binfo,
+			      struct fpga_enum_dfl *dfl,
+			      resource_size_t ofst)
+{
+	int ret;
+
+	ret = build_info_create_dev(binfo, PORT_ID, PORT_FEATURE_NUM,
+				    FPGA_FEATURE_DEV_PORT);
+	if (ret)
+		return ret;
+
+	return create_feature_instance(binfo, dfl, ofst,
+				       &port_features[PORT_FEATURE_ID_HEADER]);
+}
+
+static void enable_port_uafu(struct build_feature_devs_info *binfo)
+{
+	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
+	void __iomem *base;
+	u64 v;
+
+	base = get_feature_ioaddr_by_index(&binfo->feature_dev->dev,
+					   PORT_FEATURE_ID_HEADER);
+
+	v = readq(base + PORT_HDR_CAP);
+	port_features[id].resource_size =
+			FIELD_GET(PORT_CAP_MMIO_SIZE, v) << 10;
+
+	/*
+	 * To enable User AFU, driver needs to clear reset bit on related port,
+	 * otherwise the mmio space of this user AFU will be invalid.
+	 */
+	if (port_features[id].resource_size)
+		fpga_port_reset(binfo->feature_dev);
+}
+
+static int parse_feature_port_private(struct build_feature_devs_info *binfo,
+				      struct fpga_enum_dfl *dfl,
+				      resource_size_t ofst)
+{
+	enum port_feature_id id;
+	u32 dfh_id;
+	u64 v;
+
+	v = readq(dfl->ioaddr + ofst + DFH);
+	dfh_id = FIELD_GET(DFH_ID, v);
+
+	/*
+	 * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
+	 * which is dedicated for port-hdr.
+	 */
+	id = (dfh_id & 0x000f) + 1;
+
+	if (id >= ARRAY_SIZE(port_features)) {
+		dev_info(binfo->dev, "Port feature id %x is not supported yet.\n",
+			 dfh_id);
+		return 0;
+	}
+
+	return create_feature_instance(binfo, dfl, ofst, &port_features[id]);
+}
+
+static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
+				   struct fpga_enum_dfl *dfl,
+				   resource_size_t ofst)
+{
+	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
+	int ret;
+
+	if (port_features[id].resource_size) {
+		ret = create_feature_instance(binfo, dfl, ofst,
+					      &port_features[id]);
+		port_features[id].resource_size = 0;
+	} else {
+		dev_err(binfo->dev, "the uafu feature header is mis-configured.\n");
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
+static int parse_feature_afus(struct build_feature_devs_info *binfo,
+			      struct fpga_enum_dfl *dfl,
+			      resource_size_t ofst)
+{
+	void __iomem *start = dfl->ioaddr + ofst;
+	void __iomem *end = dfl->ioaddr + dfl->len;
+	u32 offset;
+	u64 v;
+	int ret;
+
+	for (; start < end; start += offset) {
+		if (end - start < AFU_DFH_SIZE)
+			return -EINVAL;
+
+		if (feature_is_UAFU(binfo))
+			ret = parse_feature_port_uafu(binfo, dfl,
+						      start - dfl->ioaddr);
+			if (ret)
+				return ret;
+
+		v = readq(start + NEXT_AFU);
+
+		offset = FIELD_GET(NEXT_AFU_NEXT_DFH_OFST, v);
+		if (!offset)
+			break;
+	}
+
+	return 0;
+}
+
+static int parse_feature_fiu(struct build_feature_devs_info *binfo,
+			     struct fpga_enum_dfl *dfl,
+			     resource_size_t ofst)
+{
+	u32 id, offset;
+	u64 v;
+	int ret = 0;
+
+	v = readq(dfl->ioaddr + ofst + DFH);
+	id = FIELD_GET(DFH_ID, v);
+
+	switch (id) {
+	case DFH_ID_FIU_FME:
+		return parse_feature_fme(binfo, dfl, ofst);
+	case DFH_ID_FIU_PORT:
+		ret = parse_feature_port(binfo, dfl, ofst);
+		enable_port_uafu(binfo);
+		if (ret)
+			return ret;
+
+		/* Check Port FIU's next_afu pointer to User AFU DFH */
+		v = readq(dfl->ioaddr + ofst + NEXT_AFU);
+
+		offset = FIELD_GET(NEXT_AFU_NEXT_DFH_OFST, v);
+		if (offset)
+			return parse_feature_afus(binfo, dfl, ofst + offset);
+
+		dev_dbg(binfo->dev, "No AFUs detected on Port\n");
+		break;
+	default:
+		dev_info(binfo->dev, "FIU TYPE %d is not supported yet.\n",
+			 id);
+	}
+
+	return ret;
+}
+
+static int parse_feature_private(struct build_feature_devs_info *binfo,
+				 struct fpga_enum_dfl *dfl,
+				 resource_size_t ofst)
+{
+	u64 v;
+	u32 id;
+
+	v = readq(dfl->ioaddr + ofst + DFH);
+	id = FIELD_GET(DFH_ID, v);
+
+	if (!binfo->feature_dev) {
+		dev_err(binfo->dev, "the private feature %x does not belong to any AFU.\n",
+			id);
+		return -EINVAL;
+	}
+
+	switch (feature_dev_id_type(binfo->feature_dev)) {
+	case FME_ID:
+		return parse_feature_fme_private(binfo, dfl, ofst);
+	case PORT_ID:
+		return parse_feature_port_private(binfo, dfl, ofst);
+	default:
+		dev_info(binfo->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
+			 id, binfo->feature_dev->name);
+	}
+	return 0;
+}
+
+/**
+ * parse_feature - parse a feature on given device feature list
+ *
+ * @binfo: build feature devices information.
+ * @dfl: device feature list to parse
+ * @ofst: offset to feature header on this device feature list
+ */
+static int parse_feature(struct build_feature_devs_info *binfo,
+			 struct fpga_enum_dfl *dfl, resource_size_t ofst)
+{
+	u64 v;
+	u32 type;
+
+	v = readq(dfl->ioaddr + ofst + DFH);
+	type = FIELD_GET(DFH_TYPE, v);
+
+	switch (type) {
+	case DFH_TYPE_AFU:
+		return parse_feature_afus(binfo, dfl, ofst);
+	case DFH_TYPE_PRIVATE:
+		return parse_feature_private(binfo, dfl, ofst);
+	case DFH_TYPE_FIU:
+		return parse_feature_fiu(binfo, dfl, ofst);
+	default:
+		dev_info(binfo->dev,
+			 "Feature Type %x is not supported.\n", type);
+	}
+
+	return 0;
+}
+
+static int parse_feature_list(struct build_feature_devs_info *binfo,
+			      struct fpga_enum_dfl *dfl)
+{
+	void __iomem *start = dfl->ioaddr;
+	void __iomem *end = dfl->ioaddr + dfl->len;
+	int ret = 0;
+	u32 ofst = 0;
+	u64 v;
+
+	/* walk through the device feature list via DFH's next DFH pointer. */
+	for (; start < end; start += ofst) {
+		if (end - start < DFH_SIZE) {
+			dev_err(binfo->dev, "The region is too small to contain a feature.\n");
+			return -EINVAL;
+		}
+
+		ret = parse_feature(binfo, dfl, start - dfl->ioaddr);
+		if (ret)
+			return ret;
+
+		v = readq(start + DFH);
+		ofst = FIELD_GET(DFH_NEXT_HDR_OFST, v);
+
+		/* stop parsing if EOL(End of List) is set or offset is 0 */
+		if ((v & DFH_EOL) || !ofst)
+			break;
+	}
+
+	/* commit current feature device when reach the end of list */
+	return build_info_commit_dev(binfo);
+}
+
+struct fpga_enum_info *fpga_enum_info_alloc(struct device *dev)
+{
+	struct fpga_enum_info *info;
+
+	get_device(dev);
+
+	info = devm_kzalloc(dev, sizeof(*info), GFP_KERNEL);
+	if (!info) {
+		put_device(dev);
+		return NULL;
+	}
+
+	info->dev = dev;
+	INIT_LIST_HEAD(&info->dfls);
+
+	return info;
+}
+EXPORT_SYMBOL_GPL(fpga_enum_info_alloc);
+
+int fpga_enum_info_add_dfl(struct fpga_enum_info *info, resource_size_t start,
+			   resource_size_t len, void __iomem *ioaddr)
+{
+	struct fpga_enum_dfl *dfl;
+
+	dfl = devm_kzalloc(info->dev, sizeof(*dfl), GFP_KERNEL);
+	if (!dfl)
+		return -ENOMEM;
+
+	dfl->start = start;
+	dfl->len = len;
+	dfl->ioaddr = ioaddr;
+
+	list_add_tail(&dfl->node, &info->dfls);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(fpga_enum_info_add_dfl);
+
+void fpga_enum_info_free(struct fpga_enum_info *info)
+{
+	struct fpga_enum_dfl *tmp, *dfl;
+	struct device *dev;
+
+	if (!info)
+		return;
+
+	dev = info->dev;
+
+	/* remove all device feature lists in the list. */
+	list_for_each_entry_safe(dfl, tmp, &info->dfls, node) {
+		list_del(&dfl->node);
+		devm_kfree(dev, dfl);
+	}
+
+	devm_kfree(dev, info);
+	put_device(dev);
+}
+EXPORT_SYMBOL_GPL(fpga_enum_info_free);
+
+static int remove_feature_dev(struct device *dev, void *data)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+	enum fpga_id_type type = feature_dev_id_type(pdev);
+	int id = pdev->id;
+
+	platform_device_unregister(pdev);
+
+	free_fpga_id(type, id);
+
+	return 0;
+}
+
+static void remove_feature_devs(struct fpga_cdev *cdev)
+{
+	device_for_each_child(&cdev->region.dev, NULL, remove_feature_dev);
+}
+
+/**
+ * fpga_enumerate_feature_devs - enumerate feature devices
+ * @info: information for enumeration.
+ *
+ * This function creates a container device (base FPGA region), enumerates
+ * feature devices based on the enumeration info and creates platform devices
+ * under the container device.
+ *
+ * Return: fpga_cdev struct on success, -errno on failure
+ */
+struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info)
+{
+	struct build_feature_devs_info *binfo;
+	struct fpga_cdev *cdev;
+	struct fpga_enum_dfl *dfl;
+	int ret = 0;
+
+	if (!info->dev)
+		return ERR_PTR(-ENODEV);
+
+	cdev = devm_kzalloc(info->dev, sizeof(*cdev), GFP_KERNEL);
+	if (!cdev)
+		return ERR_PTR(-ENOMEM);
+
+	cdev->parent = info->dev;
+	mutex_init(&cdev->lock);
+	INIT_LIST_HEAD(&cdev->port_dev_list);
+	cdev->region.parent = info->dev;
+
+	ret = fpga_region_register(&cdev->region);
+	if (ret)
+		goto free_cdev_exit;
+
+	/* create and init build info for enumeration */
+	binfo = devm_kzalloc(info->dev, sizeof(*binfo), GFP_KERNEL);
+	if (!binfo) {
+		ret = -ENOMEM;
+		goto unregister_region_exit;
+	}
+
+	binfo->dev = info->dev;
+	binfo->cdev = cdev;
+
+	/*
+	 * start enumeration for all feature devices based on Device Feature
+	 * Lists.
+	 */
+	list_for_each_entry(dfl, &info->dfls, node) {
+		ret = parse_feature_list(binfo, dfl);
+		if (ret) {
+			remove_feature_devs(cdev);
+			build_info_free(binfo);
+			goto unregister_region_exit;
+		}
+	}
+
+	build_info_free(binfo);
+
+	return cdev;
+
+unregister_region_exit:
+	fpga_region_unregister(&cdev->region);
+free_cdev_exit:
+	devm_kfree(cdev->parent, cdev);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(fpga_enumerate_feature_devs);
+
+/**
+ * fpga_remove_feature_devs - remove all feature devices
+ * @cdev: fpga container device.
+ *
+ * Remove the container device and all feature devices under given container
+ * devices.
+ */
+void fpga_remove_feature_devs(struct fpga_cdev *cdev)
+{
+	struct feature_platform_data *pdata, *ptmp;
+
+	remove_feature_devs(cdev);
+
+	mutex_lock(&cdev->lock);
+	if (cdev->fme_dev) {
+		/* the fme should be unregistered. */
+		WARN_ON(device_is_registered(cdev->fme_dev));
+		put_device(cdev->fme_dev);
+	}
+
+	list_for_each_entry_safe(pdata, ptmp, &cdev->port_dev_list, node) {
+		struct platform_device *port_dev = pdata->dev;
+
+		/* the port should be unregistered. */
+		WARN_ON(device_is_registered(&port_dev->dev));
+		list_del(&pdata->node);
+		put_device(&port_dev->dev);
+	}
+	mutex_unlock(&cdev->lock);
+
+	fpga_region_unregister(&cdev->region);
+	devm_kfree(cdev->parent, cdev);
+}
+EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);
+
+int fpga_port_id(struct platform_device *pdev)
+{
+	void __iomem *base;
+
+	base = get_feature_ioaddr_by_index(&pdev->dev, PORT_FEATURE_ID_HEADER);
+	WARN_ON(!base);
+
+	return FIELD_GET(PORT_CAP_PORT_NUM, readq(base + FME_HDR_CAP));
+}
+EXPORT_SYMBOL_GPL(fpga_port_id);
+
+/*
+ * Enable Port by clear the port soft reset bit, which is set by default.
+ * The User AFU is unable to respond to any MMIO access while in reset.
+ * __fpga_port_enable function should only be used after __fpga_port_disable
+ * function.
+ */
+void __fpga_port_enable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	void __iomem *base;
+	u64 v;
+
+	WARN_ON(!pdata->disable_count);
+
+	if (--pdata->disable_count != 0)
+		return;
+
+	base = get_feature_ioaddr_by_index(&pdev->dev, PORT_FEATURE_ID_HEADER);
+	WARN_ON(!base);
+
+	/* Clear port soft reset */
+	v = readq(base + PORT_HDR_CTRL);
+	v &= ~PORT_CTRL_SFTRST;
+	writeq(v, base + PORT_HDR_CTRL);
+}
+EXPORT_SYMBOL_GPL(__fpga_port_enable);
+
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
+int __fpga_port_disable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	void __iomem *base;
+	u64 v;
+
+	if (pdata->disable_count++ != 0)
+		return 0;
+
+	base = get_feature_ioaddr_by_index(&pdev->dev, PORT_FEATURE_ID_HEADER);
+	WARN_ON(!base);
+
+	/* Set port soft reset */
+	v = readq(base + PORT_HDR_CTRL);
+	v |= PORT_CTRL_SFTRST;
+	writeq(v, base + PORT_HDR_CTRL);
+
+	/*
+	 * HW sets ack bit to 1 when all outstanding requests have been drained
+	 * on this port and minimum soft reset pulse width has elapsed.
+	 * Driver polls port_soft_reset_ack to determine if reset done by HW.
+	 */
+	if (readq_poll_timeout(base + PORT_HDR_CTRL, v, v & PORT_CTRL_SFTRST,
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, fail to reset device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(__fpga_port_disable);
+
+static int __init dfl_fpga_init(void)
+{
+	fpga_ids_init();
+
+	return 0;
+}
+
+static void __exit dfl_fpga_exit(void)
+{
+	fpga_ids_destroy();
+}
+
+module_init(dfl_fpga_init);
+module_exit(dfl_fpga_exit);
+
+MODULE_DESCRIPTION("FPGA Device Feature List (DFL) Support");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
new file mode 100644
index 0000000..abcbe57
--- /dev/null
+++ b/drivers/fpga/fpga-dfl.h
@@ -0,0 +1,365 @@
+/*
+ * Driver Header File for FPGA Device Feature List (DFL) Support
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#ifndef __DFL_FPGA_H
+#define __DFL_FPGA_H
+
+#include <linux/bitfield.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/iopoll.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/uuid.h>
+#include <linux/fpga/fpga-region.h>
+
+/* maximum supported number of ports */
+#define MAX_FPGA_PORT_NUM 4
+/* plus one for fme device */
+#define MAX_FEATURE_DEV_NUM	(MAX_FPGA_PORT_NUM + 1)
+
+#define FME_FEATURE_HEADER          "fme_hdr"
+#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
+#define FME_FEATURE_POWER_MGMT      "fme_power"
+#define FME_FEATURE_GLOBAL_IPERF    "fme_iperf"
+#define FME_FEATURE_GLOBAL_ERR      "fme_error"
+#define FME_FEATURE_PR_MGMT         "fme_pr"
+#define FME_FEATURE_HSSI            "fme_hssi"
+#define FME_FEATURE_GLOBAL_DPERF    "fme_dperf"
+
+#define PORT_FEATURE_HEADER         "port_hdr"
+#define PORT_FEATURE_UAFU           "port_uafu"
+#define PORT_FEATURE_ERR            "port_err"
+#define PORT_FEATURE_UMSG           "port_umsg"
+#define PORT_FEATURE_PR             "port_pr"
+#define PORT_FEATURE_STP            "port_stp"
+
+/* Device Feature Header Register Set */
+#define DFH			0x0
+#define GUID_L			0x8
+#define GUID_H			0x10
+#define NEXT_AFU		0x18
+
+/* Device Feature Header Register Bitfield */
+#define DFH_ID			GENMASK_ULL(11, 0)	/* Feature ID */
+#define DFH_ID_FIU_FME		0
+#define DFH_ID_FIU_PORT		1
+#define DFH_REVISION		GENMASK_ULL(15, 12)	/* Feature revision */
+#define DFH_NEXT_HDR_OFST	GENMASK_ULL(39, 16)	/* Offset to next DFH */
+#define DFH_EOL			BIT(40)			/* End of list */
+#define DFH_TYPE		GENMASK_ULL(63, 60)	/* Feature type */
+#define DFH_TYPE_AFU		1
+#define DFH_TYPE_PRIVATE	3
+#define DFH_TYPE_FIU		4
+
+/* Next AFU Register Bitfield */
+#define NEXT_AFU_NEXT_DFH_OFST	GENMASK_ULL(23, 0)	/* Offset to next AFU */
+
+/*
+ * It only has DFH register as header for private feature, but for FIU/AFU
+ * For FIU/AFU features, they all have DFH + GUID + NEXT_AFU as common header
+ * registers.
+ */
+#define DFH_SIZE		0x8
+#define AFU_DFH_SIZE		0x20
+#define FIU_DFH_SIZE		0x20
+
+/* FME Header Register Set */
+#define FME_HDR_DFH		DFH
+#define FME_HDR_AFU_GUID_L	GUID_L
+#define FME_HDR_AFU_GUID_H	GUID_H
+#define FME_HDR_NEXT_AFU	NEXT_AFU
+#define FME_HDR_CAP		0x30
+#define FME_HDR_PORT_OFST(n)	(0x38 + ((n) * 0x8))
+#define FME_HDR_BITSTREAM_ID	0x60
+#define FME_HDR_BITSTREAM_MD	0x68
+#define FME_HDR_SIZE		0x70
+
+/* FME Fab Capability Register Bitfield */
+#define FME_CAP_FABRIC_VERID	GENMASK_ULL(7, 0)	/* Fabric version ID */
+#define FME_CAP_SOCKET_ID	BIT(8)			/* Socket ID */
+#define FME_CAP_PCIE0_LINK_AVL	BIT(12)			/* PCIE0 Link */
+#define FME_CAP_PCIE1_LINK_AVL	BIT(13)			/* PCIE1 Link */
+#define FME_CAP_COHR_LINK_AVL	BIT(14)			/* Coherent Link */
+#define FME_CAP_IOMMU_AVL	BIT(16)			/* IOMMU available */
+#define FME_CAP_NUM_PORTS	GENMASK_ULL(19, 17)	/* Number of ports */
+#define FME_CAP_ADDR_WIDTH	GENMASK_ULL(29, 24)	/* Address bus width */
+#define FME_CAP_CACHE_SIZE	GENMASK_ULL(43, 32)	/* cache size in KB */
+#define FME_CAP_CACHE_ASSOC	GENMASK_ULL(47, 44)	/* Associativity */
+
+/* FME Port Offset Register Bitfield */
+/* Offset to port device feature header */
+#define FME_PORT_OFST_DFH_OFST	GENMASK_ULL(23, 0)
+/* PCI Bar ID for this port */
+#define FME_PORT_OFST_BAR_ID	GENMASK_ULL(34, 32)
+/* AFU MMIO access permission. 1 - VF, 0 - PF. */
+#define FME_PORT_OFST_ACC_CTRL	BIT(55)
+#define FME_PORT_OFST_ACC_PF	0
+#define FME_PORT_OFST_ACC_VF	1
+#define FME_PORT_OFST_IMP	BIT(60)
+
+/* FME Thermal Sub Feature Register Set */
+#define FME_THERMAL_DFH		DFH
+#define FME_THERMAL_SIZE	DFH_SIZE
+
+/* FME Power Sub Feature Register Set */
+#define FME_POWER_DFH		DFH
+#define FME_POWER_SIZE		DFH_SIZE
+
+/* FME Global Performance Sub Feature Register Set */
+#define FME_IPERF_DFH		DFH
+#define FME_IPERF_SIZE		DFH_SIZE
+
+/* FME Global Error Sub Feature Register Set */
+#define FME_ERR_DFH		DFH
+#define FME_ERR_SIZE		DFH_SIZE
+
+/* FME Partial Reconfiguration Sub Feature Register Set */
+#define FME_PR_DFH		DFH
+#define FME_PR_SIZE		DFH_SIZE
+
+/* FME HSSI Sub Feature Register Set */
+#define FME_HSSI_DFH		DFH
+#define FME_HSSI_SIZE		DFH_SIZE
+
+/* FME Global Performance Sub Feature Register Set */
+#define FME_DPERF_DFH		DFH
+#define FME_DPERF_SIZE		DFH_SIZE
+
+/* PORT Header Register Set */
+#define PORT_HDR_DFG		DFH
+#define PORT_HDR_AFU_GUID_L	GUID_L
+#define PORT_HDR_AFU_GUID_H	GUID_H
+#define PORT_HDR_NEXT_AFU	NEXT_AFU
+#define PORT_HDR_CAP		0x30
+#define PORT_HDR_CTRL		0x38
+#define PORT_HDR_SIZE		0x40
+
+/* Port Capability Register Bitfield */
+#define PORT_CAP_PORT_NUM	GENMASK(1, 0)		/* ID of this port */
+#define PORT_CAP_MMIO_SIZE	GENMASK(23, 8)		/* MMIO size in KB */
+#define PORT_CAP_SUPP_INT_NUM	GENMASK(35, 32)		/* Interrupts num */
+
+/* Port Control Register Bitfield */
+#define PORT_CTRL_SFTRST	BIT(0)			/* Port soft reset */
+/* Latency tolerance reporting. '1' >= 40us, '0' < 40us.*/
+#define PORT_CTRL_LATENCY	BIT(2)
+#define PORT_CTRL_SFTRST_ACK	BIT(4)			/* HW ack for reset */
+
+/* PORT Error Sub Feature Register Set */
+#define PORT_ERR_DFH		DFH
+#define PORT_ERR_SIZE		DFH_SIZE
+
+/* PORT Unordered Message Sub Feature Register Set */
+#define PORT_UMSG_DFH		DFH
+#define PORT_UMSG_SIZE		DFH_SIZE
+
+/* PORT SignalTap Sub Feature Register Set */
+#define PORT_STP_DFH		DFH
+#define PORT_STP_SIZE		DFH_SIZE
+
+/* PORT User AFU Sub Feature Register Set */
+#define PORT_UAFU_DFH		DFH
+#define PORT_UAFU_SIZE		DFH_SIZE
+
+struct feature {
+	const char *name;
+	int resource_index;
+	void __iomem *ioaddr;
+};
+
+struct feature_platform_data {
+	/* list the feature dev to cci_drvdata->port_dev_list. */
+	struct list_head node;
+	struct mutex lock;		/* protect platform data */
+	struct platform_device *dev;
+	unsigned int disable_count;	/* count for port disable */
+
+	int num;			/* number of features */
+	struct feature features[0];
+};
+
+enum fme_feature_id {
+	FME_FEATURE_ID_HEADER = 0x0,
+	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
+	FME_FEATURE_ID_POWER_MGMT = 0x2,
+	FME_FEATURE_ID_GLOBAL_IPERF = 0x3,
+	FME_FEATURE_ID_GLOBAL_ERR = 0x4,
+	FME_FEATURE_ID_PR_MGMT = 0x5,
+	FME_FEATURE_ID_HSSI = 0x6,
+	FME_FEATURE_ID_GLOBAL_DPERF = 0x7,
+	FME_FEATURE_ID_MAX = 0x8,
+};
+
+enum port_feature_id {
+	PORT_FEATURE_ID_HEADER = 0x0,
+	PORT_FEATURE_ID_ERROR = 0x1,
+	PORT_FEATURE_ID_UMSG = 0x2,
+	PORT_FEATURE_ID_PR = 0x3,
+	PORT_FEATURE_ID_STP = 0x4,
+	PORT_FEATURE_ID_UAFU = 0x5,
+	PORT_FEATURE_ID_MAX = 0x6,
+};
+
+#define FME_FEATURE_NUM			FME_FEATURE_ID_MAX
+#define PORT_FEATURE_NUM		PORT_FEATURE_ID_MAX
+
+#define FPGA_FEATURE_DEV_FME		"fpga-dfl-fme"
+#define FPGA_FEATURE_DEV_PORT		"fpga-dfl-port"
+
+static inline int feature_platform_data_size(const int num)
+{
+	return sizeof(struct feature_platform_data) +
+		num * sizeof(struct feature);
+}
+
+int fpga_port_id(struct platform_device *pdev);
+
+static inline int fpga_port_check_id(struct platform_device *pdev,
+				     void *pport_id)
+{
+	return fpga_port_id(pdev) == *(int *)pport_id;
+}
+
+void __fpga_port_enable(struct platform_device *pdev);
+int __fpga_port_disable(struct platform_device *pdev);
+
+static inline void fpga_port_enable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	mutex_lock(&pdata->lock);
+	__fpga_port_enable(pdev);
+	mutex_unlock(&pdata->lock);
+}
+
+static inline int fpga_port_disable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	ret = __fpga_port_disable(pdev);
+	mutex_unlock(&pdata->lock);
+
+	return ret;
+}
+
+static inline int __fpga_port_reset(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = __fpga_port_disable(pdev);
+	if (ret)
+		return ret;
+
+	__fpga_port_enable(pdev);
+
+	return 0;
+}
+
+static inline int fpga_port_reset(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	ret = __fpga_port_reset(pdev);
+	mutex_unlock(&pdata->lock);
+
+	return ret;
+}
+
+static inline void __iomem *
+get_feature_ioaddr_by_index(struct device *dev, int index)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(dev);
+
+	return pdata->features[index].ioaddr;
+}
+
+static inline bool feature_is_fme(void __iomem *base)
+{
+	u64 v = readq(base + DFH);
+
+	return (FIELD_GET(DFH_TYPE, v) == DFH_TYPE_FIU) &&
+		(FIELD_GET(DFH_ID, v) == DFH_ID_FIU_FME);
+}
+
+static inline bool feature_is_port(void __iomem *base)
+{
+	u64 v = readq(base + DFH);
+
+	return (FIELD_GET(DFH_TYPE, v) == DFH_TYPE_FIU) &&
+		(FIELD_GET(DFH_ID, v) == DFH_ID_FIU_PORT);
+}
+
+/**
+ * fpga_enum_info - FPGA enumeration information
+ *
+ * @dev: parent device.
+ * @dfls: list of device feature lists.
+ */
+struct fpga_enum_info {
+	struct device *dev;
+	struct list_head dfls;
+};
+
+/**
+ * fpga_enum_dfl - FPGA enumeration device feature list information
+ *
+ * @start: base address of this device feature list.
+ * @len: size of this device feature list.
+ * @ioaddr: mapped base address of this device feature list.
+ * @node: node in list of device feature lists.
+ */
+struct fpga_enum_dfl {
+	resource_size_t start;
+	resource_size_t len;
+
+	void __iomem *ioaddr;
+
+	struct list_head node;
+};
+
+struct fpga_enum_info *fpga_enum_info_alloc(struct device *dev);
+int fpga_enum_info_add_dfl(struct fpga_enum_info *info, resource_size_t start,
+			   resource_size_t len, void __iomem *ioaddr);
+void fpga_enum_info_free(struct fpga_enum_info *info);
+
+/**
+ * fpga_cdev - fpga container device
+ * @parent: parent device of this container device.
+ * @region: base fpga region.
+ * @fme_dev: FME feature device under this container device.
+ * @lock: mutex lock to protect the port device list.
+ * @port_dev_list: list of all port feature devices under this container device.
+ */
+struct fpga_cdev {
+	struct device *parent;
+
+	struct fpga_region region;
+
+	struct device *fme_dev;
+
+	struct mutex lock; /* to protect the port device list */
+	struct list_head port_dev_list;
+};
+
+struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info);
+void fpga_remove_feature_devs(struct fpga_cdev *cdev);
+
+#endif /* __DFL_FPGA_H */
-- 
1.8.3.1

Re: [PATCH v3 04/21] fpga: add device feature list support

[Moritz Fischer]

Hi Hao,

first pass, I didn't get all the way through, yet.

On Mon, Nov 27, 2017 at 02:42:11PM +0800, Wu Hao wrote:
> Device Feature List (DFL) defines a feature list structure that creates
> a link list of feature headers within the MMIO space to provide an
> extensible way of adding features. This patch introduces a kernel module
> to provide basic infrastructure to support FPGA devices which implement
> the Device Feature List.
> 
> Usually there will be different features and their sub features linked into
> the DFL. This code provides common APIs for feature enumeration, it creates
> a container device (FPGA base region), walks through the DFLs and creates
> platform devices for feature devices (Currently it only supports two
> different feature devices, FPGA Management Engine (FME) and Port which
> the Accelerator Function Unit (AFU) connected to). In order to enumerate
> the DFLs, the common APIs required low level driver to provide necessary
> enumeration information (e.g address for each device feature list for
> given device) and fill it to the fpga_enum_info data structure. Please
> refer to below description for APIs added for enumeration.
> 
> Functions for enumeration information preparation:
>  *fpga_enum_info_alloc
>    allocate enumeration information data structure.
> 
>  *fpga_enum_info_add_dfl
>    add a device feature list to fpga_enum_info data structure.
> 
>  *fpga_enum_info_free
>    free fpga_enum_info data structure and related resources.
> 
> Functions for feature device enumeration:
>  *fpga_enumerate_feature_devs
>    enumerate feature devices and return container device.
> 
>  *fpga_remove_feature_devs
>    remove feature devices under given container device.
> 
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v3: split from another patch.
>     separate dfl enumeration code from original pcie driver.
>     provide common data structures and APIs for enumeration.
>     update device feature list parsing process according to latest hw.
>     add dperf/iperf/hssi sub feature placeholder according to latest hw.
>     remove build_info_add_sub_feature and other small functions.
>     replace *_feature_num function with macro.
>     remove writeq/readq.
> ---
>  drivers/fpga/Kconfig    |  16 +
>  drivers/fpga/Makefile   |   3 +
>  drivers/fpga/fpga-dfl.c | 884 ++++++++++++++++++++++++++++++++++++++++++++++++
>  drivers/fpga/fpga-dfl.h | 365 ++++++++++++++++++++
>  4 files changed, 1268 insertions(+)
>  create mode 100644 drivers/fpga/fpga-dfl.c
>  create mode 100644 drivers/fpga/fpga-dfl.h
> 
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index f47ef84..01ad31f 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -124,4 +124,20 @@ config OF_FPGA_REGION
>  	  Support for loading FPGA images by applying a Device Tree
>  	  overlay.
>  
> +config FPGA_DFL
> +	tristate "FPGA Device Feature List (DFL) support"
> +	select FPGA_BRIDGE
> +	select FPGA_REGION
> +	help
> +	  Device Feature List (DFL) defines a feature list structure that
> +	  creates a link list of feature headers within the MMIO space
> +	  to provide an extensible way of adding features for FPGA.
> +	  Driver can walk through the feature headers to enumerate feature
> +	  devices (e.g FPGA Management Engine, Port and Accelerator
> +	  Function Unit) and their private features for target FPGA devices.
> +
> +	  Select this option to enable common support for Field-Programmable
> +	  Gate Array (FPGA) solutions which implement Device Feature List.
> +	  It provides enumeration APIs, and feature device infrastructure.
> +
>  endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 3cb276a..447ba2b 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -27,3 +27,6 @@ obj-$(CONFIG_XILINX_PR_DECOUPLER)	+= xilinx-pr-decoupler.o
>  # High Level Interfaces
>  obj-$(CONFIG_FPGA_REGION)		+= fpga-region.o
>  obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
> +
> +# FPGA Device Feature List Support
> +obj-$(CONFIG_FPGA_DFL)			+= fpga-dfl.o
> diff --git a/drivers/fpga/fpga-dfl.c b/drivers/fpga/fpga-dfl.c
> new file mode 100644
> index 0000000..6609828
> --- /dev/null
> +++ b/drivers/fpga/fpga-dfl.c
> @@ -0,0 +1,884 @@
> +/*
> + * Driver for FPGA Device Feature List (DFL) Support
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2.

This is redundant.
> + * SPDX-License-Identifier: GPL-2.0

Also I think the current consensus is that this should go in the first
line
> + */
> +#include <linux/module.h>
> +
> +#include "fpga-dfl.h"
> +
> +static DEFINE_MUTEX(fpga_id_mutex);
> +
> +enum fpga_id_type {
> +	FME_ID,		/* fme id allocation and mapping */
> +	PORT_ID,	/* port id allocation and mapping */
> +	FPGA_ID_MAX,
> +};
> +
> +/* it is protected by fpga_id_mutex */
> +static struct idr fpga_ids[FPGA_ID_MAX];
> +
> +static void fpga_ids_init(void)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +		idr_init(fpga_ids + i);
> +}
> +
> +static void fpga_ids_destroy(void)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +		idr_destroy(fpga_ids + i);
> +}
> +
> +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> +{
> +	int id;
> +
> +	WARN_ON(type >= FPGA_ID_MAX);
> +	mutex_lock(&fpga_id_mutex);
> +	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> +	mutex_unlock(&fpga_id_mutex);
> +
> +	return id;
> +}
> +
> +static void free_fpga_id(enum fpga_id_type type, int id)
> +{
> +	WARN_ON(type >= FPGA_ID_MAX);
> +	mutex_lock(&fpga_id_mutex);
> +	idr_remove(fpga_ids + type, id);
> +	mutex_unlock(&fpga_id_mutex);
> +}
> +
> +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> +{
> +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> +		return FME_ID;
> +
> +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> +		return PORT_ID;
> +
> +	WARN_ON(1);

Do we really need a WARN_ON() here? Wouldn't returning an error be
nicer?
> +
> +	return FPGA_ID_MAX;
> +}
> +
> +/**
> + * build_feature_devs_info - info collected during feature dev build.
> + *
> + * @dev: device to enumerate.
> + * @cdev: the container device for all feature devices.
> + * @feature_dev: current feature device.
> + */
> +struct build_feature_devs_info {
> +	struct device *dev;
> +	struct fpga_cdev *cdev;
> +	struct platform_device *feature_dev;
> +};
> +
> +static void fpga_cdev_add_port_dev(struct fpga_cdev *cdev,
> +				   struct platform_device *port_pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&port_pdev->dev);
> +
> +	mutex_lock(&cdev->lock);
> +	list_add(&pdata->node, &cdev->port_dev_list);
> +	get_device(&pdata->dev->dev);
> +	mutex_unlock(&cdev->lock);
> +}
> +
> +/*
> + * register current feature device, it is called when we need to switch to
> + * another feature parsing or we have parsed all features on given device
> + * feature list.
> + */
> +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> +{
> +	int ret;
> +
> +	if (!binfo->feature_dev)
> +		return 0;
> +
> +	ret = platform_device_add(binfo->feature_dev);
> +	if (!ret) {
> +		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> +			fpga_cdev_add_port_dev(binfo->cdev, binfo->feature_dev);
> +		else

So if you get back FPGA_ID_MAX, it is automatically a fme_dev?
> +			binfo->cdev->fme_dev =
> +					get_device(&binfo->feature_dev->dev);
> +		/*
> +		 * reset it to avoid build_info_free() freeing their resource.
> +		 *
> +		 * The resource of successfully registered feature devices
> +		 * will be freed by platform_device_unregister(). See the
> +		 * comments in build_info_create_dev().
> +		 */
> +		binfo->feature_dev = NULL;
> +	}
> +
> +	return ret;
> +}
> +
> +static int
> +build_info_create_dev(struct build_feature_devs_info *binfo,
> +		      enum fpga_id_type type, int feature_nr, const char *name)
> +{
> +	struct platform_device *fdev;
> +	struct resource *res;
> +	struct feature_platform_data *pdata;
> +	int ret;
> +
> +	/* we will create a new device, commit current device first */
> +	ret = build_info_commit_dev(binfo);
> +	if (ret)
> +		return ret;
> +
> +	/*
> +	 * we use -ENODEV as the initialization indicator which indicates
> +	 * whether the id need to be reclaimed
> +	 */
> +	fdev = platform_device_alloc(name, -ENODEV);
> +	if (!fdev)
> +		return -ENOMEM;
> +
> +	binfo->feature_dev = fdev;
> +
> +	fdev->id = alloc_fpga_id(type, &fdev->dev);
> +	if (fdev->id < 0)
> +		return fdev->id;
> +
> +	fdev->dev.parent = &binfo->cdev->region.dev;
> +
> +	/*
> +	 * we do not need to care for the memory which is associated with
> +	 * the platform device. After calling platform_device_unregister(),
> +	 * it will be automatically freed by device's release() callback,
> +	 * platform_device_release().
> +	 */
> +	pdata = kzalloc(feature_platform_data_size(feature_nr), GFP_KERNEL);
> +	if (pdata) {
> +		pdata->dev = fdev;
> +		pdata->num = feature_nr;
> +		mutex_init(&pdata->lock);
> +	} else {
> +		return -ENOMEM;
Does this path clean up fdev->id? Does that happen in
platform_device_release() ?
> +	}
> +
> +	/*
> +	 * the count should be initialized to 0 to make sure
> +	 *__fpga_port_enable() following __fpga_port_disable()
> +	 * works properly for port device.
> +	 * and it should always be 0 for fme device.
> +	 */
> +	WARN_ON(pdata->disable_count);
> +
> +	fdev->dev.platform_data = pdata;
> +	fdev->num_resources = feature_nr;
> +	fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> +	if (!fdev->resource)
> +		return -ENOMEM;
> +
> +	return 0;
> +}
> +
> +static void build_info_free(struct build_feature_devs_info *binfo)
> +{
> +	/*
> +	 * it is a valid id, free it. See comments in
> +	 * build_info_create_dev()
> +	 */
> +	if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> +		free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> +			     binfo->feature_dev->id);
> +
> +	platform_device_put(binfo->feature_dev);
> +
> +	devm_kfree(binfo->dev, binfo);
> +}
> +
> +/*
> + * UAFU GUID is dynamic as it can be changed after FME downloads different
> + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> + * attached on port's feature list as UAFU.
> + */
> +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> +{
> +	if (!binfo->feature_dev ||
> +	    feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> +		return false;
> +
> +	return true;
> +}
> +
> +struct feature_info {
> +	const char *name;
> +	resource_size_t resource_size;
> +	int feature_index;
> +};
> +
> +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> +static struct feature_info fme_features[] = {
> +	{
> +		.name = FME_FEATURE_HEADER,
> +		.resource_size = FME_HDR_SIZE,
> +		.feature_index = FME_FEATURE_ID_HEADER,
> +	},
> +	{
> +		.name = FME_FEATURE_THERMAL_MGMT,
> +		.resource_size = FME_THERMAL_SIZE,
> +		.feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> +	},
> +	{
> +		.name = FME_FEATURE_POWER_MGMT,
> +		.resource_size = FME_POWER_SIZE,
> +		.feature_index = FME_FEATURE_ID_POWER_MGMT,
> +	},
> +	{
> +		.name = FME_FEATURE_GLOBAL_IPERF,
> +		.resource_size = FME_IPERF_SIZE,
> +		.feature_index = FME_FEATURE_ID_GLOBAL_IPERF,
> +	},
> +	{
> +		.name = FME_FEATURE_GLOBAL_ERR,
> +		.resource_size = FME_ERR_SIZE,
> +		.feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> +	},
> +	{
> +		.name = FME_FEATURE_PR_MGMT,
> +		.resource_size = FME_PR_SIZE,
> +		.feature_index = FME_FEATURE_ID_PR_MGMT,
> +	},
> +	{
> +		.name = FME_FEATURE_HSSI,
> +		.resource_size = FME_HSSI_SIZE,
> +		.feature_index = FME_FEATURE_ID_HSSI,
> +	},
> +	{
> +		.name = FME_FEATURE_GLOBAL_DPERF,
> +		.resource_size = FME_DPERF_SIZE,
> +		.feature_index = FME_FEATURE_ID_GLOBAL_DPERF,
> +	},
> +};
> +
> +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> +static struct feature_info port_features[] = {
> +	{
> +		.name = PORT_FEATURE_HEADER,
> +		.resource_size = PORT_HDR_SIZE,
> +		.feature_index = PORT_FEATURE_ID_HEADER,
> +	},
> +	{
> +		.name = PORT_FEATURE_ERR,
> +		.resource_size = PORT_ERR_SIZE,
> +		.feature_index = PORT_FEATURE_ID_ERROR,
> +	},
> +	{
> +		.name = PORT_FEATURE_UMSG,
> +		.resource_size = PORT_UMSG_SIZE,
> +		.feature_index = PORT_FEATURE_ID_UMSG,
> +	},
> +	{
> +		/* This feature isn't available for now */
> +		.name = PORT_FEATURE_PR,
> +		.resource_size = DFH_SIZE,
> +		.feature_index = PORT_FEATURE_ID_PR,
> +	},
> +	{
> +		.name = PORT_FEATURE_STP,
> +		.resource_size = PORT_STP_SIZE,
> +		.feature_index = PORT_FEATURE_ID_STP,
> +	},
> +	{
> +		/*
> +		 * For User AFU feature, its region size is not fixed, but
> +		 * reported by register PortCapability.mmio_size. Resource
> +		 * size of UAFU will be set while parse port device.
> +		 */
> +		.name = PORT_FEATURE_UAFU,
> +		.resource_size = 0,
> +		.feature_index = PORT_FEATURE_ID_UAFU,
> +	},
> +};
> +
> +static int
> +create_feature_instance(struct build_feature_devs_info *binfo,
> +			struct fpga_enum_dfl *dfl, resource_size_t ofst,
> +			struct feature_info *finfo)
> +{
> +	int index = finfo->feature_index;
> +	struct platform_device *fdev = binfo->feature_dev;
> +	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> +	struct resource *res = &fdev->resource[index];
> +
> +	if ((dfl->len - ofst < finfo->resource_size) || pdata->num < index)
> +		return -EINVAL;
> +
> +	res->start = dfl->start + ofst;
> +	res->end = res->start + finfo->resource_size - 1;
> +	res->flags = IORESOURCE_MEM;
> +	res->name = finfo->name;
> +
> +	pdata->features[index].name = finfo->name;
> +	pdata->features[index].resource_index = index;
> +	pdata->features[index].ioaddr = dfl->ioaddr + ofst;
> +
> +	return 0;
> +}
> +
> +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> +			     struct fpga_enum_dfl *dfl,
> +			     resource_size_t ofst)
> +{
> +	int ret;
> +
> +	ret = build_info_create_dev(binfo, FME_ID, FME_FEATURE_NUM,
> +				    FPGA_FEATURE_DEV_FME);
> +	if (ret)
> +		return ret;
> +
> +	return create_feature_instance(binfo, dfl, ofst,
> +				       &fme_features[FME_FEATURE_ID_HEADER]);
> +}
> +
> +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> +				     struct fpga_enum_dfl *dfl,
> +				     resource_size_t ofst)
> +{
> +	u64 v;
> +	int id;
> +
> +	v = readq(dfl->ioaddr + ofst + DFH);
> +	id = FIELD_GET(DFH_ID, v);
> +
> +	if (id >= ARRAY_SIZE(fme_features)) {
> +		dev_info(binfo->dev, "FME feature id %x is not supported yet.\n",
> +			 id);
> +		return 0;
> +	}
> +
> +	return create_feature_instance(binfo, dfl, ofst, &fme_features[id]);
> +}
> +
> +static int parse_feature_port(struct build_feature_devs_info *binfo,
> +			      struct fpga_enum_dfl *dfl,
> +			      resource_size_t ofst)
> +{
> +	int ret;
> +
> +	ret = build_info_create_dev(binfo, PORT_ID, PORT_FEATURE_NUM,
> +				    FPGA_FEATURE_DEV_PORT);
> +	if (ret)
> +		return ret;
> +
> +	return create_feature_instance(binfo, dfl, ofst,
> +				       &port_features[PORT_FEATURE_ID_HEADER]);
> +}
> +
> +static void enable_port_uafu(struct build_feature_devs_info *binfo)
> +{
> +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +	void __iomem *base;
> +	u64 v;
> +
> +	base = get_feature_ioaddr_by_index(&binfo->feature_dev->dev,
> +					   PORT_FEATURE_ID_HEADER);
> +
> +	v = readq(base + PORT_HDR_CAP);
> +	port_features[id].resource_size =
> +			FIELD_GET(PORT_CAP_MMIO_SIZE, v) << 10;
> +
> +	/*
> +	 * To enable User AFU, driver needs to clear reset bit on related port,
> +	 * otherwise the mmio space of this user AFU will be invalid.
> +	 */
> +	if (port_features[id].resource_size)
> +		fpga_port_reset(binfo->feature_dev);
> +}
> +
> +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> +				      struct fpga_enum_dfl *dfl,
> +				      resource_size_t ofst)
> +{
> +	enum port_feature_id id;
> +	u32 dfh_id;
> +	u64 v;
> +
> +	v = readq(dfl->ioaddr + ofst + DFH);
> +	dfh_id = FIELD_GET(DFH_ID, v);
> +
> +	/*
> +	 * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> +	 * which is dedicated for port-hdr.
> +	 */
> +	id = (dfh_id & 0x000f) + 1;
> +
> +	if (id >= ARRAY_SIZE(port_features)) {
> +		dev_info(binfo->dev, "Port feature id %x is not supported yet.\n",
> +			 dfh_id);
> +		return 0;
> +	}
> +
> +	return create_feature_instance(binfo, dfl, ofst, &port_features[id]);
> +}
> +
> +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> +				   struct fpga_enum_dfl *dfl,
> +				   resource_size_t ofst)
> +{
> +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +	int ret;
> +
> +	if (port_features[id].resource_size) {
> +		ret = create_feature_instance(binfo, dfl, ofst,
> +					      &port_features[id]);
> +		port_features[id].resource_size = 0;
> +	} else {
> +		dev_err(binfo->dev, "the uafu feature header is mis-configured.\n");
> +		ret = -EINVAL;
> +	}
> +
> +	return ret;
> +}
> +
> +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> +			      struct fpga_enum_dfl *dfl,
> +			      resource_size_t ofst)
> +{
> +	void __iomem *start = dfl->ioaddr + ofst;
> +	void __iomem *end = dfl->ioaddr + dfl->len;
> +	u32 offset;
> +	u64 v;
> +	int ret;
> +
> +	for (; start < end; start += offset) {
> +		if (end - start < AFU_DFH_SIZE)
> +			return -EINVAL;
> +
> +		if (feature_is_UAFU(binfo))
> +			ret = parse_feature_port_uafu(binfo, dfl,
> +						      start - dfl->ioaddr);
> +			if (ret)
> +				return ret;
> +
> +		v = readq(start + NEXT_AFU);
> +
> +		offset = FIELD_GET(NEXT_AFU_NEXT_DFH_OFST, v);
> +		if (!offset)
> +			break;
> +	}
> +
> +	return 0;
> +}
> +
> +static int parse_feature_fiu(struct build_feature_devs_info *binfo,
> +			     struct fpga_enum_dfl *dfl,
> +			     resource_size_t ofst)
> +{
> +	u32 id, offset;
> +	u64 v;
> +	int ret = 0;
> +
> +	v = readq(dfl->ioaddr + ofst + DFH);
> +	id = FIELD_GET(DFH_ID, v);
> +
> +	switch (id) {
> +	case DFH_ID_FIU_FME:
> +		return parse_feature_fme(binfo, dfl, ofst);
> +	case DFH_ID_FIU_PORT:
> +		ret = parse_feature_port(binfo, dfl, ofst);
> +		enable_port_uafu(binfo);
> +		if (ret)
> +			return ret;
> +
> +		/* Check Port FIU's next_afu pointer to User AFU DFH */
> +		v = readq(dfl->ioaddr + ofst + NEXT_AFU);
> +
> +		offset = FIELD_GET(NEXT_AFU_NEXT_DFH_OFST, v);
> +		if (offset)
> +			return parse_feature_afus(binfo, dfl, ofst + offset);
> +
> +		dev_dbg(binfo->dev, "No AFUs detected on Port\n");
> +		break;
> +	default:
> +		dev_info(binfo->dev, "FIU TYPE %d is not supported yet.\n",
> +			 id);
> +	}
> +
> +	return ret;
> +}
> +
> +static int parse_feature_private(struct build_feature_devs_info *binfo,
> +				 struct fpga_enum_dfl *dfl,
> +				 resource_size_t ofst)
> +{
> +	u64 v;
> +	u32 id;
> +
> +	v = readq(dfl->ioaddr + ofst + DFH);
> +	id = FIELD_GET(DFH_ID, v);
> +
> +	if (!binfo->feature_dev) {
> +		dev_err(binfo->dev, "the private feature %x does not belong to any AFU.\n",
> +			id);
> +		return -EINVAL;
> +	}
> +
> +	switch (feature_dev_id_type(binfo->feature_dev)) {
> +	case FME_ID:
> +		return parse_feature_fme_private(binfo, dfl, ofst);
> +	case PORT_ID:
> +		return parse_feature_port_private(binfo, dfl, ofst);
> +	default:
> +		dev_info(binfo->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> +			 id, binfo->feature_dev->name);
> +	}
> +	return 0;
> +}
> +
> +/**
> + * parse_feature - parse a feature on given device feature list
> + *
> + * @binfo: build feature devices information.
> + * @dfl: device feature list to parse
> + * @ofst: offset to feature header on this device feature list
> + */
> +static int parse_feature(struct build_feature_devs_info *binfo,
> +			 struct fpga_enum_dfl *dfl, resource_size_t ofst)
> +{
> +	u64 v;
> +	u32 type;
> +
> +	v = readq(dfl->ioaddr + ofst + DFH);
> +	type = FIELD_GET(DFH_TYPE, v);
> +
> +	switch (type) {
> +	case DFH_TYPE_AFU:
> +		return parse_feature_afus(binfo, dfl, ofst);
> +	case DFH_TYPE_PRIVATE:
> +		return parse_feature_private(binfo, dfl, ofst);
> +	case DFH_TYPE_FIU:
> +		return parse_feature_fiu(binfo, dfl, ofst);
> +	default:
> +		dev_info(binfo->dev,
> +			 "Feature Type %x is not supported.\n", type);
> +	}
> +
> +	return 0;
> +}
> +
> +static int parse_feature_list(struct build_feature_devs_info *binfo,
> +			      struct fpga_enum_dfl *dfl)
> +{
> +	void __iomem *start = dfl->ioaddr;
> +	void __iomem *end = dfl->ioaddr + dfl->len;
> +	int ret = 0;
> +	u32 ofst = 0;
> +	u64 v;
> +
> +	/* walk through the device feature list via DFH's next DFH pointer. */
> +	for (; start < end; start += ofst) {
> +		if (end - start < DFH_SIZE) {
> +			dev_err(binfo->dev, "The region is too small to contain a feature.\n");
> +			return -EINVAL;
> +		}
> +
> +		ret = parse_feature(binfo, dfl, start - dfl->ioaddr);
> +		if (ret)
> +			return ret;
> +
> +		v = readq(start + DFH);
> +		ofst = FIELD_GET(DFH_NEXT_HDR_OFST, v);
> +
> +		/* stop parsing if EOL(End of List) is set or offset is 0 */
> +		if ((v & DFH_EOL) || !ofst)
> +			break;
> +	}
> +
> +	/* commit current feature device when reach the end of list */
> +	return build_info_commit_dev(binfo);
> +}
> +
> +struct fpga_enum_info *fpga_enum_info_alloc(struct device *dev)
> +{
> +	struct fpga_enum_info *info;
> +
> +	get_device(dev);
> +
> +	info = devm_kzalloc(dev, sizeof(*info), GFP_KERNEL);
> +	if (!info) {
> +		put_device(dev);
> +		return NULL;
> +	}
> +
> +	info->dev = dev;
> +	INIT_LIST_HEAD(&info->dfls);
> +
> +	return info;
> +}
> +EXPORT_SYMBOL_GPL(fpga_enum_info_alloc);
> +
> +int fpga_enum_info_add_dfl(struct fpga_enum_info *info, resource_size_t start,
> +			   resource_size_t len, void __iomem *ioaddr)
> +{
> +	struct fpga_enum_dfl *dfl;
> +
> +	dfl = devm_kzalloc(info->dev, sizeof(*dfl), GFP_KERNEL);
> +	if (!dfl)
> +		return -ENOMEM;
> +
> +	dfl->start = start;
> +	dfl->len = len;
> +	dfl->ioaddr = ioaddr;
> +
> +	list_add_tail(&dfl->node, &info->dfls);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(fpga_enum_info_add_dfl);
> +
> +void fpga_enum_info_free(struct fpga_enum_info *info)
> +{
> +	struct fpga_enum_dfl *tmp, *dfl;
> +	struct device *dev;
> +
> +	if (!info)
> +		return;
> +
> +	dev = info->dev;
> +
> +	/* remove all device feature lists in the list. */
> +	list_for_each_entry_safe(dfl, tmp, &info->dfls, node) {
> +		list_del(&dfl->node);
> +		devm_kfree(dev, dfl);
> +	}
> +
> +	devm_kfree(dev, info);
> +	put_device(dev);
> +}
> +EXPORT_SYMBOL_GPL(fpga_enum_info_free);
> +
> +static int remove_feature_dev(struct device *dev, void *data)
> +{
> +	struct platform_device *pdev = to_platform_device(dev);
> +	enum fpga_id_type type = feature_dev_id_type(pdev);
> +	int id = pdev->id;
> +
> +	platform_device_unregister(pdev);
> +
> +	free_fpga_id(type, id);
> +
> +	return 0;
> +}
> +
> +static void remove_feature_devs(struct fpga_cdev *cdev)
> +{
> +	device_for_each_child(&cdev->region.dev, NULL, remove_feature_dev);
> +}
> +
> +/**
> + * fpga_enumerate_feature_devs - enumerate feature devices
> + * @info: information for enumeration.
> + *
> + * This function creates a container device (base FPGA region), enumerates
> + * feature devices based on the enumeration info and creates platform devices
> + * under the container device.
> + *
> + * Return: fpga_cdev struct on success, -errno on failure
> + */
> +struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info)
> +{
> +	struct build_feature_devs_info *binfo;
> +	struct fpga_cdev *cdev;
> +	struct fpga_enum_dfl *dfl;
> +	int ret = 0;
> +
> +	if (!info->dev)
> +		return ERR_PTR(-ENODEV);
> +
> +	cdev = devm_kzalloc(info->dev, sizeof(*cdev), GFP_KERNEL);
> +	if (!cdev)
> +		return ERR_PTR(-ENOMEM);
> +
> +	cdev->parent = info->dev;
> +	mutex_init(&cdev->lock);
> +	INIT_LIST_HEAD(&cdev->port_dev_list);
> +	cdev->region.parent = info->dev;
> +
> +	ret = fpga_region_register(&cdev->region);
> +	if (ret)
> +		goto free_cdev_exit;
> +
> +	/* create and init build info for enumeration */
> +	binfo = devm_kzalloc(info->dev, sizeof(*binfo), GFP_KERNEL);
> +	if (!binfo) {
> +		ret = -ENOMEM;
> +		goto unregister_region_exit;
> +	}
> +
> +	binfo->dev = info->dev;
> +	binfo->cdev = cdev;
> +
> +	/*
> +	 * start enumeration for all feature devices based on Device Feature
> +	 * Lists.
> +	 */
> +	list_for_each_entry(dfl, &info->dfls, node) {
> +		ret = parse_feature_list(binfo, dfl);
> +		if (ret) {
> +			remove_feature_devs(cdev);
> +			build_info_free(binfo);
> +			goto unregister_region_exit;
> +		}
> +	}
> +
> +	build_info_free(binfo);
> +
> +	return cdev;
> +
> +unregister_region_exit:
> +	fpga_region_unregister(&cdev->region);
> +free_cdev_exit:
> +	devm_kfree(cdev->parent, cdev);
> +	return ERR_PTR(ret);
> +}
> +EXPORT_SYMBOL_GPL(fpga_enumerate_feature_devs);
> +
> +/**
> + * fpga_remove_feature_devs - remove all feature devices
> + * @cdev: fpga container device.
> + *
> + * Remove the container device and all feature devices under given container
> + * devices.
> + */
> +void fpga_remove_feature_devs(struct fpga_cdev *cdev)
> +{
> +	struct feature_platform_data *pdata, *ptmp;
> +
> +	remove_feature_devs(cdev);
> +
> +	mutex_lock(&cdev->lock);
> +	if (cdev->fme_dev) {
> +		/* the fme should be unregistered. */
> +		WARN_ON(device_is_registered(cdev->fme_dev));
> +		put_device(cdev->fme_dev);
> +	}
> +
> +	list_for_each_entry_safe(pdata, ptmp, &cdev->port_dev_list, node) {
> +		struct platform_device *port_dev = pdata->dev;
> +
> +		/* the port should be unregistered. */
> +		WARN_ON(device_is_registered(&port_dev->dev));
> +		list_del(&pdata->node);
> +		put_device(&port_dev->dev);
> +	}
> +	mutex_unlock(&cdev->lock);
> +
> +	fpga_region_unregister(&cdev->region);
> +	devm_kfree(cdev->parent, cdev);
> +}
> +EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);
> +
> +int fpga_port_id(struct platform_device *pdev)
> +{
> +	void __iomem *base;
> +
> +	base = get_feature_ioaddr_by_index(&pdev->dev, PORT_FEATURE_ID_HEADER);
> +	WARN_ON(!base);
> +
> +	return FIELD_GET(PORT_CAP_PORT_NUM, readq(base + FME_HDR_CAP));
> +}
> +EXPORT_SYMBOL_GPL(fpga_port_id);
> +
> +/*
> + * Enable Port by clear the port soft reset bit, which is set by default.
> + * The User AFU is unable to respond to any MMIO access while in reset.
> + * __fpga_port_enable function should only be used after __fpga_port_disable
> + * function.
> + */
> +void __fpga_port_enable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	void __iomem *base;
> +	u64 v;
> +
> +	WARN_ON(!pdata->disable_count);
> +
> +	if (--pdata->disable_count != 0)
> +		return;
> +
> +	base = get_feature_ioaddr_by_index(&pdev->dev, PORT_FEATURE_ID_HEADER);
> +	WARN_ON(!base);
> +
> +	/* Clear port soft reset */
> +	v = readq(base + PORT_HDR_CTRL);
> +	v &= ~PORT_CTRL_SFTRST;
> +	writeq(v, base + PORT_HDR_CTRL);
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> +
> +#define RST_POLL_INVL 10 /* us */
> +#define RST_POLL_TIMEOUT 1000 /* us */
> +
> +int __fpga_port_disable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	void __iomem *base;
> +	u64 v;
> +
> +	if (pdata->disable_count++ != 0)
> +		return 0;
> +
> +	base = get_feature_ioaddr_by_index(&pdev->dev, PORT_FEATURE_ID_HEADER);
> +	WARN_ON(!base);
> +
> +	/* Set port soft reset */
> +	v = readq(base + PORT_HDR_CTRL);
> +	v |= PORT_CTRL_SFTRST;
> +	writeq(v, base + PORT_HDR_CTRL);
> +
> +	/*
> +	 * HW sets ack bit to 1 when all outstanding requests have been drained
> +	 * on this port and minimum soft reset pulse width has elapsed.
> +	 * Driver polls port_soft_reset_ack to determine if reset done by HW.
> +	 */
> +	if (readq_poll_timeout(base + PORT_HDR_CTRL, v, v & PORT_CTRL_SFTRST,
> +			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> +		dev_err(&pdev->dev, "timeout, fail to reset device\n");
> +		return -ETIMEDOUT;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> +
> +static int __init dfl_fpga_init(void)
> +{
> +	fpga_ids_init();
> +
> +	return 0;
> +}
> +
> +static void __exit dfl_fpga_exit(void)
> +{
> +	fpga_ids_destroy();
> +}
> +
> +module_init(dfl_fpga_init);
> +module_exit(dfl_fpga_exit);
> +
> +MODULE_DESCRIPTION("FPGA Device Feature List (DFL) Support");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("GPL v2");
> diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> new file mode 100644
> index 0000000..abcbe57
> --- /dev/null
> +++ b/drivers/fpga/fpga-dfl.h
> @@ -0,0 +1,365 @@
> +/*
> + * Driver Header File for FPGA Device Feature List (DFL) Support
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2.
> + * SPDX-License-Identifier: GPL-2.0
> + */
> +
> +#ifndef __DFL_FPGA_H
> +#define __DFL_FPGA_H
> +
> +#include <linux/bitfield.h>
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/iopoll.h>
> +#include <linux/io-64-nonatomic-lo-hi.h>
> +#include <linux/platform_device.h>
> +#include <linux/slab.h>
> +#include <linux/uuid.h>
> +#include <linux/fpga/fpga-region.h>
> +
> +/* maximum supported number of ports */
> +#define MAX_FPGA_PORT_NUM 4
> +/* plus one for fme device */
> +#define MAX_FEATURE_DEV_NUM	(MAX_FPGA_PORT_NUM + 1)
> +
> +#define FME_FEATURE_HEADER          "fme_hdr"
> +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> +#define FME_FEATURE_POWER_MGMT      "fme_power"
> +#define FME_FEATURE_GLOBAL_IPERF    "fme_iperf"
> +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> +#define FME_FEATURE_PR_MGMT         "fme_pr"
> +#define FME_FEATURE_HSSI            "fme_hssi"
> +#define FME_FEATURE_GLOBAL_DPERF    "fme_dperf"
> +
> +#define PORT_FEATURE_HEADER         "port_hdr"
> +#define PORT_FEATURE_UAFU           "port_uafu"
> +#define PORT_FEATURE_ERR            "port_err"
> +#define PORT_FEATURE_UMSG           "port_umsg"
> +#define PORT_FEATURE_PR             "port_pr"
> +#define PORT_FEATURE_STP            "port_stp"
> +
> +/* Device Feature Header Register Set */
> +#define DFH			0x0
> +#define GUID_L			0x8
> +#define GUID_H			0x10
> +#define NEXT_AFU		0x18
> +
> +/* Device Feature Header Register Bitfield */
> +#define DFH_ID			GENMASK_ULL(11, 0)	/* Feature ID */
> +#define DFH_ID_FIU_FME		0
> +#define DFH_ID_FIU_PORT		1
> +#define DFH_REVISION		GENMASK_ULL(15, 12)	/* Feature revision */
> +#define DFH_NEXT_HDR_OFST	GENMASK_ULL(39, 16)	/* Offset to next DFH */
> +#define DFH_EOL			BIT(40)			/* End of list */
> +#define DFH_TYPE		GENMASK_ULL(63, 60)	/* Feature type */
> +#define DFH_TYPE_AFU		1
> +#define DFH_TYPE_PRIVATE	3
> +#define DFH_TYPE_FIU		4
> +
> +/* Next AFU Register Bitfield */
> +#define NEXT_AFU_NEXT_DFH_OFST	GENMASK_ULL(23, 0)	/* Offset to next AFU */
> +
> +/*
> + * It only has DFH register as header for private feature, but for FIU/AFU
> + * For FIU/AFU features, they all have DFH + GUID + NEXT_AFU as common header
> + * registers.
> + */
> +#define DFH_SIZE		0x8
> +#define AFU_DFH_SIZE		0x20
> +#define FIU_DFH_SIZE		0x20
> +
> +/* FME Header Register Set */
> +#define FME_HDR_DFH		DFH
> +#define FME_HDR_AFU_GUID_L	GUID_L
> +#define FME_HDR_AFU_GUID_H	GUID_H
> +#define FME_HDR_NEXT_AFU	NEXT_AFU
> +#define FME_HDR_CAP		0x30
> +#define FME_HDR_PORT_OFST(n)	(0x38 + ((n) * 0x8))
> +#define FME_HDR_BITSTREAM_ID	0x60
> +#define FME_HDR_BITSTREAM_MD	0x68
> +#define FME_HDR_SIZE		0x70
> +
> +/* FME Fab Capability Register Bitfield */
> +#define FME_CAP_FABRIC_VERID	GENMASK_ULL(7, 0)	/* Fabric version ID */
> +#define FME_CAP_SOCKET_ID	BIT(8)			/* Socket ID */
> +#define FME_CAP_PCIE0_LINK_AVL	BIT(12)			/* PCIE0 Link */
> +#define FME_CAP_PCIE1_LINK_AVL	BIT(13)			/* PCIE1 Link */
> +#define FME_CAP_COHR_LINK_AVL	BIT(14)			/* Coherent Link */
> +#define FME_CAP_IOMMU_AVL	BIT(16)			/* IOMMU available */
> +#define FME_CAP_NUM_PORTS	GENMASK_ULL(19, 17)	/* Number of ports */
> +#define FME_CAP_ADDR_WIDTH	GENMASK_ULL(29, 24)	/* Address bus width */
> +#define FME_CAP_CACHE_SIZE	GENMASK_ULL(43, 32)	/* cache size in KB */
> +#define FME_CAP_CACHE_ASSOC	GENMASK_ULL(47, 44)	/* Associativity */
> +
> +/* FME Port Offset Register Bitfield */
> +/* Offset to port device feature header */
> +#define FME_PORT_OFST_DFH_OFST	GENMASK_ULL(23, 0)
> +/* PCI Bar ID for this port */
> +#define FME_PORT_OFST_BAR_ID	GENMASK_ULL(34, 32)
> +/* AFU MMIO access permission. 1 - VF, 0 - PF. */
> +#define FME_PORT_OFST_ACC_CTRL	BIT(55)
> +#define FME_PORT_OFST_ACC_PF	0
> +#define FME_PORT_OFST_ACC_VF	1
> +#define FME_PORT_OFST_IMP	BIT(60)
> +
> +/* FME Thermal Sub Feature Register Set */
> +#define FME_THERMAL_DFH		DFH
> +#define FME_THERMAL_SIZE	DFH_SIZE
> +
> +/* FME Power Sub Feature Register Set */
> +#define FME_POWER_DFH		DFH
> +#define FME_POWER_SIZE		DFH_SIZE
> +
> +/* FME Global Performance Sub Feature Register Set */
> +#define FME_IPERF_DFH		DFH
> +#define FME_IPERF_SIZE		DFH_SIZE
> +
> +/* FME Global Error Sub Feature Register Set */
> +#define FME_ERR_DFH		DFH
> +#define FME_ERR_SIZE		DFH_SIZE
> +
> +/* FME Partial Reconfiguration Sub Feature Register Set */
> +#define FME_PR_DFH		DFH
> +#define FME_PR_SIZE		DFH_SIZE
> +
> +/* FME HSSI Sub Feature Register Set */
> +#define FME_HSSI_DFH		DFH
> +#define FME_HSSI_SIZE		DFH_SIZE
> +
> +/* FME Global Performance Sub Feature Register Set */
> +#define FME_DPERF_DFH		DFH
> +#define FME_DPERF_SIZE		DFH_SIZE
> +
> +/* PORT Header Register Set */
> +#define PORT_HDR_DFG		DFH
> +#define PORT_HDR_AFU_GUID_L	GUID_L
> +#define PORT_HDR_AFU_GUID_H	GUID_H
> +#define PORT_HDR_NEXT_AFU	NEXT_AFU
> +#define PORT_HDR_CAP		0x30
> +#define PORT_HDR_CTRL		0x38
> +#define PORT_HDR_SIZE		0x40
> +
> +/* Port Capability Register Bitfield */
> +#define PORT_CAP_PORT_NUM	GENMASK(1, 0)		/* ID of this port */
> +#define PORT_CAP_MMIO_SIZE	GENMASK(23, 8)		/* MMIO size in KB */
> +#define PORT_CAP_SUPP_INT_NUM	GENMASK(35, 32)		/* Interrupts num */
> +
> +/* Port Control Register Bitfield */
> +#define PORT_CTRL_SFTRST	BIT(0)			/* Port soft reset */
> +/* Latency tolerance reporting. '1' >= 40us, '0' < 40us.*/
> +#define PORT_CTRL_LATENCY	BIT(2)
> +#define PORT_CTRL_SFTRST_ACK	BIT(4)			/* HW ack for reset */
> +
> +/* PORT Error Sub Feature Register Set */
> +#define PORT_ERR_DFH		DFH
> +#define PORT_ERR_SIZE		DFH_SIZE
> +
> +/* PORT Unordered Message Sub Feature Register Set */
> +#define PORT_UMSG_DFH		DFH
> +#define PORT_UMSG_SIZE		DFH_SIZE
> +
> +/* PORT SignalTap Sub Feature Register Set */
> +#define PORT_STP_DFH		DFH
> +#define PORT_STP_SIZE		DFH_SIZE
> +
> +/* PORT User AFU Sub Feature Register Set */
> +#define PORT_UAFU_DFH		DFH
> +#define PORT_UAFU_SIZE		DFH_SIZE
> +
> +struct feature {
> +	const char *name;
> +	int resource_index;
> +	void __iomem *ioaddr;
> +};
> +
> +struct feature_platform_data {
> +	/* list the feature dev to cci_drvdata->port_dev_list. */
> +	struct list_head node;
> +	struct mutex lock;		/* protect platform data */
> +	struct platform_device *dev;
> +	unsigned int disable_count;	/* count for port disable */
> +
> +	int num;			/* number of features */
> +	struct feature features[0];
> +};
> +
> +enum fme_feature_id {
> +	FME_FEATURE_ID_HEADER = 0x0,
> +	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> +	FME_FEATURE_ID_POWER_MGMT = 0x2,
> +	FME_FEATURE_ID_GLOBAL_IPERF = 0x3,
> +	FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> +	FME_FEATURE_ID_PR_MGMT = 0x5,
> +	FME_FEATURE_ID_HSSI = 0x6,
> +	FME_FEATURE_ID_GLOBAL_DPERF = 0x7,
> +	FME_FEATURE_ID_MAX = 0x8,
> +};
> +
> +enum port_feature_id {
> +	PORT_FEATURE_ID_HEADER = 0x0,
> +	PORT_FEATURE_ID_ERROR = 0x1,
> +	PORT_FEATURE_ID_UMSG = 0x2,
> +	PORT_FEATURE_ID_PR = 0x3,
> +	PORT_FEATURE_ID_STP = 0x4,
> +	PORT_FEATURE_ID_UAFU = 0x5,
> +	PORT_FEATURE_ID_MAX = 0x6,
> +};
> +
> +#define FME_FEATURE_NUM			FME_FEATURE_ID_MAX
> +#define PORT_FEATURE_NUM		PORT_FEATURE_ID_MAX
> +
> +#define FPGA_FEATURE_DEV_FME		"fpga-dfl-fme"
> +#define FPGA_FEATURE_DEV_PORT		"fpga-dfl-port"
> +
> +static inline int feature_platform_data_size(const int num)
> +{
> +	return sizeof(struct feature_platform_data) +
> +		num * sizeof(struct feature);
> +}
> +
> +int fpga_port_id(struct platform_device *pdev);
> +
> +static inline int fpga_port_check_id(struct platform_device *pdev,
> +				     void *pport_id)
> +{
> +	return fpga_port_id(pdev) == *(int *)pport_id;
> +}
> +
> +void __fpga_port_enable(struct platform_device *pdev);
> +int __fpga_port_disable(struct platform_device *pdev);
> +
> +static inline void fpga_port_enable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +
> +	mutex_lock(&pdata->lock);
> +	__fpga_port_enable(pdev);
> +	mutex_unlock(&pdata->lock);
> +}
> +
> +static inline int fpga_port_disable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	int ret;
> +
> +	mutex_lock(&pdata->lock);
> +	ret = __fpga_port_disable(pdev);
> +	mutex_unlock(&pdata->lock);
> +
> +	return ret;
> +}
> +
> +static inline int __fpga_port_reset(struct platform_device *pdev)
> +{
> +	int ret;
> +
> +	ret = __fpga_port_disable(pdev);
> +	if (ret)
> +		return ret;
> +
> +	__fpga_port_enable(pdev);
> +
> +	return 0;
> +}
> +
> +static inline int fpga_port_reset(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	int ret;
> +
> +	mutex_lock(&pdata->lock);
> +	ret = __fpga_port_reset(pdev);
> +	mutex_unlock(&pdata->lock);
> +
> +	return ret;
> +}
> +
> +static inline void __iomem *
> +get_feature_ioaddr_by_index(struct device *dev, int index)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(dev);
> +
> +	return pdata->features[index].ioaddr;
> +}
> +
> +static inline bool feature_is_fme(void __iomem *base)
> +{
> +	u64 v = readq(base + DFH);
> +
> +	return (FIELD_GET(DFH_TYPE, v) == DFH_TYPE_FIU) &&
> +		(FIELD_GET(DFH_ID, v) == DFH_ID_FIU_FME);
> +}
> +
> +static inline bool feature_is_port(void __iomem *base)
> +{
> +	u64 v = readq(base + DFH);
> +
> +	return (FIELD_GET(DFH_TYPE, v) == DFH_TYPE_FIU) &&
> +		(FIELD_GET(DFH_ID, v) == DFH_ID_FIU_PORT);
> +}
> +
> +/**
> + * fpga_enum_info - FPGA enumeration information
> + *
> + * @dev: parent device.
> + * @dfls: list of device feature lists.
> + */
> +struct fpga_enum_info {
> +	struct device *dev;
> +	struct list_head dfls;
> +};
> +
> +/**
> + * fpga_enum_dfl - FPGA enumeration device feature list information
> + *
> + * @start: base address of this device feature list.
> + * @len: size of this device feature list.
> + * @ioaddr: mapped base address of this device feature list.
> + * @node: node in list of device feature lists.
> + */
> +struct fpga_enum_dfl {
> +	resource_size_t start;
> +	resource_size_t len;
> +
> +	void __iomem *ioaddr;
> +
> +	struct list_head node;
> +};
> +
> +struct fpga_enum_info *fpga_enum_info_alloc(struct device *dev);
> +int fpga_enum_info_add_dfl(struct fpga_enum_info *info, resource_size_t start,
> +			   resource_size_t len, void __iomem *ioaddr);
> +void fpga_enum_info_free(struct fpga_enum_info *info);
> +
> +/**
> + * fpga_cdev - fpga container device
> + * @parent: parent device of this container device.
> + * @region: base fpga region.
> + * @fme_dev: FME feature device under this container device.
> + * @lock: mutex lock to protect the port device list.
> + * @port_dev_list: list of all port feature devices under this container device.
> + */
> +struct fpga_cdev {
> +	struct device *parent;
> +
> +	struct fpga_region region;
> +
> +	struct device *fme_dev;
> +
> +	struct mutex lock; /* to protect the port device list */
> +	struct list_head port_dev_list;
> +};
> +
> +struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info);
> +void fpga_remove_feature_devs(struct fpga_cdev *cdev);
> +
> +#endif /* __DFL_FPGA_H */
> -- 
> 1.8.3.1
> 

Thanks,

Moritz

Re: [PATCH v3 04/21] fpga: add device feature list support

[Wu Hao]

On Tue, Nov 28, 2017 at 10:07:36PM -0800, Moritz Fischer wrote:
> Hi Hao,
> 
> first pass, I didn't get all the way through, yet.

Hi Moritz

Thanks a lot for your review and comments. :)

> 
> On Mon, Nov 27, 2017 at 02:42:11PM +0800, Wu Hao wrote:
> > Device Feature List (DFL) defines a feature list structure that creates
> > a link list of feature headers within the MMIO space to provide an
> > extensible way of adding features. This patch introduces a kernel module
> > to provide basic infrastructure to support FPGA devices which implement
> > the Device Feature List.
> > 
> > Usually there will be different features and their sub features linked into
> > the DFL. This code provides common APIs for feature enumeration, it creates
> > a container device (FPGA base region), walks through the DFLs and creates
> > platform devices for feature devices (Currently it only supports two
> > different feature devices, FPGA Management Engine (FME) and Port which
> > the Accelerator Function Unit (AFU) connected to). In order to enumerate
> > the DFLs, the common APIs required low level driver to provide necessary
> > enumeration information (e.g address for each device feature list for
> > given device) and fill it to the fpga_enum_info data structure. Please
> > refer to below description for APIs added for enumeration.
> > 
> > Functions for enumeration information preparation:
> >  *fpga_enum_info_alloc
> >    allocate enumeration information data structure.
> > 
> >  *fpga_enum_info_add_dfl
> >    add a device feature list to fpga_enum_info data structure.
> > 
> >  *fpga_enum_info_free
> >    free fpga_enum_info data structure and related resources.
> > 
> > Functions for feature device enumeration:
> >  *fpga_enumerate_feature_devs
> >    enumerate feature devices and return container device.
> > 
> >  *fpga_remove_feature_devs
> >    remove feature devices under given container device.
> > 
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ----
> > v3: split from another patch.
> >     separate dfl enumeration code from original pcie driver.
> >     provide common data structures and APIs for enumeration.
> >     update device feature list parsing process according to latest hw.
> >     add dperf/iperf/hssi sub feature placeholder according to latest hw.
> >     remove build_info_add_sub_feature and other small functions.
> >     replace *_feature_num function with macro.
> >     remove writeq/readq.
> > ---
> >  drivers/fpga/Kconfig    |  16 +
> >  drivers/fpga/Makefile   |   3 +
> >  drivers/fpga/fpga-dfl.c | 884 ++++++++++++++++++++++++++++++++++++++++++++++++
> >  drivers/fpga/fpga-dfl.h | 365 ++++++++++++++++++++
> >  4 files changed, 1268 insertions(+)
> >  create mode 100644 drivers/fpga/fpga-dfl.c
> >  create mode 100644 drivers/fpga/fpga-dfl.h
> > 
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > index f47ef84..01ad31f 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -124,4 +124,20 @@ config OF_FPGA_REGION
> >  	  Support for loading FPGA images by applying a Device Tree
> >  	  overlay.
> >  
> > +config FPGA_DFL
> > +	tristate "FPGA Device Feature List (DFL) support"
> > +	select FPGA_BRIDGE
> > +	select FPGA_REGION
> > +	help
> > +	  Device Feature List (DFL) defines a feature list structure that
> > +	  creates a link list of feature headers within the MMIO space
> > +	  to provide an extensible way of adding features for FPGA.
> > +	  Driver can walk through the feature headers to enumerate feature
> > +	  devices (e.g FPGA Management Engine, Port and Accelerator
> > +	  Function Unit) and their private features for target FPGA devices.
> > +
> > +	  Select this option to enable common support for Field-Programmable
> > +	  Gate Array (FPGA) solutions which implement Device Feature List.
> > +	  It provides enumeration APIs, and feature device infrastructure.
> > +
> >  endif # FPGA
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index 3cb276a..447ba2b 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -27,3 +27,6 @@ obj-$(CONFIG_XILINX_PR_DECOUPLER)	+= xilinx-pr-decoupler.o
> >  # High Level Interfaces
> >  obj-$(CONFIG_FPGA_REGION)		+= fpga-region.o
> >  obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
> > +
> > +# FPGA Device Feature List Support
> > +obj-$(CONFIG_FPGA_DFL)			+= fpga-dfl.o
> > diff --git a/drivers/fpga/fpga-dfl.c b/drivers/fpga/fpga-dfl.c
> > new file mode 100644
> > index 0000000..6609828
> > --- /dev/null
> > +++ b/drivers/fpga/fpga-dfl.c
> > @@ -0,0 +1,884 @@
> > +/*
> > + * Driver for FPGA Device Feature List (DFL) Support
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2.
> 
> This is redundant.
> > + * SPDX-License-Identifier: GPL-2.0
> 
> Also I think the current consensus is that this should go in the first
> line

Sure, I will put this SPDX-License-Identifier to the first line and remove
the redundant line above.

> > + */
> > +#include <linux/module.h>
> > +
> > +#include "fpga-dfl.h"
> > +
> > +static DEFINE_MUTEX(fpga_id_mutex);
> > +
> > +enum fpga_id_type {
> > +	FME_ID,		/* fme id allocation and mapping */
> > +	PORT_ID,	/* port id allocation and mapping */
> > +	FPGA_ID_MAX,
> > +};
> > +
> > +/* it is protected by fpga_id_mutex */
> > +static struct idr fpga_ids[FPGA_ID_MAX];
> > +
> > +static void fpga_ids_init(void)
> > +{
> > +	int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> > +		idr_init(fpga_ids + i);
> > +}
> > +
> > +static void fpga_ids_destroy(void)
> > +{
> > +	int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> > +		idr_destroy(fpga_ids + i);
> > +}
> > +
> > +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> > +{
> > +	int id;
> > +
> > +	WARN_ON(type >= FPGA_ID_MAX);
> > +	mutex_lock(&fpga_id_mutex);
> > +	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> > +	mutex_unlock(&fpga_id_mutex);
> > +
> > +	return id;
> > +}
> > +
> > +static void free_fpga_id(enum fpga_id_type type, int id)
> > +{
> > +	WARN_ON(type >= FPGA_ID_MAX);
> > +	mutex_lock(&fpga_id_mutex);
> > +	idr_remove(fpga_ids + type, id);
> > +	mutex_unlock(&fpga_id_mutex);
> > +}
> > +
> > +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> > +{
> > +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> > +		return FME_ID;
> > +
> > +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> > +		return PORT_ID;
> > +
> > +	WARN_ON(1);
> 
> Do we really need a WARN_ON() here? Wouldn't returning an error be
> nicer?

Actually this is a static function only used in this file, and ideally
enumeration code will only create feature devices for FME and PORT.
So ideally there couldn't be any feature device with other names, only FME
and PORT.  WARN_ON(1) is used here, to just be a warning that there could
be a critical driver problem somewhere and return FPGA_ID_MAX as error code
which will never be used as an id for any feature device.

> > +
> > +	return FPGA_ID_MAX;
> > +}
> > +
> > +/**
> > + * build_feature_devs_info - info collected during feature dev build.
> > + *
> > + * @dev: device to enumerate.
> > + * @cdev: the container device for all feature devices.
> > + * @feature_dev: current feature device.
> > + */
> > +struct build_feature_devs_info {
> > +	struct device *dev;
> > +	struct fpga_cdev *cdev;
> > +	struct platform_device *feature_dev;
> > +};
> > +
> > +static void fpga_cdev_add_port_dev(struct fpga_cdev *cdev,
> > +				   struct platform_device *port_pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&port_pdev->dev);
> > +
> > +	mutex_lock(&cdev->lock);
> > +	list_add(&pdata->node, &cdev->port_dev_list);
> > +	get_device(&pdata->dev->dev);
> > +	mutex_unlock(&cdev->lock);
> > +}
> > +
> > +/*
> > + * register current feature device, it is called when we need to switch to
> > + * another feature parsing or we have parsed all features on given device
> > + * feature list.
> > + */
> > +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> > +{
> > +	int ret;
> > +
> > +	if (!binfo->feature_dev)
> > +		return 0;
> > +
> > +	ret = platform_device_add(binfo->feature_dev);
> > +	if (!ret) {
> > +		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> > +			fpga_cdev_add_port_dev(binfo->cdev, binfo->feature_dev);
> > +		else
> 
> So if you get back FPGA_ID_MAX, it is automatically a fme_dev?

If FPGA_ID_MAX returned, there would be a WARN_ON(1) triggered, so we get
notified there would be a critical driver issue somewhere, needs to be fixed
firstly.

In normal driver flow, we should never see FPGA_ID_MAX returned and the
WARN_ON(1). :)

> > +			binfo->cdev->fme_dev =
> > +					get_device(&binfo->feature_dev->dev);
> > +		/*
> > +		 * reset it to avoid build_info_free() freeing their resource.
> > +		 *
> > +		 * The resource of successfully registered feature devices
> > +		 * will be freed by platform_device_unregister(). See the
> > +		 * comments in build_info_create_dev().
> > +		 */
> > +		binfo->feature_dev = NULL;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static int
> > +build_info_create_dev(struct build_feature_devs_info *binfo,
> > +		      enum fpga_id_type type, int feature_nr, const char *name)
> > +{
> > +	struct platform_device *fdev;
> > +	struct resource *res;
> > +	struct feature_platform_data *pdata;
> > +	int ret;
> > +
> > +	/* we will create a new device, commit current device first */
> > +	ret = build_info_commit_dev(binfo);
> > +	if (ret)
> > +		return ret;
> > +
> > +	/*
> > +	 * we use -ENODEV as the initialization indicator which indicates
> > +	 * whether the id need to be reclaimed
> > +	 */
> > +	fdev = platform_device_alloc(name, -ENODEV);
> > +	if (!fdev)
> > +		return -ENOMEM;
> > +
> > +	binfo->feature_dev = fdev;
> > +
> > +	fdev->id = alloc_fpga_id(type, &fdev->dev);
> > +	if (fdev->id < 0)
> > +		return fdev->id;
> > +
> > +	fdev->dev.parent = &binfo->cdev->region.dev;
> > +
> > +	/*
> > +	 * we do not need to care for the memory which is associated with
> > +	 * the platform device. After calling platform_device_unregister(),
> > +	 * it will be automatically freed by device's release() callback,
> > +	 * platform_device_release().
> > +	 */
> > +	pdata = kzalloc(feature_platform_data_size(feature_nr), GFP_KERNEL);
> > +	if (pdata) {
> > +		pdata->dev = fdev;
> > +		pdata->num = feature_nr;
> > +		mutex_init(&pdata->lock);
> > +	} else {
> > +		return -ENOMEM;
> Does this path clean up fdev->id? Does that happen in
> platform_device_release() ?

This patch cleans up fdev->id manually, as platform_device_release can't
cover this. There are two cases, we have to clean up the fdev->id.

1) error found during enumeration (fdev is not registered yet), just like
above case, return -ENOMEM, and finally it causes parse_feature_list
function to return error code, and fdev->id will be cleaned up by
build_info_free function. (only platform_device_put required as
platform_device_add is not invoked yet).

2) normal clean up flow with registered fdev. Then fdev->id will be
cleaned up by remove_feature_dev function. (platform_device_unregister
will be used in this case). :)

Thanks
Hao

Re: [PATCH v3 04/21] fpga: add device feature list support

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

> +
> +enum port_feature_id {
> +       PORT_FEATURE_ID_HEADER = 0x0,
> +       PORT_FEATURE_ID_ERROR = 0x1,
> +       PORT_FEATURE_ID_UMSG = 0x2,
> +       PORT_FEATURE_ID_PR = 0x3,
> +       PORT_FEATURE_ID_STP = 0x4,
> +       PORT_FEATURE_ID_UAFU = 0x5,
> +       PORT_FEATURE_ID_MAX = 0x6,
> +};
> +
> +#define FME_FEATURE_NUM                        FME_FEATURE_ID_MAX
> +#define PORT_FEATURE_NUM               PORT_FEATURE_ID_MAX
> +
> +#define FPGA_FEATURE_DEV_FME           "fpga-dfl-fme"
> +#define FPGA_FEATURE_DEV_PORT          "fpga-dfl-port"
> +
> +static inline int feature_platform_data_size(const int num)
> +{
> +       return sizeof(struct feature_platform_data) +
> +               num * sizeof(struct feature);
> +}
> +
> +int fpga_port_id(struct platform_device *pdev);
> +
> +static inline int fpga_port_check_id(struct platform_device *pdev,
> +                                    void *pport_id)
> +{
> +       return fpga_port_id(pdev) == *(int *)pport_id;
> +}
> +
> +void __fpga_port_enable(struct platform_device *pdev);
> +int __fpga_port_disable(struct platform_device *pdev);
> +
> +static inline void fpga_port_enable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +
> +       mutex_lock(&pdata->lock);
> +       __fpga_port_enable(pdev);
> +       mutex_unlock(&pdata->lock);
> +}
> +
> +static inline int fpga_port_disable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = __fpga_port_disable(pdev);
> +       mutex_unlock(&pdata->lock);
> +
> +       return ret;
> +}
> +
> +static inline int __fpga_port_reset(struct platform_device *pdev)
> +{
> +       int ret;
> +
> +       ret = __fpga_port_disable(pdev);
> +       if (ret)
> +               return ret;
> +
> +       __fpga_port_enable(pdev);
> +
> +       return 0;
> +}
> +
> +static inline int fpga_port_reset(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = __fpga_port_reset(pdev);
> +       mutex_unlock(&pdata->lock);
> +
> +       return ret;
> +}

I see that the port code is included as part of the enumeration code.
This is not very future-proofed, if a different port needs to be
supported.

The port is a FPGA fabric based bridge with expanded functionality,
right?  So it is similar to the altera freeze bridge, but adds the
ability to reset the fabric and some other features are promised in
the future, IIUC.  I still think that the port could be implemented in
the bridge driver .c file instead of being here as part of the
enumeration code.   For that to happen, some APIs would need to be
added to the bridge framework and the FPGA region framework.  Then the
reset can be requested through a new FPGA region API function.

The advantage of this is that if this patchset evolves and there is
some other v2 port driver needed, it can be a different driver if it
needs to be.

If the port reset is really a fabric reset, (correct me if I'm
remembering wrongly) then it would be helpful to call it a
fabric_reset.  This would be the first bridge driver supporting fabric
reset.  I think it won't be the last.

So what I'm proposing would be added/changed would be:
* move all the bridge code to fpga-dfl-fme-br.c
* add .fabric_reset to bridge ops
* add fpga_bridges_reset to fpga-bridge.c (a new function that goes
through a list of bridges and calls the reset ops if it exists,
ignores the bridges where it doesn't exist)
* add fpga_region_fabric_reset to fpga-region.c.  This function gets
the region, gets the bridges, calls fpga_bridges_reset (can steal code
from fpga_region_program_fpga)
* the rest of the patchset can use fpga_region_fabric_reset instead of
fpga_port_reset

Alan

Re: [PATCH v3 04/21] fpga: add device feature list support

[Alan Tull]

On Wed, Dec 20, 2017 at 4:29 PM, Alan Tull <atull@kernel.org> wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
>
> Hi Hao,
>
>> +
>> +enum port_feature_id {
>> +       PORT_FEATURE_ID_HEADER = 0x0,
>> +       PORT_FEATURE_ID_ERROR = 0x1,
>> +       PORT_FEATURE_ID_UMSG = 0x2,
>> +       PORT_FEATURE_ID_PR = 0x3,
>> +       PORT_FEATURE_ID_STP = 0x4,
>> +       PORT_FEATURE_ID_UAFU = 0x5,
>> +       PORT_FEATURE_ID_MAX = 0x6,
>> +};
>> +
>> +#define FME_FEATURE_NUM                        FME_FEATURE_ID_MAX
>> +#define PORT_FEATURE_NUM               PORT_FEATURE_ID_MAX
>> +
>> +#define FPGA_FEATURE_DEV_FME           "fpga-dfl-fme"
>> +#define FPGA_FEATURE_DEV_PORT          "fpga-dfl-port"
>> +
>> +static inline int feature_platform_data_size(const int num)
>> +{
>> +       return sizeof(struct feature_platform_data) +
>> +               num * sizeof(struct feature);
>> +}
>> +
>> +int fpga_port_id(struct platform_device *pdev);
>> +
>> +static inline int fpga_port_check_id(struct platform_device *pdev,
>> +                                    void *pport_id)
>> +{
>> +       return fpga_port_id(pdev) == *(int *)pport_id;
>> +}
>> +
>> +void __fpga_port_enable(struct platform_device *pdev);
>> +int __fpga_port_disable(struct platform_device *pdev);
>> +
>> +static inline void fpga_port_enable(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +
>> +       mutex_lock(&pdata->lock);
>> +       __fpga_port_enable(pdev);
>> +       mutex_unlock(&pdata->lock);
>> +}
>> +
>> +static inline int fpga_port_disable(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       int ret;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       ret = __fpga_port_disable(pdev);
>> +       mutex_unlock(&pdata->lock);
>> +
>> +       return ret;
>> +}
>> +
>> +static inline int __fpga_port_reset(struct platform_device *pdev)
>> +{
>> +       int ret;
>> +
>> +       ret = __fpga_port_disable(pdev);
>> +       if (ret)
>> +               return ret;
>> +
>> +       __fpga_port_enable(pdev);
>> +
>> +       return 0;
>> +}
>> +
>> +static inline int fpga_port_reset(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       int ret;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       ret = __fpga_port_reset(pdev);
>> +       mutex_unlock(&pdata->lock);
>> +
>> +       return ret;
>> +}
>
> I see that the port code is included as part of the enumeration code.
> This is not very future-proofed, if a different port needs to be
> supported.
>
> The port is a FPGA fabric based bridge with expanded functionality,
> right?  So it is similar to the altera freeze bridge, but adds the
> ability to reset the fabric and some other features are promised in
> the future, IIUC.  I still think that the port could be implemented in
> the bridge driver .c file instead of being here as part of the
> enumeration code.   For that to happen, some APIs would need to be
> added to the bridge framework and the FPGA region framework.  Then the
> reset can be requested through a new FPGA region API function.
>
> The advantage of this is that if this patchset evolves and there is
> some other v2 port driver needed, it can be a different driver if it
> needs to be.
>
> If the port reset is really a fabric reset,

Actually 'fabric reset' is probably not clear enough.  It's resetting
the hardware in a partial reconfiguration region, not just resetting
the bridge.  I'm trying to come up with a term that makes that clear
what is getting reset is the contents of the region.

> (correct me if I'm
> remembering wrongly) then it would be helpful to call it a
> fabric_reset.  This would be the first bridge driver supporting fabric
> reset.  I think it won't be the last.
>
> So what I'm proposing would be added/changed would be:
> * move all the bridge code to fpga-dfl-fme-br.c
> * add .fabric_reset to bridge ops
> * add fpga_bridges_reset to fpga-bridge.c (a new function that goes
> through a list of bridges and calls the reset ops if it exists,
> ignores the bridges where it doesn't exist)
> * add fpga_region_fabric_reset to fpga-region.c.  This function gets
> the region, gets the bridges, calls fpga_bridges_reset (can steal code
> from fpga_region_program_fpga)
> * the rest of the patchset can use fpga_region_fabric_reset instead of
> fpga_port_reset
>
> Alan

Re: [PATCH v3 04/21] fpga: add device feature list support

[Wu Hao]

On Wed, Dec 20, 2017 at 06:58:01PM -0600, Alan Tull wrote:
> On Wed, Dec 20, 2017 at 4:29 PM, Alan Tull <atull@kernel.org> wrote:
> > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> >
> > Hi Hao,
> >
> >> +
> >> +enum port_feature_id {
> >> +       PORT_FEATURE_ID_HEADER = 0x0,
> >> +       PORT_FEATURE_ID_ERROR = 0x1,
> >> +       PORT_FEATURE_ID_UMSG = 0x2,
> >> +       PORT_FEATURE_ID_PR = 0x3,
> >> +       PORT_FEATURE_ID_STP = 0x4,
> >> +       PORT_FEATURE_ID_UAFU = 0x5,
> >> +       PORT_FEATURE_ID_MAX = 0x6,
> >> +};
> >> +
> >> +#define FME_FEATURE_NUM                        FME_FEATURE_ID_MAX
> >> +#define PORT_FEATURE_NUM               PORT_FEATURE_ID_MAX
> >> +
> >> +#define FPGA_FEATURE_DEV_FME           "fpga-dfl-fme"
> >> +#define FPGA_FEATURE_DEV_PORT          "fpga-dfl-port"
> >> +
> >> +static inline int feature_platform_data_size(const int num)
> >> +{
> >> +       return sizeof(struct feature_platform_data) +
> >> +               num * sizeof(struct feature);
> >> +}
> >> +
> >> +int fpga_port_id(struct platform_device *pdev);
> >> +
> >> +static inline int fpga_port_check_id(struct platform_device *pdev,
> >> +                                    void *pport_id)
> >> +{
> >> +       return fpga_port_id(pdev) == *(int *)pport_id;
> >> +}
> >> +
> >> +void __fpga_port_enable(struct platform_device *pdev);
> >> +int __fpga_port_disable(struct platform_device *pdev);
> >> +
> >> +static inline void fpga_port_enable(struct platform_device *pdev)
> >> +{
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> +
> >> +       mutex_lock(&pdata->lock);
> >> +       __fpga_port_enable(pdev);
> >> +       mutex_unlock(&pdata->lock);
> >> +}
> >> +
> >> +static inline int fpga_port_disable(struct platform_device *pdev)
> >> +{
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> +       int ret;
> >> +
> >> +       mutex_lock(&pdata->lock);
> >> +       ret = __fpga_port_disable(pdev);
> >> +       mutex_unlock(&pdata->lock);
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static inline int __fpga_port_reset(struct platform_device *pdev)
> >> +{
> >> +       int ret;
> >> +
> >> +       ret = __fpga_port_disable(pdev);
> >> +       if (ret)
> >> +               return ret;
> >> +
> >> +       __fpga_port_enable(pdev);
> >> +
> >> +       return 0;
> >> +}
> >> +
> >> +static inline int fpga_port_reset(struct platform_device *pdev)
> >> +{
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> +       int ret;
> >> +
> >> +       mutex_lock(&pdata->lock);
> >> +       ret = __fpga_port_reset(pdev);
> >> +       mutex_unlock(&pdata->lock);
> >> +
> >> +       return ret;
> >> +}
> >
> > I see that the port code is included as part of the enumeration code.
> > This is not very future-proofed, if a different port needs to be
> > supported.
> >
> > The port is a FPGA fabric based bridge with expanded functionality,
> > right?  So it is similar to the altera freeze bridge, but adds the
> > ability to reset the fabric and some other features are promised in
> > the future, IIUC.  I still think that the port could be implemented in
> > the bridge driver .c file instead of being here as part of the
> > enumeration code.   For that to happen, some APIs would need to be
> > added to the bridge framework and the FPGA region framework.  Then the
> > reset can be requested through a new FPGA region API function.
> >
> > The advantage of this is that if this patchset evolves and there is
> > some other v2 port driver needed, it can be a different driver if it
> > needs to be.
> >
> > If the port reset is really a fabric reset,
> 
> Actually 'fabric reset' is probably not clear enough.  It's resetting
> the hardware in a partial reconfiguration region, not just resetting
> the bridge.  I'm trying to come up with a term that makes that clear
> what is getting reset is the contents of the region.
> 
> > (correct me if I'm
> > remembering wrongly) then it would be helpful to call it a
> > fabric_reset.  This would be the first bridge driver supporting fabric
> > reset.  I think it won't be the last.
> >
> > So what I'm proposing would be added/changed would be:
> > * move all the bridge code to fpga-dfl-fme-br.c
> > * add .fabric_reset to bridge ops
> > * add fpga_bridges_reset to fpga-bridge.c (a new function that goes
> > through a list of bridges and calls the reset ops if it exists,
> > ignores the bridges where it doesn't exist)
> > * add fpga_region_fabric_reset to fpga-region.c.  This function gets
> > the region, gets the bridges, calls fpga_bridges_reset (can steal code
> > from fpga_region_program_fpga)
> > * the rest of the patchset can use fpga_region_fabric_reset instead of
> > fpga_port_reset

Hi Alan

Actually I think we can't move all the bridge code to fpga-dfl-fme-br.c as
this bridge (and region) is created by FME PR sub feature code, mainly for
PR function. But user may need the reset function when run some workload
on target Port/AFU, if consider virtualization case (SRIOV), there is only
Port/AFU in each VF, and no FME in VF (that means nobody creates the fpga
region/bridge/region). So it's need from port platform driver side as well.

The orignal idea that creates fpga-mgr/bridges/regions under FME, is that
even we turned all Ports/AFUs into VFs (user can not see port platform
device and the user interfaces exposed by port driver on PF), but user
still can use FME to do PR to those Ports/AFUs in turned into VFs (assigned
in different virtual machines).

I fully agree with you, that we should avoid feature specific code in the
common enumeration code and feature device framework if possible. I guess
I need some time to check and see if any other solutions (e.g export those
functions from port driver not DFL framework). Will back here once I have
some clear idea.:)

Thanks
Hao

> >
> > Alan

Re: [PATCH v3 04/21] fpga: add device feature list support

[Wu Hao]

On Thu, Dec 21, 2017 at 03:22:42PM +0800, Wu Hao wrote:
> On Wed, Dec 20, 2017 at 06:58:01PM -0600, Alan Tull wrote:
> > On Wed, Dec 20, 2017 at 4:29 PM, Alan Tull <atull@kernel.org> wrote:
> > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > >
> > > Hi Hao,
> > >
> > >> +
> > >> +enum port_feature_id {
> > >> +       PORT_FEATURE_ID_HEADER = 0x0,
> > >> +       PORT_FEATURE_ID_ERROR = 0x1,
> > >> +       PORT_FEATURE_ID_UMSG = 0x2,
> > >> +       PORT_FEATURE_ID_PR = 0x3,
> > >> +       PORT_FEATURE_ID_STP = 0x4,
> > >> +       PORT_FEATURE_ID_UAFU = 0x5,
> > >> +       PORT_FEATURE_ID_MAX = 0x6,
> > >> +};
> > >> +
> > >> +#define FME_FEATURE_NUM                        FME_FEATURE_ID_MAX
> > >> +#define PORT_FEATURE_NUM               PORT_FEATURE_ID_MAX
> > >> +
> > >> +#define FPGA_FEATURE_DEV_FME           "fpga-dfl-fme"
> > >> +#define FPGA_FEATURE_DEV_PORT          "fpga-dfl-port"
> > >> +
> > >> +static inline int feature_platform_data_size(const int num)
> > >> +{
> > >> +       return sizeof(struct feature_platform_data) +
> > >> +               num * sizeof(struct feature);
> > >> +}
> > >> +
> > >> +int fpga_port_id(struct platform_device *pdev);
> > >> +
> > >> +static inline int fpga_port_check_id(struct platform_device *pdev,
> > >> +                                    void *pport_id)
> > >> +{
> > >> +       return fpga_port_id(pdev) == *(int *)pport_id;
> > >> +}
> > >> +
> > >> +void __fpga_port_enable(struct platform_device *pdev);
> > >> +int __fpga_port_disable(struct platform_device *pdev);
> > >> +
> > >> +static inline void fpga_port_enable(struct platform_device *pdev)
> > >> +{
> > >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > >> +
> > >> +       mutex_lock(&pdata->lock);
> > >> +       __fpga_port_enable(pdev);
> > >> +       mutex_unlock(&pdata->lock);
> > >> +}
> > >> +
> > >> +static inline int fpga_port_disable(struct platform_device *pdev)
> > >> +{
> > >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > >> +       int ret;
> > >> +
> > >> +       mutex_lock(&pdata->lock);
> > >> +       ret = __fpga_port_disable(pdev);
> > >> +       mutex_unlock(&pdata->lock);
> > >> +
> > >> +       return ret;
> > >> +}
> > >> +
> > >> +static inline int __fpga_port_reset(struct platform_device *pdev)
> > >> +{
> > >> +       int ret;
> > >> +
> > >> +       ret = __fpga_port_disable(pdev);
> > >> +       if (ret)
> > >> +               return ret;
> > >> +
> > >> +       __fpga_port_enable(pdev);
> > >> +
> > >> +       return 0;
> > >> +}
> > >> +
> > >> +static inline int fpga_port_reset(struct platform_device *pdev)
> > >> +{
> > >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > >> +       int ret;
> > >> +
> > >> +       mutex_lock(&pdata->lock);
> > >> +       ret = __fpga_port_reset(pdev);
> > >> +       mutex_unlock(&pdata->lock);
> > >> +
> > >> +       return ret;
> > >> +}
> > >
> > > I see that the port code is included as part of the enumeration code.
> > > This is not very future-proofed, if a different port needs to be
> > > supported.
> > >
> > > The port is a FPGA fabric based bridge with expanded functionality,
> > > right?  So it is similar to the altera freeze bridge, but adds the
> > > ability to reset the fabric and some other features are promised in
> > > the future, IIUC.  I still think that the port could be implemented in
> > > the bridge driver .c file instead of being here as part of the
> > > enumeration code.   For that to happen, some APIs would need to be
> > > added to the bridge framework and the FPGA region framework.  Then the
> > > reset can be requested through a new FPGA region API function.
> > >
> > > The advantage of this is that if this patchset evolves and there is
> > > some other v2 port driver needed, it can be a different driver if it
> > > needs to be.
> > >
> > > If the port reset is really a fabric reset,
> > 
> > Actually 'fabric reset' is probably not clear enough.  It's resetting
> > the hardware in a partial reconfiguration region, not just resetting
> > the bridge.  I'm trying to come up with a term that makes that clear
> > what is getting reset is the contents of the region.
> > 
> > > (correct me if I'm
> > > remembering wrongly) then it would be helpful to call it a
> > > fabric_reset.  This would be the first bridge driver supporting fabric
> > > reset.  I think it won't be the last.
> > >
> > > So what I'm proposing would be added/changed would be:
> > > * move all the bridge code to fpga-dfl-fme-br.c
> > > * add .fabric_reset to bridge ops
> > > * add fpga_bridges_reset to fpga-bridge.c (a new function that goes
> > > through a list of bridges and calls the reset ops if it exists,
> > > ignores the bridges where it doesn't exist)
> > > * add fpga_region_fabric_reset to fpga-region.c.  This function gets
> > > the region, gets the bridges, calls fpga_bridges_reset (can steal code
> > > from fpga_region_program_fpga)
> > > * the rest of the patchset can use fpga_region_fabric_reset instead of
> > > fpga_port_reset
> 
> Hi Alan
> 
> Actually I think we can't move all the bridge code to fpga-dfl-fme-br.c as
> this bridge (and region) is created by FME PR sub feature code, mainly for
> PR function. But user may need the reset function when run some workload
> on target Port/AFU, if consider virtualization case (SRIOV), there is only
> Port/AFU in each VF, and no FME in VF (that means nobody creates the fpga
> region/bridge/region). So it's need from port platform driver side as well.
> 
> The orignal idea that creates fpga-mgr/bridges/regions under FME, is that
> even we turned all Ports/AFUs into VFs (user can not see port platform
> device and the user interfaces exposed by port driver on PF), but user
> still can use FME to do PR to those Ports/AFUs in turned into VFs (assigned
> in different virtual machines).
> 
> I fully agree with you, that we should avoid feature specific code in the
> common enumeration code and feature device framework if possible. I guess
> I need some time to check and see if any other solutions (e.g export those
> functions from port driver not DFL framework). Will back here once I have
> some clear idea.:)

Hi Alan

I checked further on this, it seems no good method to avoid feature_dev
specific code (e.g port/fme related code) in DFL framework, as it needs to
manage feature devices for virtualization cases. I tried that, make some
changes that the port reset code could be exported by the port platform
device instead, and fpga-dfl-fme-br.c depends on port platform device to
implement the bridge ops, but 1) it introduced more dependency between
these driver modules which seems not good. (ideally it's better that PR
could be done by FME module itself, no need to have some dependency on
other modules, e.g Port). 2) still have other port code (e.g fpga_port_id
which is useful for port management code in framework) can't be moved to
port platform driver module in the same method. As hardware is designed
this way, even we see separated device features in the DFL, but they have
a lot of dependency internally in different use cases (e.g PR, SRIOV and
etc).

Thanks
Hao

> 
> Thanks
> Hao
> 
> > >
> > > Alan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 04/21] fpga: add device feature list support

[Alan Tull]

On Fri, Dec 22, 2017 at 2:45 AM, Wu Hao <hao.wu@intel.com> wrote:

>> > >
>> > > I see that the port code is included as part of the enumeration code.
>> > > This is not very future-proofed, if a different port needs to be
>> > > supported.
>> > >
>> > > The port is a FPGA fabric based bridge with expanded functionality,
>> > > right?  So it is similar to the altera freeze bridge, but adds the
>> > > ability to reset the fabric and some other features are promised in
>> > > the future, IIUC.  I still think that the port could be implemented in
>> > > the bridge driver .c file instead of being here as part of the
>> > > enumeration code.   For that to happen, some APIs would need to be
>> > > added to the bridge framework and the FPGA region framework.  Then the
>> > > reset can be requested through a new FPGA region API function.
>> > >
>> > > The advantage of this is that if this patchset evolves and there is
>> > > some other v2 port driver needed, it can be a different driver if it
>> > > needs to be.
>> > >
>> > > If the port reset is really a fabric reset,
>> >
>> > Actually 'fabric reset' is probably not clear enough.  It's resetting
>> > the hardware in a partial reconfiguration region, not just resetting
>> > the bridge.  I'm trying to come up with a term that makes that clear
>> > what is getting reset is the contents of the region.
>> >
>> > > (correct me if I'm
>> > > remembering wrongly) then it would be helpful to call it a
>> > > fabric_reset.  This would be the first bridge driver supporting fabric
>> > > reset.  I think it won't be the last.
>> > >
>> > > So what I'm proposing would be added/changed would be:
>> > > * move all the bridge code to fpga-dfl-fme-br.c
>> > > * add .fabric_reset to bridge ops
>> > > * add fpga_bridges_reset to fpga-bridge.c (a new function that goes
>> > > through a list of bridges and calls the reset ops if it exists,
>> > > ignores the bridges where it doesn't exist)
>> > > * add fpga_region_fabric_reset to fpga-region.c.  This function gets
>> > > the region, gets the bridges, calls fpga_bridges_reset (can steal code
>> > > from fpga_region_program_fpga)
>> > > * the rest of the patchset can use fpga_region_fabric_reset instead of
>> > > fpga_port_reset
>>
>> Hi Alan
>>
>> Actually I think we can't move all the bridge code to fpga-dfl-fme-br.c as
>> this bridge (and region) is created by FME PR sub feature code, mainly for
>> PR function. But user may need the reset function when run some workload
>> on target Port/AFU, if consider virtualization case (SRIOV), there is only
>> Port/AFU in each VF, and no FME in VF (that means nobody creates the fpga
>> region/bridge/region). So it's need from port platform driver side as well.
>>
>> The orignal idea that creates fpga-mgr/bridges/regions under FME, is that
>> even we turned all Ports/AFUs into VFs (user can not see port platform
>> device and the user interfaces exposed by port driver on PF), but user
>> still can use FME to do PR to those Ports/AFUs in turned into VFs (assigned
>> in different virtual machines).
>>
>> I fully agree with you, that we should avoid feature specific code in the
>> common enumeration code and feature device framework if possible. I guess
>> I need some time to check and see if any other solutions (e.g export those
>> functions from port driver not DFL framework). Will back here once I have
>> some clear idea.:)
>
> Hi Alan
>
> I checked further on this, it seems no good method to avoid feature_dev
> specific code (e.g port/fme related code) in DFL framework, as it needs to
> manage feature devices for virtualization cases. I tried that, make some
> changes that the port reset code could be exported by the port platform
> device instead, and fpga-dfl-fme-br.c depends on port platform device to
> implement the bridge ops, but 1) it introduced more dependency between
> these driver modules which seems not good. (ideally it's better that PR
> could be done by FME module itself, no need to have some dependency on
> other modules, e.g Port). 2) still have other port code (e.g fpga_port_id
> which is useful for port management code in framework) can't be moved to
> port platform driver module in the same method. As hardware is designed
> this way, even we see separated device features in the DFL, but they have
> a lot of dependency internally in different use cases (e.g PR, SRIOV and
> etc).

Hi Hao,

OK, well sounds like it's not feasible then.  Thanks for looking into it.

Alan

>
> Thanks
> Hao
>
>>
>> Thanks
>> Hao
>>
>> > >
>> > > Alan
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 05/21] fpga: dfl: add chardev support for feature devices

[Wu Hao]

For feature devices drivers, both the FPGA Management Engine (FME) and
Accelerated Function Unit (AFU) driver need to expose user interfaces via
the device file, for example, mmap and ioctls.

This patch adds chardev support in the fpga-dfl driver for feature devices,
FME and AFU. It reserves the chardev regions for FME and AFU, and provide
interfaces for FME and AFU driver to register their device file operations.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v2: rebased
v3: move chardev support to fpga-dfl framework
---
 drivers/fpga/fpga-dfl.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++-
 drivers/fpga/fpga-dfl.h | 13 +++++++
 2 files changed, 103 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/fpga-dfl.c b/drivers/fpga/fpga-dfl.c
index 6609828..9294c0a 100644
--- a/drivers/fpga/fpga-dfl.c
+++ b/drivers/fpga/fpga-dfl.c
@@ -76,6 +76,83 @@ static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
 	return FPGA_ID_MAX;
 }
 
+struct fpga_chardev_info {
+	const char *name;
+	dev_t devt;
+};
+
+/* indexed by enum fpga_devt_type */
+struct fpga_chardev_info fpga_chrdevs[] = {
+	{.name = FPGA_FEATURE_DEV_FME},		/* FPGA_DEVT_FME */
+	{.name = FPGA_FEATURE_DEV_PORT},	/* FPGA_DEVT_AFU */
+};
+
+static void fpga_chardev_uinit(void)
+{
+	int i;
+
+	for (i = 0; i < FPGA_DEVT_MAX; i++)
+		if (MAJOR(fpga_chrdevs[i].devt)) {
+			unregister_chrdev_region(fpga_chrdevs[i].devt,
+						 MINORMASK);
+			fpga_chrdevs[i].devt = MKDEV(0, 0);
+		}
+}
+
+static int fpga_chardev_init(void)
+{
+	int i, ret;
+
+	for (i = 0; i < FPGA_DEVT_MAX; i++) {
+		ret = alloc_chrdev_region(&fpga_chrdevs[i].devt, 0, MINORMASK,
+					  fpga_chrdevs[i].name);
+		if (ret)
+			goto exit;
+	}
+
+	return 0;
+
+exit:
+	fpga_chardev_uinit();
+	return ret;
+}
+
+static dev_t fpga_get_devt(enum fpga_devt_type type, int id)
+{
+	WARN_ON(type >= FPGA_DEVT_MAX);
+
+	return MKDEV(MAJOR(fpga_chrdevs[type].devt), id);
+}
+
+int fpga_register_dev_ops(struct platform_device *pdev,
+			  const struct file_operations *fops,
+			  struct module *owner)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	cdev_init(&pdata->cdev, fops);
+	pdata->cdev.owner = owner;
+
+	/*
+	 * set parent to the feature device so that its refcount is
+	 * decreased after the last refcount of cdev is gone, that
+	 * makes sure the feature device is valid during device
+	 * file's life-cycle.
+	 */
+	pdata->cdev.kobj.parent = &pdev->dev.kobj;
+
+	return cdev_add(&pdata->cdev, pdev->dev.devt, 1);
+}
+EXPORT_SYMBOL_GPL(fpga_register_dev_ops);
+
+void fpga_unregister_dev_ops(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	cdev_del(&pdata->cdev);
+}
+EXPORT_SYMBOL_GPL(fpga_unregister_dev_ops);
+
 /**
  * build_feature_devs_info - info collected during feature dev build.
  *
@@ -139,8 +216,12 @@ static int build_info_commit_dev(struct build_feature_devs_info *binfo)
 	struct platform_device *fdev;
 	struct resource *res;
 	struct feature_platform_data *pdata;
+	enum fpga_devt_type devt_type = FPGA_DEVT_FME;
 	int ret;
 
+	if (type == PORT_ID)
+		devt_type = FPGA_DEVT_PORT;
+
 	/* we will create a new device, commit current device first */
 	ret = build_info_commit_dev(binfo);
 	if (ret)
@@ -161,6 +242,7 @@ static int build_info_commit_dev(struct build_feature_devs_info *binfo)
 		return fdev->id;
 
 	fdev->dev.parent = &binfo->cdev->region.dev;
+	fdev->dev.devt = fpga_get_devt(devt_type, fdev->id);
 
 	/*
 	 * we do not need to care for the memory which is associated with
@@ -866,13 +948,20 @@ int __fpga_port_disable(struct platform_device *pdev)
 
 static int __init dfl_fpga_init(void)
 {
+	int ret;
+
 	fpga_ids_init();
 
-	return 0;
+	ret = fpga_chardev_init();
+	if (ret)
+		fpga_ids_destroy();
+
+	return ret;
 }
 
 static void __exit dfl_fpga_exit(void)
 {
+	fpga_chardev_uinit();
 	fpga_ids_destroy();
 }
 
diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
index abcbe57..e569a13 100644
--- a/drivers/fpga/fpga-dfl.h
+++ b/drivers/fpga/fpga-dfl.h
@@ -17,6 +17,7 @@
 #define __DFL_FPGA_H
 
 #include <linux/bitfield.h>
+#include <linux/cdev.h>
 #include <linux/delay.h>
 #include <linux/fs.h>
 #include <linux/iopoll.h>
@@ -185,6 +186,7 @@ struct feature_platform_data {
 	/* list the feature dev to cci_drvdata->port_dev_list. */
 	struct list_head node;
 	struct mutex lock;		/* protect platform data */
+	struct cdev cdev;
 	struct platform_device *dev;
 	unsigned int disable_count;	/* count for port disable */
 
@@ -226,6 +228,17 @@ static inline int feature_platform_data_size(const int num)
 		num * sizeof(struct feature);
 }
 
+enum fpga_devt_type {
+	FPGA_DEVT_FME,
+	FPGA_DEVT_PORT,
+	FPGA_DEVT_MAX,
+};
+
+int fpga_register_dev_ops(struct platform_device *pdev,
+			  const struct file_operations *fops,
+			  struct module *owner);
+void fpga_unregister_dev_ops(struct platform_device *pdev);
+
 int fpga_port_id(struct platform_device *pdev);
 
 static inline int fpga_port_check_id(struct platform_device *pdev,
-- 
1.8.3.1

[PATCH v3 06/21] fpga: dfl: adds fpga_cdev_find_port

[Wu Hao]

For feature devices, e.g FPGA Management Engine (FME), it may
require fpga_cdev_find_port function to find dedicate port for
further actions, so export this function from feature device
driver module.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: s/fpga_for_each_port/fpga_cdev_find_port/
    move fpga_cdev_find_port to fpga-dfl module.
---
 drivers/fpga/fpga-dfl.c | 27 +++++++++++++++++++++++++++
 drivers/fpga/fpga-dfl.h | 16 ++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/drivers/fpga/fpga-dfl.c b/drivers/fpga/fpga-dfl.c
index 9294c0a..ce03b17 100644
--- a/drivers/fpga/fpga-dfl.c
+++ b/drivers/fpga/fpga-dfl.c
@@ -873,6 +873,33 @@ void fpga_remove_feature_devs(struct fpga_cdev *cdev)
 }
 EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);
 
+/**
+ * __fpga_cdev_find_port - find a port under given container device
+ * @cdev: container device
+ * @data: data passed to match function
+ * @match: match function used to find specific port from the port device list
+ *
+ * Find a port device under container device. This function needs to be
+ * invoked with lock held.
+ */
+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+		      int (*match)(struct platform_device *, void *))
+{
+	struct feature_platform_data *pdata;
+	struct platform_device *port_dev;
+
+	list_for_each_entry(pdata, &cdev->port_dev_list, node) {
+		port_dev = pdata->dev;
+
+		if (match(port_dev, data) && get_device(&port_dev->dev))
+			return port_dev;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__fpga_cdev_find_port);
+
 int fpga_port_id(struct platform_device *pdev)
 {
 	void __iomem *base;
diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
index e569a13..36e2394 100644
--- a/drivers/fpga/fpga-dfl.h
+++ b/drivers/fpga/fpga-dfl.h
@@ -375,4 +375,20 @@ struct fpga_cdev {
 struct fpga_cdev *fpga_enumerate_feature_devs(struct fpga_enum_info *info);
 void fpga_remove_feature_devs(struct fpga_cdev *cdev);
 
+struct platform_device *
+__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+		      int (*match)(struct platform_device *, void *));
+
+static inline struct platform_device *
+fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
+		    int (*match)(struct platform_device *, void *))
+{
+	struct platform_device *pdev;
+
+	mutex_lock(&cdev->lock);
+	pdev = __fpga_cdev_find_port(cdev, data, match);
+	mutex_unlock(&cdev->lock);
+
+	return pdev;
+}
 #endif /* __DFL_FPGA_H */
-- 
1.8.3.1

Re: [PATCH v3 06/21] fpga: dfl: adds fpga_cdev_find_port

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

A couple comments below.

> For feature devices, e.g FPGA Management Engine (FME),

The first use I see of this function in this patchset is the FME PR,
not the FME, or am I misunderstanding?

> it may
> require fpga_cdev_find_port function to find dedicate port for
> further actions, so export this function from feature device
> driver module.

I understand the need for this function, but I'm having a hard time
following this explanation here.   Suggestion: "For feature devices,
we need a method to find the port dedicated to the device.  Add a
function fpga_cdev_find_port to do that"

>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v3: s/fpga_for_each_port/fpga_cdev_find_port/
>     move fpga_cdev_find_port to fpga-dfl module.
> ---
>  drivers/fpga/fpga-dfl.c | 27 +++++++++++++++++++++++++++
>  drivers/fpga/fpga-dfl.h | 16 ++++++++++++++++
>  2 files changed, 43 insertions(+)
>
> diff --git a/drivers/fpga/fpga-dfl.c b/drivers/fpga/fpga-dfl.c
> index 9294c0a..ce03b17 100644
> --- a/drivers/fpga/fpga-dfl.c
> +++ b/drivers/fpga/fpga-dfl.c
> @@ -873,6 +873,33 @@ void fpga_remove_feature_devs(struct fpga_cdev *cdev)
>  }
>  EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);
>
> +/**
> + * __fpga_cdev_find_port - find a port under given container device
> + * @cdev: container device
> + * @data: data passed to match function
> + * @match: match function used to find specific port from the port device list
> + *
> + * Find a port device under container device. This function needs to be
> + * invoked with lock held.
> + */
> +struct platform_device *
> +__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
> +                     int (*match)(struct platform_device *, void *))
> +{
> +       struct feature_platform_data *pdata;
> +       struct platform_device *port_dev;
> +
> +       list_for_each_entry(pdata, &cdev->port_dev_list, node) {
> +               port_dev = pdata->dev;
> +
> +               if (match(port_dev, data) && get_device(&port_dev->dev))
> +                       return port_dev;

The caller will need to put the port_dev->dev.  Please document that
in the function header.  I mentioned that in v2.

Alan

Re: [PATCH v3 06/21] fpga: dfl: adds fpga_cdev_find_port

[Wu Hao]

On Mon, Feb 05, 2018 at 04:08:19PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> A couple comments below.
> 
> > For feature devices, e.g FPGA Management Engine (FME),
> 
> The first use I see of this function in this patchset is the FME PR,
> not the FME, or am I misunderstanding?

Hi Alan

Thanks for the review.

Yes, the first use is in the FME PR, the description should be more
clear here. I will improve it.

> 
> > it may
> > require fpga_cdev_find_port function to find dedicate port for
> > further actions, so export this function from feature device
> > driver module.
> 
> I understand the need for this function, but I'm having a hard time
> following this explanation here.   Suggestion: "For feature devices,
> we need a method to find the port dedicated to the device.  Add a
> function fpga_cdev_find_port to do that"

Will fix it, thanks for the suggestion.

> 
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ----
> > v3: s/fpga_for_each_port/fpga_cdev_find_port/
> >     move fpga_cdev_find_port to fpga-dfl module.
> > ---
> >  drivers/fpga/fpga-dfl.c | 27 +++++++++++++++++++++++++++
> >  drivers/fpga/fpga-dfl.h | 16 ++++++++++++++++
> >  2 files changed, 43 insertions(+)
> >
> > diff --git a/drivers/fpga/fpga-dfl.c b/drivers/fpga/fpga-dfl.c
> > index 9294c0a..ce03b17 100644
> > --- a/drivers/fpga/fpga-dfl.c
> > +++ b/drivers/fpga/fpga-dfl.c
> > @@ -873,6 +873,33 @@ void fpga_remove_feature_devs(struct fpga_cdev *cdev)
> >  }
> >  EXPORT_SYMBOL_GPL(fpga_remove_feature_devs);
> >
> > +/**
> > + * __fpga_cdev_find_port - find a port under given container device
> > + * @cdev: container device
> > + * @data: data passed to match function
> > + * @match: match function used to find specific port from the port device list
> > + *
> > + * Find a port device under container device. This function needs to be
> > + * invoked with lock held.
> > + */
> > +struct platform_device *
> > +__fpga_cdev_find_port(struct fpga_cdev *cdev, void *data,
> > +                     int (*match)(struct platform_device *, void *))
> > +{
> > +       struct feature_platform_data *pdata;
> > +       struct platform_device *port_dev;
> > +
> > +       list_for_each_entry(pdata, &cdev->port_dev_list, node) {
> > +               port_dev = pdata->dev;
> > +
> > +               if (match(port_dev, data) && get_device(&port_dev->dev))
> > +                       return port_dev;
> 
> The caller will need to put the port_dev->dev.  Please document that
> in the function header.  I mentioned that in v2.

Sorry, I missed it, will add description on this function header.

Thanks again for the review and comments.

Hao

[PATCH v3 07/21] fpga: dfl: add feature device infrastructure

[Wu Hao]

From: Xiao Guangrong <guangrong.xiao@linux.intel.com>

This patch abstracts the common operations of the sub features, and defines
the feature_ops data structure, including init, uinit and ioctl function
pointers. And this patch adds some common helper functions for FME and AFU
drivers, e.g feature_dev_use_begin/end which are used to ensure exclusive
usage of the feature device file.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v2: rebased
v3: use const for feature_ops.
    replace pci related function.
---
 drivers/fpga/fpga-dfl.c | 68 +++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/fpga-dfl.h | 80 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 148 insertions(+)

diff --git a/drivers/fpga/fpga-dfl.c b/drivers/fpga/fpga-dfl.c
index ce03b17..1714803 100644
--- a/drivers/fpga/fpga-dfl.c
+++ b/drivers/fpga/fpga-dfl.c
@@ -76,6 +76,74 @@ static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
 	return FPGA_ID_MAX;
 }
 
+int fme_feature_to_resource_index(int feature_id)
+{
+	WARN_ON(feature_id >= FME_FEATURE_ID_MAX);
+	return feature_id;
+}
+
+void fpga_dev_feature_uinit(struct platform_device *pdev)
+{
+	struct feature *feature;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	fpga_dev_for_each_feature(pdata, feature)
+		if (feature->ops) {
+			feature->ops->uinit(pdev, feature);
+			feature->ops = NULL;
+		}
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_uinit);
+
+static int
+feature_instance_init(struct platform_device *pdev,
+		      struct feature_platform_data *pdata,
+		      struct feature *feature, struct feature_driver *drv)
+{
+	int ret;
+
+	WARN_ON(!feature->ioaddr);
+
+	ret = drv->ops->init(pdev, feature);
+	if (ret)
+		return ret;
+
+	feature->ops = drv->ops;
+
+	return ret;
+}
+
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs)
+{
+	struct feature *feature;
+	struct feature_driver *drv = feature_drvs;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	while (drv->ops) {
+		fpga_dev_for_each_feature(pdata, feature) {
+			/* skip the feature which is not initialized. */
+			if (!feature->name)
+				continue;
+
+			if (!strcmp(drv->name, feature->name)) {
+				ret = feature_instance_init(pdev, pdata,
+							    feature, drv);
+				if (ret)
+					goto exit;
+			}
+		}
+		drv++;
+	}
+
+	return 0;
+exit:
+	fpga_dev_feature_uinit(pdev);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_init);
+
 struct fpga_chardev_info {
 	const char *name;
 	dev_t devt;
diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
index 36e2394..e5a1094 100644
--- a/drivers/fpga/fpga-dfl.h
+++ b/drivers/fpga/fpga-dfl.h
@@ -176,12 +176,20 @@
 #define PORT_UAFU_DFH		DFH
 #define PORT_UAFU_SIZE		DFH_SIZE
 
+struct feature_driver {
+	const char *name;
+	const struct feature_ops *ops;
+};
+
 struct feature {
 	const char *name;
 	int resource_index;
 	void __iomem *ioaddr;
+	const struct feature_ops *ops;
 };
 
+#define DEV_STATUS_IN_USE	0
+
 struct feature_platform_data {
 	/* list the feature dev to cci_drvdata->port_dev_list. */
 	struct list_head node;
@@ -189,11 +197,46 @@ struct feature_platform_data {
 	struct cdev cdev;
 	struct platform_device *dev;
 	unsigned int disable_count;	/* count for port disable */
+	unsigned long dev_status;
+
+	void *private;			/* ptr to feature dev private data */
 
 	int num;			/* number of features */
 	struct feature features[0];
 };
 
+static inline int feature_dev_use_begin(struct feature_platform_data *pdata)
+{
+	/* Test and set IN_USE flags to ensure file is exclusively used */
+	if (test_and_set_bit_lock(DEV_STATUS_IN_USE, &pdata->dev_status))
+		return -EBUSY;
+
+	return 0;
+}
+
+static inline void feature_dev_use_end(struct feature_platform_data *pdata)
+{
+	clear_bit_unlock(DEV_STATUS_IN_USE, &pdata->dev_status);
+}
+
+static inline void
+fpga_pdata_set_private(struct feature_platform_data *pdata, void *private)
+{
+	pdata->private = private;
+}
+
+static inline void *fpga_pdata_get_private(struct feature_platform_data *pdata)
+{
+	return pdata->private;
+}
+
+struct feature_ops {
+	int (*init)(struct platform_device *pdev, struct feature *feature);
+	void (*uinit)(struct platform_device *pdev, struct feature *feature);
+	long (*ioctl)(struct platform_device *pdev, struct feature *feature,
+		      unsigned int cmd, unsigned long arg);
+};
+
 enum fme_feature_id {
 	FME_FEATURE_ID_HEADER = 0x0,
 	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
@@ -228,6 +271,10 @@ static inline int feature_platform_data_size(const int num)
 		num * sizeof(struct feature);
 }
 
+void fpga_dev_feature_uinit(struct platform_device *pdev);
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs);
+
 enum fpga_devt_type {
 	FPGA_DEVT_FME,
 	FPGA_DEVT_PORT,
@@ -296,6 +343,15 @@ static inline int fpga_port_reset(struct platform_device *pdev)
 	return ret;
 }
 
+static inline
+struct platform_device *fpga_inode_to_feature_dev(struct inode *inode)
+{
+	struct feature_platform_data *pdata;
+
+	pdata = container_of(inode->i_cdev, struct feature_platform_data, cdev);
+	return pdata->dev;
+}
+
 static inline void __iomem *
 get_feature_ioaddr_by_index(struct device *dev, int index)
 {
@@ -304,6 +360,21 @@ static inline int fpga_port_reset(struct platform_device *pdev)
 	return pdata->features[index].ioaddr;
 }
 
+static inline bool is_feature_present(struct device *dev, int index)
+{
+	return !!get_feature_ioaddr_by_index(dev, index);
+}
+
+static inline struct device *
+fpga_pdata_to_parent(struct feature_platform_data *pdata)
+{
+	return pdata->dev->dev.parent->parent;
+}
+
+#define fpga_dev_for_each_feature(pdata, feature)			    \
+	for ((feature) = (pdata)->features;				    \
+	   (feature) < (pdata)->features + (pdata)->num; (feature)++)
+
 static inline bool feature_is_fme(void __iomem *base)
 {
 	u64 v = readq(base + DFH);
@@ -391,4 +462,13 @@ struct platform_device *
 
 	return pdev;
 }
+
+static inline struct fpga_cdev *
+fpga_pdata_to_fpga_cdev(struct feature_platform_data *pdata)
+{
+	struct device *dev = pdata->dev->dev.parent;
+	struct fpga_region *region = to_fpga_region(dev);
+
+	return container_of(region, struct fpga_cdev, region);
+}
 #endif /* __DFL_FPGA_H */
-- 
1.8.3.1

[PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Wu Hao]

From: Zhang Yi <yi.z.zhang@intel.com>

The Intel FPGA device appears as a PCIe device on the system. This patch
implements the basic framework of the driver for Intel PCIe device which
is located between CPU and Accelerated Function Units (AFUs), and has
the Device Feature List (DFL) implemented in its MMIO space.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v2: move the code to drivers/fpga folder as suggested by Alan Tull.
    switch to GPLv2 license.
    fix comments from Moritz Fischer.
v3: switch to pci_set_dma_mask/consistent_dma_mask() function.
    remove pci_save_state() in probe function.
    rename driver to INTEL_FPGA_DFL_PCI and intel-dfl-pci.c to indicate
    this driver supports Intel FPGA PCI devices which implement DFL.
    improve Kconfig description for INTEL_FPGA_DFL_PCI
---
 drivers/fpga/Kconfig         |  18 ++++++
 drivers/fpga/Makefile        |   3 +
 drivers/fpga/intel-dfl-pci.c | 129 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 150 insertions(+)
 create mode 100644 drivers/fpga/intel-dfl-pci.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 01ad31f..cc35d12 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -140,4 +140,22 @@ config FPGA_DFL
 	  Gate Array (FPGA) solutions which implement Device Feature List.
 	  It provides enumeration APIs, and feature device infrastructure.
 
+config INTEL_FPGA_DFL_PCI
+	tristate "Intel FPGA DFL PCIe Device Driver"
+	depends on PCI && FPGA_DFL
+	help
+	  Select this option to enable PCIe driver for Intel(R) PCIe based
+	  Field-Programmable Gate Array (FPGA) solutions which implemented
+	  the Device Feature List (DFL). It supports both integrated (e.g
+	  Intel Server Platform with In-package FPGA) and discrete (e.g
+	  Intel FPGA PCIe Acceleration Cards) solutions. This driver
+	  provides interfaces for userspace applications to configure,
+	  enumerate, open and access FPGA accelerators on platforms
+	  equipped with Intel(R) FPGA solutions and enables system level
+	  management functions such as FPGA partial reconfiguration, power
+	  management, and virtualization via DFL framework and DFL feature
+	  device drivers.
+
+	  To compile this as a module, choose M here.
+
 endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 447ba2b..d39a431 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -30,3 +30,6 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)			+= fpga-dfl.o
+
+# Drivers for FPGAs which implement DFL
+obj-$(CONFIG_INTEL_FPGA_DFL_PCI)	+= intel-dfl-pci.o
diff --git a/drivers/fpga/intel-dfl-pci.c b/drivers/fpga/intel-dfl-pci.c
new file mode 100644
index 0000000..4774a77
--- /dev/null
+++ b/drivers/fpga/intel-dfl-pci.c
@@ -0,0 +1,129 @@
+/*
+ * Driver for Intel FPGA DFL PCIe device
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Zhang Yi <Yi.Z.Zhang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/pci.h>
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/stddef.h>
+#include <linux/errno.h>
+#include <linux/aer.h>
+
+#define DRV_VERSION	"0.8"
+#define DRV_NAME	"intel-dfl-pci"
+
+/* PCI Device ID */
+#define PCIE_DEVICE_ID_PF_INT_5_X	0xBCBD
+#define PCIE_DEVICE_ID_PF_INT_6_X	0xBCC0
+#define PCIE_DEVICE_ID_PF_DSC_1_X	0x09C4
+/* VF Device */
+#define PCIE_DEVICE_ID_VF_INT_5_X	0xBCBF
+#define PCIE_DEVICE_ID_VF_INT_6_X	0xBCC1
+#define PCIE_DEVICE_ID_VF_DSC_1_X	0x09C5
+
+static struct pci_device_id cci_pcie_id_tbl[] = {
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_PF_DSC_1_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIE_DEVICE_ID_VF_DSC_1_X),},
+	{0,}
+};
+MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
+
+static
+int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
+{
+	int ret;
+
+	ret = pci_enable_device(pcidev);
+	if (ret < 0) {
+		dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
+		return ret;
+	}
+
+	ret = pci_enable_pcie_error_reporting(pcidev);
+	if (ret && ret != -EINVAL)
+		dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
+
+	ret = pci_request_regions(pcidev, DRV_NAME);
+	if (ret) {
+		dev_err(&pcidev->dev, "Failed to request regions.\n");
+		goto disable_error_report_exit;
+	}
+
+	pci_set_master(pcidev);
+
+	if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(64))) {
+		ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(64));
+		if (ret)
+			goto release_region_exit;
+	} else if (!pci_set_dma_mask(pcidev, DMA_BIT_MASK(32))) {
+		ret = pci_set_consistent_dma_mask(pcidev, DMA_BIT_MASK(32));
+		if (ret)
+			goto release_region_exit;
+	} else {
+		ret = -EIO;
+		dev_err(&pcidev->dev, "No suitable DMA support available.\n");
+		goto release_region_exit;
+	}
+
+	/* TODO: create and add the platform device per feature list */
+	return 0;
+
+release_region_exit:
+	pci_release_regions(pcidev);
+disable_error_report_exit:
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+	return ret;
+}
+
+static void cci_pci_remove(struct pci_dev *pcidev)
+{
+	pci_release_regions(pcidev);
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+}
+
+static struct pci_driver cci_pci_driver = {
+	.name = DRV_NAME,
+	.id_table = cci_pcie_id_tbl,
+	.probe = cci_pci_probe,
+	.remove = cci_pci_remove,
+};
+
+static int __init ccidrv_init(void)
+{
+	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
+
+	return pci_register_driver(&cci_pci_driver);
+}
+
+static void __exit ccidrv_exit(void)
+{
+	pci_unregister_driver(&cci_pci_driver);
+}
+
+module_init(ccidrv_init);
+module_exit(ccidrv_exit);
+
+MODULE_DESCRIPTION("Intel FPGA DFL PCIe Device Driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
-- 
1.8.3.1

RE: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[David Laight]

From: Wu Hao
> Sent: 27 November 2017 06:42
> From: Zhang Yi <yi.z.zhang@intel.com>
> 
> The Intel FPGA device appears as a PCIe device on the system. This patch
> implements the basic framework of the driver for Intel PCIe device which
> is located between CPU and Accelerated Function Units (AFUs), and has
> the Device Feature List (DFL) implemented in its MMIO space.

This ought to have a better name than 'Intel FPGA'.
An fpga can be used for all sorts of things, this looks like
a very specific architecture using a common VHDL environment to
allow certain types of user VHDL be accessed over PCIe.

	David

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Wu Hao]

On Mon, Nov 27, 2017 at 10:28:04AM +0000, David Laight wrote:
> From: Wu Hao
> > Sent: 27 November 2017 06:42
> > From: Zhang Yi <yi.z.zhang@intel.com>
> > 
> > The Intel FPGA device appears as a PCIe device on the system. This patch
> > implements the basic framework of the driver for Intel PCIe device which
> > is located between CPU and Accelerated Function Units (AFUs), and has
> > the Device Feature List (DFL) implemented in its MMIO space.
> 
> This ought to have a better name than 'Intel FPGA'.
> An fpga can be used for all sorts of things, this looks like
> a very specific architecture using a common VHDL environment to
> allow certain types of user VHDL be accessed over PCIe.

Hi David

This patch adds a pcie device driver for Intel FPGA devices which implements
the DFL, e.g Intel Server Platform with In-package FPGA and Intel FPGA PCIe
Acceleration Cards. They are pcie devices, and all have DFL implemented in
the MMIO space, so we would like to use one kernel driver to handle them.

With this full patchset, it just provides user the interfaces to configure
and access the FPGA accelerators on Intel DFL based FPGA devices. For sure,
users can develop and build their own logics via tools provided by Intel,
program them to accelerators on these Intel FPGA devices, and access them
for their workloads.

Thanks
Hao

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Alan Tull]

On Mon, Nov 27, 2017 at 9:15 PM, Wu Hao <hao.wu@intel.com> wrote:
> On Mon, Nov 27, 2017 at 10:28:04AM +0000, David Laight wrote:
>> From: Wu Hao
>> > Sent: 27 November 2017 06:42
>> > From: Zhang Yi <yi.z.zhang@intel.com>
>> >
>> > The Intel FPGA device appears as a PCIe device on the system. This patch
>> > implements the basic framework of the driver for Intel PCIe device which
>> > is located between CPU and Accelerated Function Units (AFUs), and has
>> > the Device Feature List (DFL) implemented in its MMIO space.
>>
>> This ought to have a better name than 'Intel FPGA'.
>> An fpga can be used for all sorts of things, this looks like
>> a very specific architecture using a common VHDL environment to
>> allow certain types of user VHDL be accessed over PCIe.
>
> Hi David
>
> This patch adds a pcie device driver for Intel FPGA devices which implements
> the DFL, e.g Intel Server Platform with In-package FPGA and Intel FPGA PCIe
> Acceleration Cards. They are pcie devices, and all have DFL implemented in
> the MMIO space, so we would like to use one kernel driver to handle them.
>
> With this full patchset, it just provides user the interfaces to configure
> and access the FPGA accelerators on Intel DFL based FPGA devices. For sure,
> users can develop and build their own logics via tools provided by Intel,
> program them to accelerators on these Intel FPGA devices, and access them
> for their workloads.

I don't see anything Intel specific here.  This could all be named dfl-*

Alan

>
> Thanks
> Hao

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Wu Hao]

On Mon, Dec 04, 2017 at 01:46:59PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 9:15 PM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Nov 27, 2017 at 10:28:04AM +0000, David Laight wrote:
> >> From: Wu Hao
> >> > Sent: 27 November 2017 06:42
> >> > From: Zhang Yi <yi.z.zhang@intel.com>
> >> >
> >> > The Intel FPGA device appears as a PCIe device on the system. This patch
> >> > implements the basic framework of the driver for Intel PCIe device which
> >> > is located between CPU and Accelerated Function Units (AFUs), and has
> >> > the Device Feature List (DFL) implemented in its MMIO space.
> >>
> >> This ought to have a better name than 'Intel FPGA'.
> >> An fpga can be used for all sorts of things, this looks like
> >> a very specific architecture using a common VHDL environment to
> >> allow certain types of user VHDL be accessed over PCIe.
> >
> > Hi David
> >
> > This patch adds a pcie device driver for Intel FPGA devices which implements
> > the DFL, e.g Intel Server Platform with In-package FPGA and Intel FPGA PCIe
> > Acceleration Cards. They are pcie devices, and all have DFL implemented in
> > the MMIO space, so we would like to use one kernel driver to handle them.
> >
> > With this full patchset, it just provides user the interfaces to configure
> > and access the FPGA accelerators on Intel DFL based FPGA devices. For sure,
> > users can develop and build their own logics via tools provided by Intel,
> > program them to accelerators on these Intel FPGA devices, and access them
> > for their workloads.
> 
> I don't see anything Intel specific here.  This could all be named dfl-*

The maybe some device specific things, e.g Intel FPGA devices supported by this
driver always have FME DFL at the beginning on the BAR0 for PF device.

But I think this should be the right direction for better code reuse, it could
save efforts for other vendors who want to use DFL and follow the same way.

Thanks for the comments. I will rename this driver in the next version.

Hao

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Alan Tull]

On Mon, Dec 4, 2017 at 9:33 PM, Wu Hao <hao.wu@intel.com> wrote:
> On Mon, Dec 04, 2017 at 01:46:59PM -0600, Alan Tull wrote:
>> On Mon, Nov 27, 2017 at 9:15 PM, Wu Hao <hao.wu@intel.com> wrote:
>> > On Mon, Nov 27, 2017 at 10:28:04AM +0000, David Laight wrote:
>> >> From: Wu Hao
>> >> > Sent: 27 November 2017 06:42
>> >> > From: Zhang Yi <yi.z.zhang@intel.com>
>> >> >
>> >> > The Intel FPGA device appears as a PCIe device on the system. This patch
>> >> > implements the basic framework of the driver for Intel PCIe device which
>> >> > is located between CPU and Accelerated Function Units (AFUs), and has
>> >> > the Device Feature List (DFL) implemented in its MMIO space.
>> >>
>> >> This ought to have a better name than 'Intel FPGA'.
>> >> An fpga can be used for all sorts of things, this looks like
>> >> a very specific architecture using a common VHDL environment to
>> >> allow certain types of user VHDL be accessed over PCIe.
>> >
>> > Hi David
>> >
>> > This patch adds a pcie device driver for Intel FPGA devices which implements
>> > the DFL, e.g Intel Server Platform with In-package FPGA and Intel FPGA PCIe
>> > Acceleration Cards. They are pcie devices, and all have DFL implemented in
>> > the MMIO space, so we would like to use one kernel driver to handle them.
>> >
>> > With this full patchset, it just provides user the interfaces to configure
>> > and access the FPGA accelerators on Intel DFL based FPGA devices. For sure,
>> > users can develop and build their own logics via tools provided by Intel,
>> > program them to accelerators on these Intel FPGA devices, and access them
>> > for their workloads.
>>
>> I don't see anything Intel specific here.  This could all be named dfl-*
>
> The maybe some device specific things, e.g Intel FPGA devices supported by this
> driver always have FME DFL at the beginning on the BAR0 for PF device.
>
> But I think this should be the right direction for better code reuse, it could
> save efforts for other vendors who want to use DFL and follow the same way.
>
> Thanks for the comments. I will rename this driver in the next version.

Thanks!

Regarding file names, it seems like the files added to drivers/fpga
could be uniformly named dfl-*.[ch].  Some are fpga-dfl-*.[ch] while
other are currently dfl-*.[ch] currently.

Alan

>
> Hao

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Wu Hao]

On Tue, Dec 05, 2017 at 11:00:22AM -0600, Alan Tull wrote:
> On Mon, Dec 4, 2017 at 9:33 PM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Dec 04, 2017 at 01:46:59PM -0600, Alan Tull wrote:
> >> On Mon, Nov 27, 2017 at 9:15 PM, Wu Hao <hao.wu@intel.com> wrote:
> >> > On Mon, Nov 27, 2017 at 10:28:04AM +0000, David Laight wrote:
> >> >> From: Wu Hao
> >> >> > Sent: 27 November 2017 06:42
> >> >> > From: Zhang Yi <yi.z.zhang@intel.com>
> >> >> >
> >> >> > The Intel FPGA device appears as a PCIe device on the system. This patch
> >> >> > implements the basic framework of the driver for Intel PCIe device which
> >> >> > is located between CPU and Accelerated Function Units (AFUs), and has
> >> >> > the Device Feature List (DFL) implemented in its MMIO space.
> >> >>
> >> >> This ought to have a better name than 'Intel FPGA'.
> >> >> An fpga can be used for all sorts of things, this looks like
> >> >> a very specific architecture using a common VHDL environment to
> >> >> allow certain types of user VHDL be accessed over PCIe.
> >> >
> >> > Hi David
> >> >
> >> > This patch adds a pcie device driver for Intel FPGA devices which implements
> >> > the DFL, e.g Intel Server Platform with In-package FPGA and Intel FPGA PCIe
> >> > Acceleration Cards. They are pcie devices, and all have DFL implemented in
> >> > the MMIO space, so we would like to use one kernel driver to handle them.
> >> >
> >> > With this full patchset, it just provides user the interfaces to configure
> >> > and access the FPGA accelerators on Intel DFL based FPGA devices. For sure,
> >> > users can develop and build their own logics via tools provided by Intel,
> >> > program them to accelerators on these Intel FPGA devices, and access them
> >> > for their workloads.
> >>
> >> I don't see anything Intel specific here.  This could all be named dfl-*
> >
> > The maybe some device specific things, e.g Intel FPGA devices supported by this
> > driver always have FME DFL at the beginning on the BAR0 for PF device.
> >
> > But I think this should be the right direction for better code reuse, it could
> > save efforts for other vendors who want to use DFL and follow the same way.
> >
> > Thanks for the comments. I will rename this driver in the next version.
> 
> Thanks!
> 
> Regarding file names, it seems like the files added to drivers/fpga
> could be uniformly named dfl-*.[ch].  Some are fpga-dfl-*.[ch] while
> other are currently dfl-*.[ch] currently.

Sure, will have all related drivers files renamed to dfl-*.[ch].

Thanks
Hao

> 
> Alan
> 
> >
> > Hao

RE: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[David Laight]

From: Wu Hao
> Sent: 06 December 2017 05:30
...
> > Regarding file names, it seems like the files added to drivers/fpga
> > could be uniformly named dfl-*.[ch].  Some are fpga-dfl-*.[ch] while
> > other are currently dfl-*.[ch] currently.

They don't even want to do into a drivers/fgpa directory.
Maybe drivers/dfl or drivers/dfl/intel

	David

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Alan Tull]

On Wed, Dec 6, 2017 at 3:44 AM, David Laight <David.Laight@aculab.com> wrote:
> From: Wu Hao
>> Sent: 06 December 2017 05:30
> ...
>> > Regarding file names, it seems like the files added to drivers/fpga
>> > could be uniformly named dfl-*.[ch].  Some are fpga-dfl-*.[ch] while
>> > other are currently dfl-*.[ch] currently.
>
> They don't even want to do into a drivers/fgpa directory.
> Maybe drivers/dfl or drivers/dfl/intel

It's plugged into the fpga framework in drivers/fpga.  This patchset
also handles reprogramming the fpga, not just the dfl style
enumeration.  But your points about this being not just for FPGA are
interesting to me.  Do you have a use for this that isn't
FPGA-centric?

Alan

>
>         David
>

RE: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[David Laight]

From: Alan Tull
> Sent: 06 December 2017 15:30
> On Wed, Dec 6, 2017 at 3:44 AM, David Laight <David.Laight@aculab.com> wrote:
> > From: Wu Hao
> >> Sent: 06 December 2017 05:30
> > ...
> >> > Regarding file names, it seems like the files added to drivers/fpga
> >> > could be uniformly named dfl-*.[ch].  Some are fpga-dfl-*.[ch] while
> >> > other are currently dfl-*.[ch] currently.
> >
> > They don't even want to do into a drivers/fgpa directory.
> > Maybe drivers/dfl or drivers/dfl/intel
> 
> It's plugged into the fpga framework in drivers/fpga.  This patchset
> also handles reprogramming the fpga, not just the dfl style
> enumeration.  But your points about this being not just for FPGA are
> interesting to me.  Do you have a use for this that isn't
> FPGA-centric?

That all just seems wrong to me.
If you've managed to invent some common code for reprogramming fpga
I'd have though it would be library functions.

The driver ought to sit somewhere related to its functionality.

Our fpga loads from a serial EEPROM, the image is about 6.5MB.
We can rewrite it from userspace by mmap()ing part of one of the BARs
to access some very locally written (by me) VHDL that does most of
the required bit-banging for 32it word accesses.
You really wouldn't want to load 6.5MB into kernel space!

We also had to solve the problem of 9 separate driver modules that
want to access different parts of the BARs.
I think we have 46 separate slaves in the fpgas BARs (most are in 1 BAR).
Some of these are common between different boards (or completely different
memory maps for the same board.

I can imagine some generic method of having a 'board' driver for a
specific PCI-id that knows the BAR offsets of various functions so that
other sub-drivers could be loaded to access those functions.
But that is some kind of pseudo-bus not fpga specific in any way.

	David

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Alan Tull]

On Wed, Dec 6, 2017 at 10:28 AM, David Laight <David.Laight@aculab.com> wrote:
> From: Alan Tull
>> Sent: 06 December 2017 15:30
>> On Wed, Dec 6, 2017 at 3:44 AM, David Laight <David.Laight@aculab.com> wrote:
>> > From: Wu Hao
>> >> Sent: 06 December 2017 05:30
>> > ...
>> >> > Regarding file names, it seems like the files added to drivers/fpga
>> >> > could be uniformly named dfl-*.[ch].  Some are fpga-dfl-*.[ch] while
>> >> > other are currently dfl-*.[ch] currently.
>> >
>> > They don't even want to do into a drivers/fgpa directory.
>> > Maybe drivers/dfl or drivers/dfl/intel
>>
>> It's plugged into the fpga framework in drivers/fpga.  This patchset
>> also handles reprogramming the fpga, not just the dfl style
>> enumeration.  But your points about this being not just for FPGA are
>> interesting to me.  Do you have a use for this that isn't
>> FPGA-centric?
>
> That all just seems wrong to me.
> If you've managed to invent some common code for reprogramming fpga
> I'd have though it would be library functions.

Why don't you familiarize yourself with the fpga framework before
commenting on it? ;)

>
> The driver ought to sit somewhere related to its functionality.
>
> Our fpga loads from a serial EEPROM, the image is about 6.5MB.
> We can rewrite it from userspace by mmap()ing part of one of the BARs
> to access some very locally written (by me) VHDL that does most of
> the required bit-banging for 32it word accesses.

The fpga framework is intended to handle cases where the it is desired
to reprogram the fpga a lot without having to reboot.  It doesn't
sound like that is your use case.

> You really wouldn't want to load 6.5MB into kernel space!

No, and there have been proposals, shot down so far regarding
streaming firmware images a page at a time.

>
> We also had to solve the problem of 9 separate driver modules that
> want to access different parts of the BARs.
> I think we have 46 separate slaves in the fpgas BARs (most are in 1 BAR).
> Some of these are common between different boards (or completely different
> memory maps for the same board.
>
> I can imagine some generic method of having a 'board' driver for a
> specific PCI-id that knows the BAR offsets of various functions so that
> other sub-drivers could be loaded to access those functions.
> But that is some kind of pseudo-bus not fpga specific in any way.
>
>         David
>

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Alan Tull]

On Tue, Dec 5, 2017 at 11:30 PM, Wu Hao <hao.wu@intel.com> wrote:
> On Tue, Dec 05, 2017 at 11:00:22AM -0600, Alan Tull wrote:
>> On Mon, Dec 4, 2017 at 9:33 PM, Wu Hao <hao.wu@intel.com> wrote:
>> > On Mon, Dec 04, 2017 at 01:46:59PM -0600, Alan Tull wrote:
>> >> On Mon, Nov 27, 2017 at 9:15 PM, Wu Hao <hao.wu@intel.com> wrote:
>> >> > On Mon, Nov 27, 2017 at 10:28:04AM +0000, David Laight wrote:
>> >> >> From: Wu Hao
>> >> >> > Sent: 27 November 2017 06:42
>> >> >> > From: Zhang Yi <yi.z.zhang@intel.com>
>> >> >> >
>> >> >> > The Intel FPGA device appears as a PCIe device on the system. This patch
>> >> >> > implements the basic framework of the driver for Intel PCIe device which
>> >> >> > is located between CPU and Accelerated Function Units (AFUs), and has
>> >> >> > the Device Feature List (DFL) implemented in its MMIO space.
>> >> >>
>> >> >> This ought to have a better name than 'Intel FPGA'.
>> >> >> An fpga can be used for all sorts of things, this looks like
>> >> >> a very specific architecture using a common VHDL environment to
>> >> >> allow certain types of user VHDL be accessed over PCIe.
>> >> >
>> >> > Hi David
>> >> >
>> >> > This patch adds a pcie device driver for Intel FPGA devices which implements
>> >> > the DFL, e.g Intel Server Platform with In-package FPGA and Intel FPGA PCIe
>> >> > Acceleration Cards. They are pcie devices, and all have DFL implemented in
>> >> > the MMIO space, so we would like to use one kernel driver to handle them.
>> >> >
>> >> > With this full patchset, it just provides user the interfaces to configure
>> >> > and access the FPGA accelerators on Intel DFL based FPGA devices. For sure,
>> >> > users can develop and build their own logics via tools provided by Intel,
>> >> > program them to accelerators on these Intel FPGA devices, and access them
>> >> > for their workloads.
>> >>
>> >> I don't see anything Intel specific here.  This could all be named dfl-*
>> >
>> > The maybe some device specific things, e.g Intel FPGA devices supported by this
>> > driver always have FME DFL at the beginning on the BAR0 for PF device.

I'm thinking that another user could add their PCI id's and a static
FPGA image that has a DFL in the right place for this to work for
them.

>> >
>> > But I think this should be the right direction for better code reuse, it could
>> > save efforts for other vendors who want to use DFL and follow the same way.

I appreciate your understanding here.

>> >
>> > Thanks for the comments. I will rename this driver in the next version.
>>
>> Thanks!
>>
>> Regarding file names, it seems like the files added to drivers/fpga
>> could be uniformly named dfl-*.[ch].  Some are fpga-dfl-*.[ch] while
>> other are currently dfl-*.[ch] currently.
>
> Sure, will have all related drivers files renamed to dfl-*.[ch].

Actually, I'll reverse that a bit.  The enumeration code, including
the pcie part is all sufficiently general to run on anything that has
a DFL struct located in the right place.  But individual feature
drivers (currently only the fme-mgr) will be vendor specific and could
be named intel-*.

Alan

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Wu Hao]

On Thu, Feb 01, 2018 at 03:59:21PM -0600, Alan Tull wrote:
> On Tue, Dec 5, 2017 at 11:30 PM, Wu Hao <hao.wu@intel.com> wrote:
> > On Tue, Dec 05, 2017 at 11:00:22AM -0600, Alan Tull wrote:
> >> On Mon, Dec 4, 2017 at 9:33 PM, Wu Hao <hao.wu@intel.com> wrote:
> >> > On Mon, Dec 04, 2017 at 01:46:59PM -0600, Alan Tull wrote:
> >> >> On Mon, Nov 27, 2017 at 9:15 PM, Wu Hao <hao.wu@intel.com> wrote:
> >> >> > On Mon, Nov 27, 2017 at 10:28:04AM +0000, David Laight wrote:
> >> >> >> From: Wu Hao
> >> >> >> > Sent: 27 November 2017 06:42
> >> >> >> > From: Zhang Yi <yi.z.zhang@intel.com>
> >> >> >> >
> >> >> >> > The Intel FPGA device appears as a PCIe device on the system. This patch
> >> >> >> > implements the basic framework of the driver for Intel PCIe device which
> >> >> >> > is located between CPU and Accelerated Function Units (AFUs), and has
> >> >> >> > the Device Feature List (DFL) implemented in its MMIO space.
> >> >> >>
> >> >> >> This ought to have a better name than 'Intel FPGA'.
> >> >> >> An fpga can be used for all sorts of things, this looks like
> >> >> >> a very specific architecture using a common VHDL environment to
> >> >> >> allow certain types of user VHDL be accessed over PCIe.
> >> >> >
> >> >> > Hi David
> >> >> >
> >> >> > This patch adds a pcie device driver for Intel FPGA devices which implements
> >> >> > the DFL, e.g Intel Server Platform with In-package FPGA and Intel FPGA PCIe
> >> >> > Acceleration Cards. They are pcie devices, and all have DFL implemented in
> >> >> > the MMIO space, so we would like to use one kernel driver to handle them.
> >> >> >
> >> >> > With this full patchset, it just provides user the interfaces to configure
> >> >> > and access the FPGA accelerators on Intel DFL based FPGA devices. For sure,
> >> >> > users can develop and build their own logics via tools provided by Intel,
> >> >> > program them to accelerators on these Intel FPGA devices, and access them
> >> >> > for their workloads.
> >> >>
> >> >> I don't see anything Intel specific here.  This could all be named dfl-*
> >> >
> >> > The maybe some device specific things, e.g Intel FPGA devices supported by this
> >> > driver always have FME DFL at the beginning on the BAR0 for PF device.
> 
> I'm thinking that another user could add their PCI id's and a static
> FPGA image that has a DFL in the right place for this to work for
> them.
> 
> >> >
> >> > But I think this should be the right direction for better code reuse, it could
> >> > save efforts for other vendors who want to use DFL and follow the same way.
> 
> I appreciate your understanding here.
> 
> >> >
> >> > Thanks for the comments. I will rename this driver in the next version.
> >>
> >> Thanks!
> >>
> >> Regarding file names, it seems like the files added to drivers/fpga
> >> could be uniformly named dfl-*.[ch].  Some are fpga-dfl-*.[ch] while
> >> other are currently dfl-*.[ch] currently.
> >
> > Sure, will have all related drivers files renamed to dfl-*.[ch].
> 
> Actually, I'll reverse that a bit.  The enumeration code, including
> the pcie part is all sufficiently general to run on anything that has
> a DFL struct located in the right place.  But individual feature
> drivers (currently only the fme-mgr) will be vendor specific and could
> be named intel-*.

Hi Alan,

I unified all the drivers to use dfl-* in the v4 patchset, including fme-mgr.
As I feel it's hard to say which FME functions (sub features, registers) are
vendor specific and which FME functions are not, from the spec, they all belong
to FME, and people can be reused for sure. So I didn't rename it back to Intel
driver in the v4 patchset. :)

Thanks
Hao


> 
> Alan

RE: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[David Laight]

From: Wu Hao
> Sent: 05 December 2017 03:34
...
> > I don't see anything Intel specific here.  This could all be named dfl-*
> 
> The maybe some device specific things, e.g Intel FPGA devices supported by this
> driver always have FME DFL at the beginning on the BAR0 for PF device.

Since when has that been a method for specifying what the card does?
You need to allocate a PCI-id for your DFL accelerator.

	David

Re: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[Wu Hao]

On Wed, Dec 06, 2017 at 09:34:14AM +0000, David Laight wrote:
> From: Wu Hao
> > Sent: 05 December 2017 03:34
> ...
> > > I don't see anything Intel specific here.  This could all be named dfl-*
> > 
> > The maybe some device specific things, e.g Intel FPGA devices supported by this
> > driver always have FME DFL at the beginning on the BAR0 for PF device.
> 
> Since when has that been a method for specifying what the card does?
> You need to allocate a PCI-id for your DFL accelerator.

This driver only supports Intel FPGA devices (see PCI Device Ids table
in this patch) as mentioned above now, per my current understanding, if
other vendors follow the same hardware design on using DFL for their PCIe
based devices, then it's possible for them to fully reuse this code,
otherwise they need to develop new drivers for their own designs or extend
this driver in some ways (it depends on actual hardware implementation).

Thanks
Hao

> 
> 	David

RE: [PATCH v3 08/21] fpga: add Intel FPGA DFL PCIe device

[David Laight]

From: Alan Tull
> Sent: 04 December 2017 19:47
> 
> On Mon, Nov 27, 2017 at 9:15 PM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Nov 27, 2017 at 10:28:04AM +0000, David Laight wrote:
> >> From: Wu Hao
> >> > Sent: 27 November 2017 06:42
> >> > From: Zhang Yi <yi.z.zhang@intel.com>
> >> >
> >> > The Intel FPGA device appears as a PCIe device on the system. This patch
> >> > implements the basic framework of the driver for Intel PCIe device which
> >> > is located between CPU and Accelerated Function Units (AFUs), and has
> >> > the Device Feature List (DFL) implemented in its MMIO space.
> >>
> >> This ought to have a better name than 'Intel FPGA'.
> >> An fpga can be used for all sorts of things, this looks like
> >> a very specific architecture using a common VHDL environment to
> >> allow certain types of user VHDL be accessed over PCIe.
> >
> > Hi David
> >
> > This patch adds a pcie device driver for Intel FPGA devices which implements
> > the DFL, e.g Intel Server Platform with In-package FPGA and Intel FPGA PCIe
> > Acceleration Cards. They are pcie devices, and all have DFL implemented in
> > the MMIO space, so we would like to use one kernel driver to handle them.
> >
> > With this full patchset, it just provides user the interfaces to configure
> > and access the FPGA accelerators on Intel DFL based FPGA devices. For sure,
> > users can develop and build their own logics via tools provided by Intel,
> > program them to accelerators on these Intel FPGA devices, and access them
> > for their workloads.
> 
> I don't see anything Intel specific here.  This could all be named dfl-*

Indeed, doesn't even seem to have to be implemented in an fpga.
It might also not be the only device that implements DFL.
You really need a name for your DFL acceleration implementation/interface.

We make a board that uses an Intel/Altera fpga as a PCIe device, won't look
anything like your one!

	David

[PATCH v3 09/21] fpga: intel-dfl-pci: add enumeration for feature devices

[Wu Hao]

The Device Feature List (DFL) is implemented in MMIO, and features
are linked via the DFLs. This patch enables pcie driver to prepare
enumeration information (e.g locations of all device feature lists
in MMIO) and use common APIs provided by the Device Feature List
framework to enumerate each feature device linked.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: split from another patch
    use common functions from DFL framework for enumeration.
---
 drivers/fpga/intel-dfl-pci.c | 199 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 197 insertions(+), 2 deletions(-)

diff --git a/drivers/fpga/intel-dfl-pci.c b/drivers/fpga/intel-dfl-pci.c
index 4774a77..1c7ba22 100644
--- a/drivers/fpga/intel-dfl-pci.c
+++ b/drivers/fpga/intel-dfl-pci.c
@@ -24,9 +24,52 @@
 #include <linux/errno.h>
 #include <linux/aer.h>
 
+#include "fpga-dfl.h"
+
 #define DRV_VERSION	"0.8"
 #define DRV_NAME	"intel-dfl-pci"
 
+struct cci_drvdata {
+	struct fpga_cdev *cdev;	/* container device */
+	struct list_head regions; /* list of pci bar mapping region */
+};
+
+/* pci bar mapping info */
+struct cci_region {
+	int bar;
+	void __iomem *ioaddr;	/* pointer to mapped bar region */
+	struct list_head node;
+};
+
+static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pcidev, int bar)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+	struct cci_region *region;
+
+	list_for_each_entry(region, &drvdata->regions, node)
+		if (region->bar == bar) {
+			dev_dbg(&pcidev->dev, "BAR %d region exists\n", bar);
+			return region->ioaddr;
+		}
+
+	region = devm_kzalloc(&pcidev->dev, sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return NULL;
+
+	region->bar = bar;
+	region->ioaddr = pci_ioremap_bar(pcidev, bar);
+	if (!region->ioaddr) {
+		dev_err(&pcidev->dev, "can't ioremap memory from BAR %d.\n",
+			bar);
+		devm_kfree(&pcidev->dev, region);
+		return NULL;
+	}
+
+	list_add(&region->node, &drvdata->regions);
+
+	return region->ioaddr;
+}
+
 /* PCI Device ID */
 #define PCIE_DEVICE_ID_PF_INT_5_X	0xBCBD
 #define PCIE_DEVICE_ID_PF_INT_6_X	0xBCC0
@@ -47,6 +90,143 @@
 };
 MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
 
+static int cci_init_drvdata(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata;
+
+	drvdata = devm_kzalloc(&pcidev->dev, sizeof(*drvdata), GFP_KERNEL);
+	if (!drvdata)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&drvdata->regions);
+
+	pci_set_drvdata(pcidev, drvdata);
+
+	return 0;
+}
+
+static void cci_pci_release_regions(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+	struct cci_region *tmp, *region;
+
+	list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
+		list_del(&region->node);
+		if (region->ioaddr)
+			pci_iounmap(pcidev, region->ioaddr);
+		devm_kfree(&pcidev->dev, region);
+	}
+}
+
+static void cci_remove_drvdata(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+
+	cci_pci_release_regions(pcidev);
+	pci_set_drvdata(pcidev, NULL);
+	devm_kfree(&pcidev->dev, drvdata);
+}
+
+static void cci_remove_feature_devs(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+
+	/* remove all children feature devices */
+	fpga_remove_feature_devs(drvdata->cdev);
+}
+
+/* enumerate feature devices under pci device */
+static int cci_enumerate_feature_devs(struct pci_dev *pcidev)
+{
+	struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
+	struct fpga_cdev *cdev;
+	struct fpga_enum_info *info;
+	resource_size_t start, len;
+	void __iomem *base;
+	int port_num, bar, i, ret = 0;
+	u32 offset;
+	u64 v;
+
+	/* allocate enumeration info via pci_dev */
+	info = fpga_enum_info_alloc(&pcidev->dev);
+	if (!info)
+		return -ENOMEM;
+
+	/* start to find Device Feature List from Bar 0 */
+	base = cci_pci_ioremap_bar(pcidev, 0);
+	if (!base) {
+		ret = -ENOMEM;
+		goto enum_info_free_exit;
+	}
+
+	/*
+	 * PF device has FME and Ports/AFUs, and VF device only has 1 Port/AFU.
+	 * check them and add related "Device Feature List" info for the next
+	 * step enumeration.
+	 */
+	if (feature_is_fme(base)) {
+		start = pci_resource_start(pcidev, 0);
+		len = pci_resource_len(pcidev, 0);
+
+		fpga_enum_info_add_dfl(info, start, len, base);
+
+		/*
+		 * find more Device Feature Lists (e.g Ports) per information
+		 * indicated by FME module.
+		 */
+		v = readq(base + FME_HDR_CAP);
+		port_num = FIELD_GET(FME_CAP_NUM_PORTS, v);
+
+		WARN_ON(port_num > MAX_FPGA_PORT_NUM);
+
+		for (i = 0; i < port_num; i++) {
+			v = readq(base + FME_HDR_PORT_OFST(i));
+
+			/* skip ports which are not implemented. */
+			if (!(v & FME_PORT_OFST_IMP))
+				continue;
+
+			/*
+			 * add Port's Device Feature List information for next
+			 * step enumeration.
+			 */
+			bar = FIELD_GET(FME_PORT_OFST_BAR_ID, v);
+			offset = FIELD_GET(FME_PORT_OFST_DFH_OFST, v);
+			base = cci_pci_ioremap_bar(pcidev, bar);
+			if (!base)
+				continue;
+
+			start = pci_resource_start(pcidev, bar) + offset;
+			len = pci_resource_len(pcidev, bar) - offset;
+
+			fpga_enum_info_add_dfl(info, start, len, base + offset);
+		}
+	} else if (feature_is_port(base)) {
+		start = pci_resource_start(pcidev, 0);
+		len = pci_resource_len(pcidev, 0);
+
+		fpga_enum_info_add_dfl(info, start, len, base);
+	} else {
+		ret = -ENODEV;
+		goto enum_info_free_exit;
+	}
+
+	/* start enumeration with prepared enumeration information */
+	cdev = fpga_enumerate_feature_devs(info);
+	if (IS_ERR(cdev)) {
+		dev_err(&pcidev->dev, "Enumeration failure\n");
+		ret = PTR_ERR(cdev);
+		goto enum_info_free_exit;
+	}
+
+	drvdata->cdev = cdev;
+
+enum_info_free_exit:
+	fpga_enum_info_free(info);
+
+	return ret;
+}
+
 static
 int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 {
@@ -84,9 +264,22 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 		goto release_region_exit;
 	}
 
-	/* TODO: create and add the platform device per feature list */
-	return 0;
+	ret = cci_init_drvdata(pcidev);
+	if (ret) {
+		dev_err(&pcidev->dev, "Fail to init drvdata %d.\n", ret);
+		goto release_region_exit;
+	}
+
+	ret = cci_enumerate_feature_devs(pcidev);
+	if (ret) {
+		dev_err(&pcidev->dev, "enumeration failure %d.\n", ret);
+		goto remove_drvdata_exit;
+	}
+
+	return ret;
 
+remove_drvdata_exit:
+	cci_remove_drvdata(pcidev);
 release_region_exit:
 	pci_release_regions(pcidev);
 disable_error_report_exit:
@@ -97,6 +290,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 
 static void cci_pci_remove(struct pci_dev *pcidev)
 {
+	cci_remove_feature_devs(pcidev);
+	cci_remove_drvdata(pcidev);
 	pci_release_regions(pcidev);
 	pci_disable_pcie_error_reporting(pcidev);
 	pci_disable_device(pcidev);
-- 
1.8.3.1

Re: [PATCH v3 09/21] fpga: intel-dfl-pci: add enumeration for feature devices

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

> +/* enumerate feature devices under pci device */
> +static int cci_enumerate_feature_devs(struct pci_dev *pcidev)
> +{
> +       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
> +       struct fpga_cdev *cdev;
> +       struct fpga_enum_info *info;
> +       resource_size_t start, len;
> +       void __iomem *base;
> +       int port_num, bar, i, ret = 0;
> +       u32 offset;
> +       u64 v;
> +
> +       /* allocate enumeration info via pci_dev */
> +       info = fpga_enum_info_alloc(&pcidev->dev);
> +       if (!info)
> +               return -ENOMEM;
> +
> +       /* start to find Device Feature List from Bar 0 */
> +       base = cci_pci_ioremap_bar(pcidev, 0);
> +       if (!base) {
> +               ret = -ENOMEM;
> +               goto enum_info_free_exit;
> +       }
> +
> +       /*
> +        * PF device has FME and Ports/AFUs, and VF device only has 1 Port/AFU.
> +        * check them and add related "Device Feature List" info for the next
> +        * step enumeration.
> +        */
> +       if (feature_is_fme(base)) {
> +               start = pci_resource_start(pcidev, 0);
> +               len = pci_resource_len(pcidev, 0);
> +
> +               fpga_enum_info_add_dfl(info, start, len, base);
> +
> +               /*
> +                * find more Device Feature Lists (e.g Ports) per information
> +                * indicated by FME module.
> +                */
> +               v = readq(base + FME_HDR_CAP);
> +               port_num = FIELD_GET(FME_CAP_NUM_PORTS, v);
> +
> +               WARN_ON(port_num > MAX_FPGA_PORT_NUM);
> +
> +               for (i = 0; i < port_num; i++) {
> +                       v = readq(base + FME_HDR_PORT_OFST(i));
> +
> +                       /* skip ports which are not implemented. */
> +                       if (!(v & FME_PORT_OFST_IMP))
> +                               continue;
> +
> +                       /*
> +                        * add Port's Device Feature List information for next
> +                        * step enumeration.
> +                        */
> +                       bar = FIELD_GET(FME_PORT_OFST_BAR_ID, v);
> +                       offset = FIELD_GET(FME_PORT_OFST_DFH_OFST, v);
> +                       base = cci_pci_ioremap_bar(pcidev, bar);
> +                       if (!base)
> +                               continue;
> +
> +                       start = pci_resource_start(pcidev, bar) + offset;
> +                       len = pci_resource_len(pcidev, bar) - offset;
> +
> +                       fpga_enum_info_add_dfl(info, start, len, base + offset);
> +               }
> +       } else if (feature_is_port(base)) {
> +               start = pci_resource_start(pcidev, 0);
> +               len = pci_resource_len(pcidev, 0);
> +
> +               fpga_enum_info_add_dfl(info, start, len, base);
> +       } else {
> +               ret = -ENODEV;
> +               goto enum_info_free_exit;
> +       }
> +
> +       /* start enumeration with prepared enumeration information */
> +       cdev = fpga_enumerate_feature_devs(info);

Hi Hao,

I appreciate you separating the DFL enumeration code from this PCIe
module.  This made the pcie part quite small.  It should work for
embedded platforms just by adding a platform device whose function is
to find the DFL structures at some address and then call these same
fpga_enum_info_add_dfl adn fpga_enumerate_feature_devs functions.

Alan

> +       if (IS_ERR(cdev)) {
> +               dev_err(&pcidev->dev, "Enumeration failure\n");
> +               ret = PTR_ERR(cdev);
> +               goto enum_info_free_exit;
> +       }
> +
> +       drvdata->cdev = cdev;
> +
> +enum_info_free_exit:
> +       fpga_enum_info_free(info);
> +
> +       return ret;
> +}
> +
>  static
>  int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>  {
> @@ -84,9 +264,22 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>                 goto release_region_exit;
>         }
>
> -       /* TODO: create and add the platform device per feature list */
> -       return 0;
> +       ret = cci_init_drvdata(pcidev);
> +       if (ret) {
> +               dev_err(&pcidev->dev, "Fail to init drvdata %d.\n", ret);
> +               goto release_region_exit;
> +       }
> +
> +       ret = cci_enumerate_feature_devs(pcidev);
> +       if (ret) {
> +               dev_err(&pcidev->dev, "enumeration failure %d.\n", ret);
> +               goto remove_drvdata_exit;
> +       }
> +
> +       return ret;
>
> +remove_drvdata_exit:
> +       cci_remove_drvdata(pcidev);
>  release_region_exit:
>         pci_release_regions(pcidev);
>  disable_error_report_exit:
> @@ -97,6 +290,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>
>  static void cci_pci_remove(struct pci_dev *pcidev)
>  {
> +       cci_remove_feature_devs(pcidev);
> +       cci_remove_drvdata(pcidev);
>         pci_release_regions(pcidev);
>         pci_disable_pcie_error_reporting(pcidev);
>         pci_disable_device(pcidev);
> --
> 1.8.3.1
>

Re: [PATCH v3 09/21] fpga: intel-dfl-pci: add enumeration for feature devices

[Wu Hao]

On Thu, Dec 07, 2017 at 03:41:41PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> 
> > +/* enumerate feature devices under pci device */
> > +static int cci_enumerate_feature_devs(struct pci_dev *pcidev)
> > +{
> > +       struct cci_drvdata *drvdata = pci_get_drvdata(pcidev);
> > +       struct fpga_cdev *cdev;
> > +       struct fpga_enum_info *info;
> > +       resource_size_t start, len;
> > +       void __iomem *base;
> > +       int port_num, bar, i, ret = 0;
> > +       u32 offset;
> > +       u64 v;
> > +
> > +       /* allocate enumeration info via pci_dev */
> > +       info = fpga_enum_info_alloc(&pcidev->dev);
> > +       if (!info)
> > +               return -ENOMEM;
> > +
> > +       /* start to find Device Feature List from Bar 0 */
> > +       base = cci_pci_ioremap_bar(pcidev, 0);
> > +       if (!base) {
> > +               ret = -ENOMEM;
> > +               goto enum_info_free_exit;
> > +       }
> > +
> > +       /*
> > +        * PF device has FME and Ports/AFUs, and VF device only has 1 Port/AFU.
> > +        * check them and add related "Device Feature List" info for the next
> > +        * step enumeration.
> > +        */
> > +       if (feature_is_fme(base)) {
> > +               start = pci_resource_start(pcidev, 0);
> > +               len = pci_resource_len(pcidev, 0);
> > +
> > +               fpga_enum_info_add_dfl(info, start, len, base);
> > +
> > +               /*
> > +                * find more Device Feature Lists (e.g Ports) per information
> > +                * indicated by FME module.
> > +                */
> > +               v = readq(base + FME_HDR_CAP);
> > +               port_num = FIELD_GET(FME_CAP_NUM_PORTS, v);
> > +
> > +               WARN_ON(port_num > MAX_FPGA_PORT_NUM);
> > +
> > +               for (i = 0; i < port_num; i++) {
> > +                       v = readq(base + FME_HDR_PORT_OFST(i));
> > +
> > +                       /* skip ports which are not implemented. */
> > +                       if (!(v & FME_PORT_OFST_IMP))
> > +                               continue;
> > +
> > +                       /*
> > +                        * add Port's Device Feature List information for next
> > +                        * step enumeration.
> > +                        */
> > +                       bar = FIELD_GET(FME_PORT_OFST_BAR_ID, v);
> > +                       offset = FIELD_GET(FME_PORT_OFST_DFH_OFST, v);
> > +                       base = cci_pci_ioremap_bar(pcidev, bar);
> > +                       if (!base)
> > +                               continue;
> > +
> > +                       start = pci_resource_start(pcidev, bar) + offset;
> > +                       len = pci_resource_len(pcidev, bar) - offset;
> > +
> > +                       fpga_enum_info_add_dfl(info, start, len, base + offset);
> > +               }
> > +       } else if (feature_is_port(base)) {
> > +               start = pci_resource_start(pcidev, 0);
> > +               len = pci_resource_len(pcidev, 0);
> > +
> > +               fpga_enum_info_add_dfl(info, start, len, base);
> > +       } else {
> > +               ret = -ENODEV;
> > +               goto enum_info_free_exit;
> > +       }
> > +
> > +       /* start enumeration with prepared enumeration information */
> > +       cdev = fpga_enumerate_feature_devs(info);
> 
> Hi Hao,
> 
> I appreciate you separating the DFL enumeration code from this PCIe
> module.  This made the pcie part quite small.  It should work for
> embedded platforms just by adding a platform device whose function is
> to find the DFL structures at some address and then call these same
> fpga_enum_info_add_dfl adn fpga_enumerate_feature_devs functions.

Yes, I think this is the right direction as you suggested. : )

Thanks
Hao

> 
> Alan

[PATCH v3 10/21] fpga: dfl: add FPGA Management Engine driver basic framework

[Wu Hao]

From: Kang Luwei <luwei.kang@intel.com>

The FPGA Management Engine (FME) provides power, thermal management,
performance counters, partial reconfiguration and other functions. For each
function, it is packaged into a private feature linked to the FME feature
device in the 'Device Feature List'. It's a platform device created by
DFL framework.

This patch adds the basic framework of FME platform driver. It defines
sub feature drivers to handle the different sub features, including init,
uinit and ioctl. It also registers the file operations for the device file.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: rename driver from intel-fpga-fme to dfl-fme
    rename Kconfig from INTEL_FPGA_FME to FPGA_DFL_FME
---
 drivers/fpga/Kconfig        |  10 +++
 drivers/fpga/Makefile       |   3 +
 drivers/fpga/dfl-fme-main.c | 160 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 173 insertions(+)
 create mode 100644 drivers/fpga/dfl-fme-main.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index cc35d12..57da904 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -140,6 +140,16 @@ config FPGA_DFL
 	  Gate Array (FPGA) solutions which implement Device Feature List.
 	  It provides enumeration APIs, and feature device infrastructure.
 
+config FPGA_DFL_FME
+	tristate "FPGA DFL FME Driver"
+	depends on FPGA_DFL
+	help
+	  The FPGA Management Engine (FME) is a feature device implemented
+	  under Device Feature List (DFL) framework. Select this option to
+	  enable the platform device driver for FME which implements all
+	  FPGA platform level management features. There shall be 1 FME
+	  per DFL based FPGA device.
+
 config INTEL_FPGA_DFL_PCI
 	tristate "Intel FPGA DFL PCIe Device Driver"
 	depends on PCI && FPGA_DFL
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index d39a431..e5ba158 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -30,6 +30,9 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)			+= fpga-dfl.o
+obj-$(CONFIG_FPGA_DFL_FME)		+= fpga-dfl-fme.o
+
+fpga-dfl-fme-objs := dfl-fme-main.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_INTEL_FPGA_DFL_PCI)	+= intel-dfl-pci.o
diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
new file mode 100644
index 0000000..f7b5f7d
--- /dev/null
+++ b/drivers/fpga/dfl-fme-main.c
@@ -0,0 +1,160 @@
+/*
+ * Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "fpga-dfl.h"
+
+static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "FME HDR Init.\n");
+
+	return 0;
+}
+
+static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "FME HDR UInit.\n");
+}
+
+static const struct feature_ops fme_hdr_ops = {
+	.init = fme_hdr_init,
+	.uinit = fme_hdr_uinit,
+};
+
+static struct feature_driver fme_feature_drvs[] = {
+	{
+		.name = FME_FEATURE_HEADER,
+		.ops = &fme_hdr_ops,
+	},
+	{
+		.ops = NULL,
+	},
+};
+
+static int fme_open(struct inode *inode, struct file *filp)
+{
+	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
+	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
+	int ret;
+
+	if (WARN_ON(!pdata))
+		return -ENODEV;
+
+	ret = feature_dev_use_begin(pdata);
+	if (ret)
+		return ret;
+
+	dev_dbg(&fdev->dev, "Device File Open\n");
+	filp->private_data = pdata;
+
+	return 0;
+}
+
+static int fme_release(struct inode *inode, struct file *filp)
+{
+	struct feature_platform_data *pdata = filp->private_data;
+	struct platform_device *pdev = pdata->dev;
+
+	dev_dbg(&pdev->dev, "Device File Release\n");
+	feature_dev_use_end(pdata);
+
+	return 0;
+}
+
+static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct feature_platform_data *pdata = filp->private_data;
+	struct platform_device *pdev = pdata->dev;
+	struct feature *f;
+	long ret;
+
+	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
+
+	switch (cmd) {
+	default:
+		/*
+		 * Let sub-feature's ioctl function to handle the cmd
+		 * Sub-feature's ioctl returns -ENODEV when cmd is not
+		 * handled in this sub feature, and returns 0 and other
+		 * error code if cmd is handled.
+		 */
+		fpga_dev_for_each_feature(pdata, f) {
+			if (f->ops && f->ops->ioctl) {
+				ret = f->ops->ioctl(pdev, f, cmd, arg);
+				if (ret == -ENODEV)
+					continue;
+				else
+					return ret;
+			}
+		}
+	}
+
+	return -EINVAL;
+}
+
+static const struct file_operations fme_fops = {
+	.owner		= THIS_MODULE,
+	.open		= fme_open,
+	.release	= fme_release,
+	.unlocked_ioctl = fme_ioctl,
+};
+
+static int fme_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	if (ret)
+		goto exit;
+
+	ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
+	if (ret)
+		goto feature_uinit;
+
+	return 0;
+
+feature_uinit:
+	fpga_dev_feature_uinit(pdev);
+exit:
+	return ret;
+}
+
+static int fme_remove(struct platform_device *pdev)
+{
+	fpga_dev_feature_uinit(pdev);
+	fpga_unregister_dev_ops(pdev);
+
+	return 0;
+}
+
+static struct platform_driver fme_driver = {
+	.driver	= {
+		.name    = FPGA_FEATURE_DEV_FME,
+	},
+	.probe   = fme_probe,
+	.remove  = fme_remove,
+};
+
+module_platform_driver(fme_driver);
+
+MODULE_DESCRIPTION("FPGA Management Engine driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:fpga-dfl-fme");
-- 
1.8.3.1

[PATCH v3 11/21] fpga: dfl: fme: add header sub feature support

[Wu Hao]

From: Kang Luwei <luwei.kang@intel.com>

The header register set is always present for FPGA Management Engine (FME),
this patch implements init and uinit function for header sub feature and
introduce several read-only sysfs interfaces for the capability and status.

Sysfs interfaces:
* /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.x>/ports_num
  Read-only. Number of ports implemented

* /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.x>/bitstream_id
  Read-only. Blue Bitstream (static FPGA region) identifier number

* /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.x>/bitstream_metadata
  Read-only. Blue Bitstream (static FPGA region) meta data

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v2: add sysfs documentation
v3: rename driver to fpga-dfl-fme.
    improve sysfs doc and commit description.
    replace bitfield.
---
 .../ABI/testing/sysfs-platform-fpga-dfl-fme        | 21 ++++++++
 drivers/fpga/dfl-fme-main.c                        | 60 ++++++++++++++++++++++
 2 files changed, 81 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme

diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme
new file mode 100644
index 0000000..6b32799
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme
@@ -0,0 +1,21 @@
+What:		/sys/bus/platform/devices/fpga-dfl-fme.0/ports_num
+Date:		November 2017
+KernelVersion:  4.15
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. One DFL FPGA device may have more than 1
+		port/Accelerator Function Unit (AFU). It returns the
+		number of ports on the FPGA device when read it.
+
+What:		/sys/bus/platform/devices/fpga-dfl-fme.0/bitstream_id
+Date:		November 2017
+KernelVersion:  4.15
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. It returns Blue Bitstream (static FPGA region)
+		identifier number.
+
+What:		/sys/bus/platform/devices/fpga-dfl-fme.0/bitstream_meta
+Date:		November 2017
+KernelVersion:  4.15
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. It returns Blue Bitstream (static FPGA region)
+		meta data.
diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index f7b5f7d..d17c66a 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c
@@ -21,9 +21,68 @@
 
 #include "fpga-dfl.h"
 
+static ssize_t ports_num_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	void __iomem *base;
+	u64 v;
+
+	base = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
+
+	v = readq(base + FME_HDR_CAP);
+
+	return scnprintf(buf, PAGE_SIZE, "%u\n",
+			 (unsigned int)FIELD_GET(FME_CAP_NUM_PORTS, v));
+}
+static DEVICE_ATTR_RO(ports_num);
+
+static ssize_t bitstream_id_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	void __iomem *base;
+	u64 v;
+
+	base = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
+
+	v = readq(base + FME_HDR_BITSTREAM_ID);
+
+	return scnprintf(buf, PAGE_SIZE, "0x%llx\n", (unsigned long long)v);
+}
+static DEVICE_ATTR_RO(bitstream_id);
+
+static ssize_t bitstream_metadata_show(struct device *dev,
+				       struct device_attribute *attr, char *buf)
+{
+	void __iomem *base;
+	u64 v;
+
+	base = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
+
+	v = readq(base + FME_HDR_BITSTREAM_MD);
+
+	return scnprintf(buf, PAGE_SIZE, "0x%llx\n", (unsigned long long)v);
+}
+static DEVICE_ATTR_RO(bitstream_metadata);
+
+static const struct attribute *fme_hdr_attrs[] = {
+	&dev_attr_ports_num.attr,
+	&dev_attr_bitstream_id.attr,
+	&dev_attr_bitstream_metadata.attr,
+	NULL,
+};
+
 static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
 {
+	void __iomem *base = feature->ioaddr;
+	int ret;
+
 	dev_dbg(&pdev->dev, "FME HDR Init.\n");
+	dev_dbg(&pdev->dev, "FME cap %llx.\n",
+		(unsigned long long)readq(base + FME_HDR_CAP));
+
+	ret = sysfs_create_files(&pdev->dev.kobj, fme_hdr_attrs);
+	if (ret)
+		return ret;
 
 	return 0;
 }
@@ -31,6 +90,7 @@ static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
 static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "FME HDR UInit.\n");
+	sysfs_remove_files(&pdev->dev.kobj, fme_hdr_attrs);
 }
 
 static const struct feature_ops fme_hdr_ops = {
-- 
1.8.3.1

Re: [PATCH v3 11/21] fpga: dfl: fme: add header sub feature support

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> From: Kang Luwei <luwei.kang@intel.com>
>
> The header register set is always present for FPGA Management Engine (FME),
> this patch implements init and uinit function for header sub feature and
> introduce several read-only sysfs interfaces for the capability and status.
>
> Sysfs interfaces:
> * /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.x>/ports_num
>   Read-only. Number of ports implemented
>
> * /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.x>/bitstream_id
>   Read-only. Blue Bitstream (static FPGA region) identifier number
>
> * /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.x>/bitstream_metadata
>   Read-only. Blue Bitstream (static FPGA region) meta data

Please document the meta data.  I don't see anywhere that describes it
or how it differs from the bitstream_id.  So it could be useful to
document it in this header a bit, and in the code, in the sysfs
document, and in the Documentation/fpga so that people don't have to
go hunting.

We discussed elsewhere the static region bitstream_id verses the other
interface_ids, so that's been discussed.

Besides that, this patch looks good and straightforward.

Thanks,
Alan

>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v2: add sysfs documentation
> v3: rename driver to fpga-dfl-fme.
>     improve sysfs doc and commit description.
>     replace bitfield.
> ---
>  .../ABI/testing/sysfs-platform-fpga-dfl-fme        | 21 ++++++++
>  drivers/fpga/dfl-fme-main.c                        | 60 ++++++++++++++++++++++
>  2 files changed, 81 insertions(+)
>  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme
>
> diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme
> new file mode 100644
> index 0000000..6b32799
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme
> @@ -0,0 +1,21 @@
> +What:          /sys/bus/platform/devices/fpga-dfl-fme.0/ports_num
> +Date:          November 2017
> +KernelVersion:  4.15
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read-only. One DFL FPGA device may have more than 1
> +               port/Accelerator Function Unit (AFU). It returns the
> +               number of ports on the FPGA device when read it.
> +
> +What:          /sys/bus/platform/devices/fpga-dfl-fme.0/bitstream_id
> +Date:          November 2017
> +KernelVersion:  4.15
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read-only. It returns Blue Bitstream (static FPGA region)
> +               identifier number.
> +
> +What:          /sys/bus/platform/devices/fpga-dfl-fme.0/bitstream_meta
> +Date:          November 2017
> +KernelVersion:  4.15
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read-only. It returns Blue Bitstream (static FPGA region)
> +               meta data.
> diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
> index f7b5f7d..d17c66a 100644
> --- a/drivers/fpga/dfl-fme-main.c
> +++ b/drivers/fpga/dfl-fme-main.c
> @@ -21,9 +21,68 @@
>
>  #include "fpga-dfl.h"
>
> +static ssize_t ports_num_show(struct device *dev,
> +                             struct device_attribute *attr, char *buf)
> +{
> +       void __iomem *base;
> +       u64 v;
> +
> +       base = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
> +
> +       v = readq(base + FME_HDR_CAP);
> +
> +       return scnprintf(buf, PAGE_SIZE, "%u\n",
> +                        (unsigned int)FIELD_GET(FME_CAP_NUM_PORTS, v));
> +}
> +static DEVICE_ATTR_RO(ports_num);
> +
> +static ssize_t bitstream_id_show(struct device *dev,
> +                                struct device_attribute *attr, char *buf)
> +{
> +       void __iomem *base;
> +       u64 v;
> +
> +       base = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
> +
> +       v = readq(base + FME_HDR_BITSTREAM_ID);
> +
> +       return scnprintf(buf, PAGE_SIZE, "0x%llx\n", (unsigned long long)v);
> +}
> +static DEVICE_ATTR_RO(bitstream_id);
> +
> +static ssize_t bitstream_metadata_show(struct device *dev,
> +                                      struct device_attribute *attr, char *buf)
> +{
> +       void __iomem *base;
> +       u64 v;
> +
> +       base = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
> +
> +       v = readq(base + FME_HDR_BITSTREAM_MD);
> +
> +       return scnprintf(buf, PAGE_SIZE, "0x%llx\n", (unsigned long long)v);
> +}
> +static DEVICE_ATTR_RO(bitstream_metadata);
> +
> +static const struct attribute *fme_hdr_attrs[] = {
> +       &dev_attr_ports_num.attr,
> +       &dev_attr_bitstream_id.attr,
> +       &dev_attr_bitstream_metadata.attr,
> +       NULL,
> +};
> +
>  static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
>  {
> +       void __iomem *base = feature->ioaddr;
> +       int ret;
> +
>         dev_dbg(&pdev->dev, "FME HDR Init.\n");
> +       dev_dbg(&pdev->dev, "FME cap %llx.\n",
> +               (unsigned long long)readq(base + FME_HDR_CAP));
> +
> +       ret = sysfs_create_files(&pdev->dev.kobj, fme_hdr_attrs);
> +       if (ret)
> +               return ret;
>
>         return 0;
>  }
> @@ -31,6 +90,7 @@ static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
>  static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
>  {
>         dev_dbg(&pdev->dev, "FME HDR UInit.\n");
> +       sysfs_remove_files(&pdev->dev.kobj, fme_hdr_attrs);
>  }
>
>  static const struct feature_ops fme_hdr_ops = {
> --
> 1.8.3.1
>

Re: [PATCH v3 11/21] fpga: dfl: fme: add header sub feature support

[Wu Hao]

On Mon, Feb 12, 2018 at 10:51:44AM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > From: Kang Luwei <luwei.kang@intel.com>
> >
> > The header register set is always present for FPGA Management Engine (FME),
> > this patch implements init and uinit function for header sub feature and
> > introduce several read-only sysfs interfaces for the capability and status.
> >
> > Sysfs interfaces:
> > * /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.x>/ports_num
> >   Read-only. Number of ports implemented
> >
> > * /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.x>/bitstream_id
> >   Read-only. Blue Bitstream (static FPGA region) identifier number
> >
> > * /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.x>/bitstream_metadata
> >   Read-only. Blue Bitstream (static FPGA region) meta data
> 
> Please document the meta data.  I don't see anywhere that describes it
> or how it differs from the bitstream_id.  So it could be useful to
> document it in this header a bit, and in the code, in the sysfs
> document, and in the Documentation/fpga so that people don't have to
> go hunting.

Hi Alan

Thanks a lot for the review.

The Blue Bitstream (static FPGA region) meta data contains information
like synthesis date and seed.

The Blue Bitstream (static FPGA region) id contains version information
like major, minor, patch number (e.g version 6.4.0)and etc.

I used bitstream_id a lot, but bitstream_metadata is useful in some cases
as well.

I will add more descriptions for them.

Thanks
Hao

> 
> We discussed elsewhere the static region bitstream_id verses the other
> interface_ids, so that's been discussed.
> 
> Besides that, this patch looks good and straightforward.
> 
> Thanks,
> Alan
> 
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ----
> > v2: add sysfs documentation
> > v3: rename driver to fpga-dfl-fme.
> >     improve sysfs doc and commit description.
> >     replace bitfield.
> > ---
> >  .../ABI/testing/sysfs-platform-fpga-dfl-fme        | 21 ++++++++
> >  drivers/fpga/dfl-fme-main.c                        | 60 ++++++++++++++++++++++
> >  2 files changed, 81 insertions(+)
> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme
> >
> > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme
> > new file mode 100644
> > index 0000000..6b32799
> > --- /dev/null
> > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme
> > @@ -0,0 +1,21 @@
> > +What:          /sys/bus/platform/devices/fpga-dfl-fme.0/ports_num
> > +Date:          November 2017
> > +KernelVersion:  4.15
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read-only. One DFL FPGA device may have more than 1
> > +               port/Accelerator Function Unit (AFU). It returns the
> > +               number of ports on the FPGA device when read it.
> > +
> > +What:          /sys/bus/platform/devices/fpga-dfl-fme.0/bitstream_id
> > +Date:          November 2017
> > +KernelVersion:  4.15
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read-only. It returns Blue Bitstream (static FPGA region)
> > +               identifier number.
> > +
> > +What:          /sys/bus/platform/devices/fpga-dfl-fme.0/bitstream_meta
> > +Date:          November 2017
> > +KernelVersion:  4.15
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read-only. It returns Blue Bitstream (static FPGA region)
> > +               meta data.
> > diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
> > index f7b5f7d..d17c66a 100644
> > --- a/drivers/fpga/dfl-fme-main.c
> > +++ b/drivers/fpga/dfl-fme-main.c
> > @@ -21,9 +21,68 @@
> >
> >  #include "fpga-dfl.h"
> >
> > +static ssize_t ports_num_show(struct device *dev,
> > +                             struct device_attribute *attr, char *buf)
> > +{
> > +       void __iomem *base;
> > +       u64 v;
> > +
> > +       base = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
> > +
> > +       v = readq(base + FME_HDR_CAP);
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "%u\n",
> > +                        (unsigned int)FIELD_GET(FME_CAP_NUM_PORTS, v));
> > +}
> > +static DEVICE_ATTR_RO(ports_num);
> > +
> > +static ssize_t bitstream_id_show(struct device *dev,
> > +                                struct device_attribute *attr, char *buf)
> > +{
> > +       void __iomem *base;
> > +       u64 v;
> > +
> > +       base = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
> > +
> > +       v = readq(base + FME_HDR_BITSTREAM_ID);
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "0x%llx\n", (unsigned long long)v);
> > +}
> > +static DEVICE_ATTR_RO(bitstream_id);
> > +
> > +static ssize_t bitstream_metadata_show(struct device *dev,
> > +                                      struct device_attribute *attr, char *buf)
> > +{
> > +       void __iomem *base;
> > +       u64 v;
> > +
> > +       base = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
> > +
> > +       v = readq(base + FME_HDR_BITSTREAM_MD);
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "0x%llx\n", (unsigned long long)v);
> > +}
> > +static DEVICE_ATTR_RO(bitstream_metadata);
> > +
> > +static const struct attribute *fme_hdr_attrs[] = {
> > +       &dev_attr_ports_num.attr,
> > +       &dev_attr_bitstream_id.attr,
> > +       &dev_attr_bitstream_metadata.attr,
> > +       NULL,
> > +};
> > +
> >  static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
> >  {
> > +       void __iomem *base = feature->ioaddr;
> > +       int ret;
> > +
> >         dev_dbg(&pdev->dev, "FME HDR Init.\n");
> > +       dev_dbg(&pdev->dev, "FME cap %llx.\n",
> > +               (unsigned long long)readq(base + FME_HDR_CAP));
> > +
> > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_hdr_attrs);
> > +       if (ret)
> > +               return ret;
> >
> >         return 0;
> >  }
> > @@ -31,6 +90,7 @@ static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
> >  static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
> >  {
> >         dev_dbg(&pdev->dev, "FME HDR UInit.\n");
> > +       sysfs_remove_files(&pdev->dev.kobj, fme_hdr_attrs);
> >  }
> >
> >  static const struct feature_ops fme_hdr_ops = {
> > --
> > 1.8.3.1
> >

[PATCH v3 12/21] fpga: dfl: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Wu Hao]

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
Userspace application can use these ioctl interfaces to get the API info
and check if specific extension is supported or not in current driver.

This patch implements above 2 ioctls in FPGA Management Engine (FME)
driver.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v2: switched to GPLv2 license.
v3: rename intel-fpga.h to fpga-dfl.h and rebased.
---
 Documentation/ioctl/ioctl-number.txt |  1 +
 drivers/fpga/dfl-fme-main.c          | 12 +++++++++
 include/uapi/linux/fpga-dfl.h        | 50 ++++++++++++++++++++++++++++++++++++
 3 files changed, 63 insertions(+)
 create mode 100644 include/uapi/linux/fpga-dfl.h

diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
index 3e3fdae..91ad573 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt
@@ -324,6 +324,7 @@ Code  Seq#(hex)	Include File		Comments
 0xB3	00	linux/mmc/ioctl.h
 0xB4	00-0F	linux/gpio.h		<mailto:linux-gpio@vger.kernel.org>
 0xB5	00-0F	uapi/linux/rpmsg.h	<mailto:linux-remoteproc@vger.kernel.org>
+0xB6	all	linux/fpga-dfl.h
 0xC0	00-0F	linux/usb/iowarrior.h
 0xCA	00-0F	uapi/misc/cxl.h
 0xCA	80-BF	uapi/scsi/cxlflash_ioctl.h
diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index d17c66a..b4e0b54 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c
@@ -18,6 +18,7 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/fpga-dfl.h>
 
 #include "fpga-dfl.h"
 
@@ -108,6 +109,13 @@ static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
 	},
 };
 
+static long fme_ioctl_check_extension(struct feature_platform_data *pdata,
+				      unsigned long arg)
+{
+	/* No extension support for now */
+	return 0;
+}
+
 static int fme_open(struct inode *inode, struct file *filp)
 {
 	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
@@ -148,6 +156,10 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
 
 	switch (cmd) {
+	case FPGA_GET_API_VERSION:
+		return FPGA_API_VERSION;
+	case FPGA_CHECK_EXTENSION:
+		return fme_ioctl_check_extension(pdata, arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
new file mode 100644
index 0000000..b46d124
--- /dev/null
+++ b/include/uapi/linux/fpga-dfl.h
@@ -0,0 +1,50 @@
+/*
+ * Header File for FPGA DFL User API
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#ifndef _UAPI_LINUX_FPGA_DFL_H
+#define _UAPI_LINUX_FPGA_DFL_H
+
+#define FPGA_API_VERSION 0
+
+/*
+ * The IOCTL interface for DFL based FPGA is designed for extensibility by
+ * embedding the structure length (argsz) and flags into structures passed
+ * between kernel and userspace. This design referenced the VFIO IOCTL
+ * interface (include/uapi/linux/vfio.h).
+ */
+
+#define FPGA_MAGIC 0xB6
+
+#define FPGA_BASE 0
+
+/**
+ * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
+ *
+ * Report the version of the driver API.
+ * Return: Driver API Version.
+ */
+
+#define FPGA_GET_API_VERSION	_IO(FPGA_MAGIC, FPGA_BASE + 0)
+
+/**
+ * FPGA_CHECK_EXTENSION - _IO(FPGA_MAGIC, FPGA_BASE + 1)
+ *
+ * Check whether an extension is supported.
+ * Return: 0 if not supported, otherwise the extension is supported.
+ */
+
+#define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
+
+#endif /* _UAPI_LINUX_FPGA_DFL_H */
-- 
1.8.3.1

Re: [PATCH v3 12/21] fpga: dfl: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

One fix again, otherwise please add my ack to subsequent versions.

> FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> need to be supported by all feature devices drivers including FME and AFU.
> Userspace application can use these ioctl interfaces to get the API info
> and check if specific extension is supported or not in current driver.
>
> This patch implements above 2 ioctls in FPGA Management Engine (FME)
> driver.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>

Acked-by: Alan Tull <atull@kernel.org>

> diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
> new file mode 100644
> index 0000000..b46d124
> --- /dev/null
> +++ b/include/uapi/linux/fpga-dfl.h
> @@ -0,0 +1,50 @@
> +/*
> + * Header File for FPGA DFL User API
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2.

Please remove above line.  My understanding is that we are supposed to
not combine SPDX with other license statements.

> + * SPDX-License-Identifier: GPL-2.0
> + */

// SPDX-License-Identifier: GPL-2.0

As first line of file.

Thanks,
Alan

Re: [PATCH v3 12/21] fpga: dfl: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Wu Hao]

On Wed, Jan 31, 2018 at 09:31:59AM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> One fix again, otherwise please add my ack to subsequent versions.

Sure. Thanks for the review.

> 
> > FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> > need to be supported by all feature devices drivers including FME and AFU.
> > Userspace application can use these ioctl interfaces to get the API info
> > and check if specific extension is supported or not in current driver.
> >
> > This patch implements above 2 ioctls in FPGA Management Engine (FME)
> > driver.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> 
> Acked-by: Alan Tull <atull@kernel.org>
> 
> > diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
> > new file mode 100644
> > index 0000000..b46d124
> > --- /dev/null
> > +++ b/include/uapi/linux/fpga-dfl.h
> > @@ -0,0 +1,50 @@
> > +/*
> > + * Header File for FPGA DFL User API
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2.
> 
> Please remove above line.  My understanding is that we are supposed to
> not combine SPDX with other license statements.
> 
> > + * SPDX-License-Identifier: GPL-2.0
> > + */
> 
> // SPDX-License-Identifier: GPL-2.0
> 
> As first line of file.

Sure, I will fix this in the next version.

Thanks
Hao

> 
> Thanks,
> Alan

Re: [PATCH v3 12/21] fpga: dfl: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Moritz Fischer]

[-- Attachment #1: Type: text/plain, Size: 2287 bytes --]

On Thu, Feb 01, 2018 at 01:11:07PM +0800, Wu Hao wrote:
> On Wed, Jan 31, 2018 at 09:31:59AM -0600, Alan Tull wrote:
> > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > 
> > Hi Hao,
> > 
> > One fix again, otherwise please add my ack to subsequent versions.
> 
> Sure. Thanks for the review.
> 
> > 
> > > FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> > > need to be supported by all feature devices drivers including FME and AFU.
> > > Userspace application can use these ioctl interfaces to get the API info
> > > and check if specific extension is supported or not in current driver.
> > >
> > > This patch implements above 2 ioctls in FPGA Management Engine (FME)
> > > driver.
> > >
> > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > 
> > Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
> > 
> > > diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
> > > new file mode 100644
> > > index 0000000..b46d124
> > > --- /dev/null
> > > +++ b/include/uapi/linux/fpga-dfl.h
> > > @@ -0,0 +1,50 @@
> > > +/*
> > > + * Header File for FPGA DFL User API
> > > + *
> > > + * Copyright (C) 2017 Intel Corporation, Inc.
> > > + *
> > > + * Authors:
> > > + *   Kang Luwei <luwei.kang@intel.com>
> > > + *   Zhang Yi <yi.z.zhang@intel.com>
> > > + *   Wu Hao <hao.wu@intel.com>
> > > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > + *
> > > + * This work is licensed under the terms of the GNU GPL version 2.
> > 
> > Please remove above line.  My understanding is that we are supposed to
> > not combine SPDX with other license statements.
> > 
> > > + * SPDX-License-Identifier: GPL-2.0
> > > + */
> > 
> > // SPDX-License-Identifier: GPL-2.0
> > 
> > As first line of file.
> 
> Sure, I will fix this in the next version.
> 
> Thanks
> Hao
> 
> > 
> > Thanks,
> > Alan

Thanks,

Moritz

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v3 13/21] fpga: dfl: fme: add partial reconfiguration sub feature support

[Wu Hao]

From: Kang Luwei <luwei.kang@intel.com>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

It creates platform devices for fpga-mgr, fpga-regions and fpga-bridges,
and invokes fpga-region's interface (fpga_region_program_fpga) for PR
operation once PR request received via ioctl. Below user space interface
is exposed by this sub feature.

Ioctl interface:
* FPGA_FME_PORT_PR
  Do partial reconfiguration per information from userspace, including
  target port(AFU), buffer size and address info. It returns error code
  to userspace if failed. For detailed PR error information, user needs
  to read fpga-mgr's status sysfs interface.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    removed status from FPGA_FME_PORT_PR ioctl data structure.
    added platform devices creation for fpga-mgr/fpga-region/fpga-bridge.
    switched to fpga-region interface fpga_region_program_fpga for PR.
    fixed comments from Alan Tull on FPGA_MGR_PARTIAL_RECONFIG flag usage.
    fixed kbuild warnings.
v3: rename driver files to dfl-fme-*.
    rebase due to fpga APIs change.
    replace bitfields.
    switch to fpga_cdev_find_port to find port device.
---
 drivers/fpga/Makefile         |   2 +-
 drivers/fpga/dfl-fme-main.c   |  45 +++-
 drivers/fpga/dfl-fme-pr.c     | 492 ++++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl-fme.h        |  57 +++++
 include/uapi/linux/fpga-dfl.h |  27 +++
 5 files changed, 621 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/dfl-fme-pr.c
 create mode 100644 drivers/fpga/dfl-fme.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index e5ba158..cc75bb3 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -32,7 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 obj-$(CONFIG_FPGA_DFL)			+= fpga-dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)		+= fpga-dfl-fme.o
 
-fpga-dfl-fme-objs := dfl-fme-main.o
+fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_INTEL_FPGA_DFL_PCI)	+= intel-dfl-pci.o
diff --git a/drivers/fpga/dfl-fme-main.c b/drivers/fpga/dfl-fme-main.c
index b4e0b54..13154d1 100644
--- a/drivers/fpga/dfl-fme-main.c
+++ b/drivers/fpga/dfl-fme-main.c
@@ -21,6 +21,7 @@
 #include <linux/fpga-dfl.h>
 
 #include "fpga-dfl.h"
+#include "dfl-fme.h"
 
 static ssize_t ports_num_show(struct device *dev,
 			      struct device_attribute *attr, char *buf)
@@ -105,6 +106,10 @@ static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
 		.ops = &fme_hdr_ops,
 	},
 	{
+		.name = FME_FEATURE_PR_MGMT,
+		.ops = &pr_mgmt_ops,
+	},
+	{
 		.ops = NULL,
 	},
 };
@@ -188,14 +193,49 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	.unlocked_ioctl = fme_ioctl,
 };
 
+static int fme_dev_init(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+	if (!fme)
+		return -ENOMEM;
+
+	fme->pdata = pdata;
+
+	mutex_lock(&pdata->lock);
+	fpga_pdata_set_private(pdata, fme);
+	mutex_unlock(&pdata->lock);
+
+	return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	fpga_pdata_set_private(pdata, NULL);
+	mutex_unlock(&pdata->lock);
+
+	devm_kfree(&pdev->dev, fme);
+}
+
 static int fme_probe(struct platform_device *pdev)
 {
 	int ret;
 
-	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	ret = fme_dev_init(pdev);
 	if (ret)
 		goto exit;
 
+	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	if (ret)
+		goto dev_destroy;
+
 	ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
 	if (ret)
 		goto feature_uinit;
@@ -204,6 +244,8 @@ static int fme_probe(struct platform_device *pdev)
 
 feature_uinit:
 	fpga_dev_feature_uinit(pdev);
+dev_destroy:
+	fme_dev_destroy(pdev);
 exit:
 	return ret;
 }
@@ -212,6 +254,7 @@ static int fme_remove(struct platform_device *pdev)
 {
 	fpga_dev_feature_uinit(pdev);
 	fpga_unregister_dev_ops(pdev);
+	fme_dev_destroy(pdev);
 
 	return 0;
 }
diff --git a/drivers/fpga/dfl-fme-pr.c b/drivers/fpga/dfl-fme-pr.c
new file mode 100644
index 0000000..2ed9ece
--- /dev/null
+++ b/drivers/fpga/dfl-fme-pr.c
@@ -0,0 +1,492 @@
+/*
+ * Driver for FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/fpga-dfl.h>
+
+#include "fpga-dfl.h"
+#include "dfl-fme.h"
+
+static struct fme_region *
+find_fme_region_by_port_id(struct fpga_fme *fme, int port_id)
+{
+	struct fme_region *fme_region;
+
+	list_for_each_entry(fme_region, &fme->region_list, node)
+		if (fme_region->port_id == port_id)
+			return fme_region;
+
+	return NULL;
+}
+
+static int fpga_fme_region_match(struct device *dev, const void *data)
+{
+	return dev->parent == data;
+}
+
+static struct fpga_region *
+fpga_fme_region_find(struct fpga_fme *fme, int port_id)
+{
+	struct fme_region *fme_region;
+	struct fpga_region *region;
+
+	fme_region = find_fme_region_by_port_id(fme, port_id);
+	if (!fme_region)
+		return NULL;
+
+	region = fpga_region_class_find(NULL, &fme_region->region->dev,
+					fpga_fme_region_match);
+	if (!region)
+		return NULL;
+
+	return region;
+}
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+	void __user *argp = (void __user *)arg;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+	struct fpga_image_info *info;
+	struct fpga_region *region;
+	struct fpga_fme_port_pr port_pr;
+	unsigned long minsz;
+	void __iomem *fme_hdr;
+	void *buf = NULL;
+	int ret = 0;
+	u64 v;
+
+	minsz = offsetofend(struct fpga_fme_port_pr, buffer_address);
+
+	if (copy_from_user(&port_pr, argp, minsz))
+		return -EFAULT;
+
+	if (port_pr.argsz < minsz || port_pr.flags)
+		return -EINVAL;
+
+	if (!IS_ALIGNED(port_pr.buffer_size, 4))
+		return -EINVAL;
+
+	/* get fme header region */
+	fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					      FME_FEATURE_ID_HEADER);
+	if (WARN_ON(!fme_hdr))
+		return -EINVAL;
+
+	/* check port id */
+	v = readq(fme_hdr + FME_HDR_CAP);
+	if (port_pr.port_id >= FIELD_GET(FME_CAP_NUM_PORTS, v)) {
+		dev_dbg(&pdev->dev, "port number more than maximum\n");
+		return -EINVAL;
+	}
+
+	if (!access_ok(VERIFY_READ,
+		       (void __user *)(unsigned long)port_pr.buffer_address,
+		       port_pr.buffer_size))
+		return -EFAULT;
+
+	buf = vmalloc(port_pr.buffer_size);
+	if (!buf)
+		return -ENOMEM;
+
+	if (copy_from_user(buf,
+			   (void __user *)(unsigned long)port_pr.buffer_address,
+			   port_pr.buffer_size)) {
+		ret = -EFAULT;
+		goto free_exit;
+	}
+
+	/* prepare fpga_image_info for PR */
+	info = fpga_image_info_alloc(&pdev->dev);
+	if (!info) {
+		ret = -ENOMEM;
+		goto free_exit;
+	}
+
+	info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	/* fme device has been unregistered. */
+	if (!fme) {
+		ret = -EINVAL;
+		goto unlock_exit;
+	}
+
+	region = fpga_fme_region_find(fme, port_pr.port_id);
+	if (!region) {
+		ret = -EINVAL;
+		goto unlock_exit;
+	}
+
+	fpga_image_info_free(region->info);
+
+	info->buf = buf;
+	info->count = port_pr.buffer_size;
+	info->region_id = port_pr.port_id;
+	region->info = info;
+
+	ret = fpga_region_program_fpga(region);
+
+	if (region->get_bridges)
+		fpga_bridges_put(&region->bridge_list);
+
+	put_device(&region->dev);
+unlock_exit:
+	mutex_unlock(&pdata->lock);
+free_exit:
+	vfree(buf);
+	if (copy_to_user((void __user *)arg, &port_pr, minsz))
+		return -EFAULT;
+
+	return ret;
+}
+
+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+	struct platform_device *mgr, *fme = pdata->dev;
+	struct resource res;
+	struct resource *pres;
+	int ret = -ENOMEM;
+
+	/*
+	 * Each FME has only one fpga-mgr, so allocate platform device using
+	 * the same FME platform device id.
+	 */
+	mgr = platform_device_alloc(FPGA_DFL_FME_MGR, fme->id);
+	if (!mgr)
+		return ERR_PTR(ret);
+
+	mgr->dev.parent = &fme->dev;
+
+	pres = platform_get_resource_byname(fme, IORESOURCE_MEM,
+					    FME_FEATURE_PR_MGMT);
+	if (!pres) {
+		ret = -ENODEV;
+		goto create_mgr_err;
+	}
+
+	memset(&res, 0, sizeof(struct resource));
+
+	res.start = pres->start;
+	res.end = pres->end;
+	res.name = pres->name;
+	res.flags = IORESOURCE_MEM;
+
+	ret = platform_device_add_resources(mgr, &res, 1);
+	if (ret)
+		goto create_mgr_err;
+
+	ret = platform_device_add(mgr);
+	if (ret)
+		goto create_mgr_err;
+
+	return mgr;
+
+create_mgr_err:
+	platform_device_put(mgr);
+	return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_mgr - destroy fpga mgr platform device
+ *
+ * @mgr: fpga mgr platform device to be destroy
+ */
+static void fpga_fme_destroy_mgr(struct feature_platform_data *pdata)
+{
+	struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+
+	platform_device_unregister(priv->mgr);
+}
+
+/**
+ * fpga_fme_create_bridge - create fme fpga bridge platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @port_id: port id for the bridge to be created.
+ *
+ * Return: bridge platform device if successful, and error code otherwise.
+ */
+static struct fme_bridge *
+fpga_fme_create_bridge(struct feature_platform_data *pdata, int port_id)
+{
+	struct device *dev = &pdata->dev->dev;
+	struct fme_br_pdata br_pdata;
+	struct fme_bridge *fme_br;
+	int ret = -ENOMEM;
+
+	fme_br = devm_kzalloc(dev, sizeof(*fme_br), GFP_KERNEL);
+	if (!fme_br)
+		return ERR_PTR(ret);
+
+	br_pdata.port = fpga_cdev_find_port(fpga_pdata_to_fpga_cdev(pdata),
+					    &port_id, fpga_port_check_id);
+	if (!br_pdata.port)
+		return ERR_PTR(-ENODEV);
+
+	/*
+	 * Each FPGA device may have more than one port, so allocate platform
+	 * device using the same port platform device id.
+	 */
+	fme_br->br = platform_device_alloc(FPGA_DFL_FME_BRIDGE,
+					   br_pdata.port->id);
+	if (!fme_br->br) {
+		ret = -ENOMEM;
+		goto create_br_err;
+	}
+
+	fme_br->br->dev.parent = dev;
+
+	ret = platform_device_add_data(fme_br->br, &br_pdata, sizeof(br_pdata));
+	if (ret)
+		goto create_br_err;
+
+	ret = platform_device_add(fme_br->br);
+	if (ret)
+		goto create_br_err;
+
+	return fme_br;
+
+create_br_err:
+	platform_device_put(fme_br->br);
+	put_device(&br_pdata.port->dev);
+	return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy fpga bridge platform device
+ * @br: fpga bridge platform device to be destroy
+ */
+static void fpga_fme_destroy_bridge(struct feature_platform_data *pdata,
+				    struct fme_bridge *fme_br)
+{
+	struct fme_br_pdata *br_pdata = dev_get_platdata(&fme_br->br->dev);
+
+	put_device(&br_pdata->port->dev);
+	platform_device_unregister(fme_br->br);
+}
+
+static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
+{
+	struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+	struct fme_bridge *fbridge, *tmp;
+
+	list_for_each_entry_safe(fbridge, tmp, &priv->bridge_list, node) {
+		list_del(&fbridge->node);
+		fpga_fme_destroy_bridge(pdata, fbridge);
+	}
+}
+
+/**
+ * fpga_fme_create_region - create fpga region platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @mgr: mgr platform device needed for region
+ * @br: br platform device needed for region
+ * @port_id: port id
+ *
+ * Return: fme region if successful, and error code otherwise.
+ */
+static struct fme_region *
+fpga_fme_create_region(struct feature_platform_data *pdata,
+		       struct platform_device *mgr,
+		       struct platform_device *br, int port_id)
+{
+	struct device *dev = &pdata->dev->dev;
+	struct fme_region_pdata region_pdata;
+	struct fme_region *fme_region;
+	int ret = -ENOMEM;
+
+	fme_region = devm_kzalloc(dev, sizeof(*fme_region), GFP_KERNEL);
+	if (!fme_region)
+		return ERR_PTR(ret);
+
+	region_pdata.mgr = mgr;
+	region_pdata.br = br;
+
+	/*
+	 * Each FPGA device may have more than one port, so allocate platform
+	 * device using the same port platform device id.
+	 */
+	fme_region->region = platform_device_alloc(FPGA_DFL_FME_REGION, br->id);
+	if (!fme_region->region)
+		return ERR_PTR(ret);
+
+	fme_region->region->dev.parent = dev;
+
+	ret = platform_device_add_data(fme_region->region, &region_pdata,
+				       sizeof(region_pdata));
+	if (ret)
+		goto create_region_err;
+
+	ret = platform_device_add(fme_region->region);
+	if (ret)
+		goto create_region_err;
+
+	fme_region->port_id = port_id;
+
+	return fme_region;
+
+create_region_err:
+	platform_device_put(fme_region->region);
+	return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_region - destroy fme region
+ * @region: fme region to be destroy
+ */
+static void fpga_fme_destroy_region(struct feature_platform_data *pdata,
+				    struct fme_region *fme_region)
+{
+	platform_device_unregister(fme_region->region);
+}
+
+static void fpga_fme_destroy_regions(struct feature_platform_data *pdata)
+{
+	struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+	struct fme_region *fme_region, *tmp;
+
+	list_for_each_entry_safe(fme_region, tmp, &priv->region_list, node) {
+		list_del(&fme_region->node);
+		fpga_fme_destroy_region(pdata, fme_region);
+	}
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	void __iomem *fme_hdr;
+	struct platform_device *mgr;
+	struct fme_region *fme_region;
+	struct fme_bridge *fme_br;
+	struct fpga_fme *priv;
+	int ret = -ENODEV, i = 0;
+	u64 fme_cap, port_offset;
+
+	fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					      FME_FEATURE_ID_HEADER);
+	if (WARN_ON(!fme_hdr))
+		return -EINVAL;
+
+	mutex_lock(&pdata->lock);
+	priv = fpga_pdata_get_private(pdata);
+
+	/* Initialize the region and bridge sub device list */
+	INIT_LIST_HEAD(&priv->region_list);
+	INIT_LIST_HEAD(&priv->bridge_list);
+
+	/* Create fpga mgr platform device */
+	mgr = fpga_fme_create_mgr(pdata);
+	if (IS_ERR(mgr)) {
+		dev_err(&pdev->dev, "fail to create fpga mgr pdev\n");
+		goto unlock;
+	}
+
+	priv->mgr = mgr;
+
+	/* Read capability register to check number of regions and bridges */
+	fme_cap = readq(fme_hdr + FME_HDR_CAP);
+	for (; i < FIELD_GET(FME_CAP_NUM_PORTS, fme_cap); i++) {
+		port_offset = readq(fme_hdr + FME_HDR_PORT_OFST(i));
+		if (!(port_offset & FME_PORT_OFST_IMP))
+			continue;
+
+		/* Create bridge for each port */
+		fme_br = fpga_fme_create_bridge(pdata, i);
+		if (IS_ERR(fme_br)) {
+			ret = PTR_ERR(fme_br);
+			goto destroy_region;
+		}
+
+		list_add(&fme_br->node, &priv->bridge_list);
+
+		/* Create region for each port */
+		fme_region = fpga_fme_create_region(pdata, mgr, fme_br->br, i);
+		if (!fme_region) {
+			ret = PTR_ERR(fme_region);
+			goto destroy_region;
+		}
+
+		list_add(&fme_region->node, &priv->region_list);
+	}
+	mutex_unlock(&pdata->lock);
+
+	return 0;
+
+destroy_region:
+	fpga_fme_destroy_regions(pdata);
+	fpga_fme_destroy_bridges(pdata);
+	fpga_fme_destroy_mgr(pdata);
+unlock:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *priv;
+
+	mutex_lock(&pdata->lock);
+	priv = fpga_pdata_get_private(pdata);
+
+	fpga_fme_destroy_regions(pdata);
+	fpga_fme_destroy_bridges(pdata);
+	fpga_fme_destroy_mgr(pdata);
+	mutex_unlock(&pdata->lock);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+			 unsigned int cmd, unsigned long arg)
+{
+	long ret;
+
+	switch (cmd) {
+	case FPGA_FME_PORT_PR:
+		ret = fme_pr(pdev, arg);
+		break;
+	default:
+		ret = -ENODEV;
+	}
+
+	return ret;
+}
+
+const struct feature_ops pr_mgmt_ops = {
+	.init = pr_mgmt_init,
+	.uinit = pr_mgmt_uinit,
+	.ioctl = fme_pr_ioctl,
+};
diff --git a/drivers/fpga/dfl-fme.h b/drivers/fpga/dfl-fme.h
new file mode 100644
index 0000000..88d6dfd
--- /dev/null
+++ b/drivers/fpga/dfl-fme.h
@@ -0,0 +1,57 @@
+/*
+ * Header file for FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#ifndef __DFL_FME_H
+#define __DFL_FME_H
+
+#define FPGA_DFL_FME_MGR		"fpga-dfl-fme-mgr"
+#define FPGA_DFL_FME_BRIDGE		"fpga-dfl-fme-bridge"
+#define FPGA_DFL_FME_REGION		"fpga-dfl-fme-region"
+
+struct fpga_fme {
+	struct platform_device *mgr;
+	struct list_head region_list;
+	struct list_head bridge_list;
+	struct feature_platform_data *pdata;
+};
+
+struct fme_region {
+	struct platform_device *region;
+	struct list_head node;
+	int port_id;
+};
+
+struct fme_region_pdata {
+	struct platform_device *mgr;
+	struct platform_device *br;
+	int region_id;
+};
+
+struct fme_bridge {
+	struct platform_device *br;
+	struct list_head node;
+};
+
+struct fme_br_pdata {
+	struct platform_device *port;
+};
+
+extern const struct feature_ops pr_mgmt_ops;
+
+#endif /* __DFL_FME_H */
diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index b46d124..75bdf88 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h
@@ -16,6 +16,8 @@
 #ifndef _UAPI_LINUX_FPGA_DFL_H
 #define _UAPI_LINUX_FPGA_DFL_H
 
+#include <linux/types.h>
+
 #define FPGA_API_VERSION 0
 
 /*
@@ -28,6 +30,7 @@
 #define FPGA_MAGIC 0xB6
 
 #define FPGA_BASE 0
+#define FME_BASE 0x80
 
 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
@@ -47,4 +50,28 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOW(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error info
+ * from the status of FME's fpga manager.
+ */
+
+struct fpga_fme_port_pr {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u32 port_id;
+	__u32 buffer_size;
+	__u64 buffer_address;	/* Userspace address to the buffer for PR */
+};
+
+#define FPGA_FME_PORT_PR	_IO(FPGA_MAGIC, FME_BASE + 0)
+
 #endif /* _UAPI_LINUX_FPGA_DFL_H */
-- 
1.8.3.1

[PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Wu Hao]

This patch adds fpga manager driver for FPGA Management Engine (FME). It
implements fpga_manager_ops for FPGA Partial Reconfiguration function.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: rename driver to dfl-fpga-fme-mgr
    implemented status callback for fpga manager
    rebased due to fpga api changes
---
 .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
 drivers/fpga/Kconfig                               |   6 +
 drivers/fpga/Makefile                              |   1 +
 drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
 drivers/fpga/fpga-dfl.h                            |  39 ++-
 5 files changed, 371 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
 create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c

diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
new file mode 100644
index 0000000..2d4f917
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
@@ -0,0 +1,8 @@
+What:		/sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
+Date:		November 2017
+KernelVersion:  4.15
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. It returns interface id of partial reconfiguration
+		hardware. Userspace could use this information to check if
+		current hardware is compatible with given image before FPGA
+		programming.
diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 57da904..0171ecb 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -150,6 +150,12 @@ config FPGA_DFL_FME
 	  FPGA platform level management features. There shall be 1 FME
 	  per DFL based FPGA device.
 
+config FPGA_DFL_FME_MGR
+	tristate "FPGA DFL FME Manager Driver"
+	depends on FPGA_DFL_FME
+	help
+	  Say Y to enable FPGA Manager driver for FPGA Management Engine.
+
 config INTEL_FPGA_DFL_PCI
 	tristate "Intel FPGA DFL PCIe Device Driver"
 	depends on PCI && FPGA_DFL
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index cc75bb3..6378580 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 # FPGA Device Feature List Support
 obj-$(CONFIG_FPGA_DFL)			+= fpga-dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)		+= fpga-dfl-fme.o
+obj-$(CONFIG_FPGA_DFL_FME_MGR)		+= fpga-dfl-fme-mgr.o
 
 fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
 
diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
new file mode 100644
index 0000000..70356ce
--- /dev/null
+++ b/drivers/fpga/fpga-dfl-fme-mgr.c
@@ -0,0 +1,318 @@
+/*
+ * FPGA Manager Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/module.h>
+#include <linux/iopoll.h>
+#include <linux/fpga/fpga-mgr.h>
+
+#include "fpga-dfl.h"
+#include "dfl-fme.h"
+
+#define PR_WAIT_TIMEOUT   8000000
+#define PR_HOST_STATUS_IDLE	0
+
+struct fme_mgr_priv {
+	void __iomem *ioaddr;
+	u64 pr_error;
+};
+
+static ssize_t interface_id_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	struct fpga_manager *mgr = dev_get_drvdata(dev);
+	struct fme_mgr_priv *priv = mgr->priv;
+	u64 intfc_id_l, intfc_id_h;
+
+	intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
+	intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
+
+	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
+			(unsigned long long)intfc_id_h,
+			(unsigned long long)intfc_id_l);
+}
+static DEVICE_ATTR_RO(interface_id);
+
+static const struct attribute *fme_mgr_attrs[] = {
+	&dev_attr_interface_id.attr,
+	NULL,
+};
+
+static u64 pr_error_to_mgr_status(u64 err)
+{
+	u64 status = 0;
+
+	if (err & FME_PR_ERR_OPERATION_ERR)
+		status |= FPGA_MGR_STATUS_OPERATION_ERR;
+	if (err & FME_PR_ERR_CRC_ERR)
+		status |= FPGA_MGR_STATUS_CRC_ERR;
+	if (err & FME_PR_ERR_INCOMPATIBLE_BS)
+		status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
+	if (err & FME_PR_ERR_PROTOCOL_ERR)
+		status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
+	if (err & FME_PR_ERR_FIFO_OVERFLOW)
+		status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
+
+	return status;
+}
+
+static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
+{
+	u64 pr_status, pr_error;
+
+	pr_status = readq(fme_pr + FME_PR_STS);
+	if (!(pr_status & FME_PR_STS_PR_STS))
+		return 0;
+
+	pr_error = readq(fme_pr + FME_PR_ERR);
+	writeq(pr_error, fme_pr + FME_PR_ERR);
+
+	return pr_error;
+}
+
+static int fme_mgr_write_init(struct fpga_manager *mgr,
+			      struct fpga_image_info *info,
+			      const char *buf, size_t count)
+{
+	struct device *dev = &mgr->dev;
+	struct fme_mgr_priv *priv = mgr->priv;
+	void __iomem *fme_pr = priv->ioaddr;
+	u64 pr_ctrl, pr_status;
+
+	if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
+		dev_err(dev, "only support partial reconfiguration.\n");
+		return -EINVAL;
+	}
+
+	dev_dbg(dev, "resetting PR before initiated PR\n");
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl |= FME_PR_CTRL_PR_RST;
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+			       pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
+			       PR_WAIT_TIMEOUT)) {
+		dev_err(dev, "maximum PR timeout\n");
+		return -ETIMEDOUT;
+	}
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl &= ~FME_PR_CTRL_PR_RST;
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	dev_dbg(dev,
+		"waiting for PR resource in HW to be initialized and ready\n");
+
+	if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
+			       (pr_status & FME_PR_STS_PR_STS) ==
+			       FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
+		dev_err(dev, "maximum PR timeout\n");
+		priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(dev, "check and clear previous PR error\n");
+	priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+	if (priv->pr_error)
+		dev_dbg(dev, "previous PR error detected %llx\n",
+			(unsigned long long)priv->pr_error);
+
+	dev_dbg(dev, "set PR port ID\n");
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
+	pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	return 0;
+}
+
+static int fme_mgr_write(struct fpga_manager *mgr,
+			 const char *buf, size_t count)
+{
+	struct device *dev = &mgr->dev;
+	struct fme_mgr_priv *priv = mgr->priv;
+	void __iomem *fme_pr = priv->ioaddr;
+	u64 pr_ctrl, pr_status, pr_data;
+	int delay = 0, pr_credit, i = 0;
+
+	dev_dbg(dev, "start request\n");
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl |= FME_PR_CTRL_PR_START;
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	dev_dbg(dev, "pushing data from bitstream to HW\n");
+
+	pr_status = readq(fme_pr + FME_PR_STS);
+	pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+
+	while (count > 0) {
+		while (pr_credit <= 1) {
+			if (delay++ > PR_WAIT_TIMEOUT) {
+				dev_err(dev, "maximum try\n");
+				return -ETIMEDOUT;
+			}
+			udelay(1);
+
+			pr_status = readq(fme_pr + FME_PR_STS);
+			pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
+		}
+
+		if (count >= 4) {
+			pr_data = 0;
+			pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
+					      *(((u32 *)buf) + i));
+			writeq(pr_data, fme_pr + FME_PR_DATA);
+			count -= 4;
+			pr_credit--;
+			i++;
+		} else {
+			WARN_ON(1);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int fme_mgr_write_complete(struct fpga_manager *mgr,
+				  struct fpga_image_info *info)
+{
+	struct device *dev = &mgr->dev;
+	struct fme_mgr_priv *priv = mgr->priv;
+	void __iomem *fme_pr = priv->ioaddr;
+	u64 pr_ctrl;
+
+	pr_ctrl = readq(fme_pr + FME_PR_CTRL);
+	pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
+	writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
+
+	dev_dbg(dev, "green bitstream push complete\n");
+	dev_dbg(dev, "waiting for HW to release PR resource\n");
+
+	if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
+			       !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
+			       PR_WAIT_TIMEOUT)) {
+		dev_err(dev, "maximum try.\n");
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(dev, "PR operation complete, checking status\n");
+	priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
+	if (priv->pr_error) {
+		dev_dbg(dev, "PR error detected %llx\n",
+			(unsigned long long)priv->pr_error);
+		return -EIO;
+	}
+
+	dev_dbg(dev, "PR done successfully\n");
+
+	return 0;
+}
+
+static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
+{
+	return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static u64 fme_mgr_status(struct fpga_manager *mgr)
+{
+	struct fme_mgr_priv *priv = mgr->priv;
+
+	return pr_error_to_mgr_status(priv->pr_error);
+}
+
+static const struct fpga_manager_ops fme_mgr_ops = {
+	.write_init = fme_mgr_write_init,
+	.write = fme_mgr_write,
+	.write_complete = fme_mgr_write_complete,
+	.state = fme_mgr_state,
+	.status = fme_mgr_status,
+};
+
+static int fme_mgr_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct fme_mgr_priv *priv;
+	struct fpga_manager *mgr;
+	struct resource *res;
+	int ret;
+
+	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+	priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
+	if (IS_ERR(priv->ioaddr))
+		return PTR_ERR(priv->ioaddr);
+
+	ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
+	if (ret)
+		return ret;
+
+	mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
+	if (!mgr)
+		goto sysfs_remove_exit;
+
+	mgr->name = "DFL FPGA Manager";
+	mgr->mops = &fme_mgr_ops;
+	mgr->priv = priv;
+	mgr->parent = dev;
+	platform_set_drvdata(pdev, mgr);
+
+	ret = fpga_mgr_register(mgr);
+	if (ret) {
+		dev_err(dev, "unable to register FPGA manager\n");
+		goto sysfs_remove_exit;
+	}
+
+	return 0;
+
+sysfs_remove_exit:
+	sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
+	return ret;
+}
+
+static int fme_mgr_remove(struct platform_device *pdev)
+{
+	struct fpga_manager *mgr = platform_get_drvdata(pdev);
+
+	fpga_mgr_unregister(mgr);
+	sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
+
+	return 0;
+}
+
+static struct platform_driver fme_mgr_driver = {
+	.driver	= {
+		.name    = FPGA_DFL_FME_MGR,
+	},
+	.probe   = fme_mgr_probe,
+	.remove  = fme_mgr_remove,
+};
+
+module_platform_driver(fme_mgr_driver);
+
+MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
index e5a1094..d45eb82 100644
--- a/drivers/fpga/fpga-dfl.h
+++ b/drivers/fpga/fpga-dfl.h
@@ -130,7 +130,44 @@
 
 /* FME Partial Reconfiguration Sub Feature Register Set */
 #define FME_PR_DFH		DFH
-#define FME_PR_SIZE		DFH_SIZE
+#define FME_PR_CTRL		0x8
+#define FME_PR_STS		0x10
+#define FME_PR_DATA		0x18
+#define FME_PR_ERR		0x20
+#define FME_PR_INTFC_ID_H	0xA8
+#define FME_PR_INTFC_ID_L	0xB0
+#define FME_PR_SIZE		0xB8
+
+/* FME PR Control Register Bitfield */
+#define FME_PR_CTRL_PR_RST	BIT(0)	/* Reset PR engine */
+#define FME_PR_CTRL_PR_RSTACK	BIT(4)	/* Ack for PR engine reset */
+#define FME_PR_CTRL_PR_RGN_ID	GENMASK_ULL(9, 7)	/* PR Region ID */
+#define FME_PR_CTRL_PR_START	BIT(12)	/* Start to request for PR service */
+#define FME_PR_CTRL_PR_COMPLETE	BIT(13)	/* PR data push complete notification */
+
+/* FME PR Status Register Bitfield */
+/* Number of available entries in HW queue inside the PR engine. */
+#define FME_PR_STS_PR_CREDIT	GENMASK_ULL(8, 0)
+#define FME_PR_STS_PR_STS	BIT(16)	/* PR operation status */
+#define FME_PR_STS_PR_STS_IDLE	0
+#define FME_PR_STS_PR_CTRLR_STS	GENMASK_ULL(22, 20)	/* Controller status */
+#define FME_PR_STS_PR_HOST_STS	GENMASK_ULL(27, 24)	/* PR host status */
+
+/* FME PR Data Register Bitfield */
+/* PR data from the raw-binary file. */
+#define FME_PR_DATA_PR_DATA_RAW	GENMASK_ULL(32, 0)
+
+/* FME PR Error Register */
+/* Previous PR Operation errors detected. */
+#define FME_PR_ERR_OPERATION_ERR	BIT(0)
+/* CRC error detected. */
+#define FME_PR_ERR_CRC_ERR		BIT(1)
+/* Incompatible PR bitstream detected. */
+#define FME_PR_ERR_INCOMPATIBLE_BS	BIT(2)
+/* PR data push protocol violated. */
+#define FME_PR_ERR_PROTOCOL_ERR		BIT(3)
+/* PR data fifo overflow error detected */
+#define FME_PR_ERR_FIFO_OVERFLOW	BIT(4)
 
 /* FME HSSI Sub Feature Register Set */
 #define FME_HSSI_DFH		DFH
-- 
1.8.3.1

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

A few comments below.   Besides that, looks good.

> This patch adds fpga manager driver for FPGA Management Engine (FME). It
> implements fpga_manager_ops for FPGA Partial Reconfiguration function.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v3: rename driver to dfl-fpga-fme-mgr
>     implemented status callback for fpga manager
>     rebased due to fpga api changes
> ---
>  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
>  drivers/fpga/Kconfig                               |   6 +
>  drivers/fpga/Makefile                              |   1 +
>  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
>  drivers/fpga/fpga-dfl.h                            |  39 ++-
>  5 files changed, 371 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
>
> diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> new file mode 100644
> index 0000000..2d4f917
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> @@ -0,0 +1,8 @@
> +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> +Date:          November 2017
> +KernelVersion:  4.15
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read-only. It returns interface id of partial reconfiguration
> +               hardware. Userspace could use this information to check if
> +               current hardware is compatible with given image before FPGA
> +               programming.

I'm a little confused by this.  I can understand that the PR bitstream
has a dependency on the FPGA's static image, but I don't understand
the dependency of the bistream on the hardware that is used to program
the bitstream to the FPGA.

> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index 57da904..0171ecb 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -150,6 +150,12 @@ config FPGA_DFL_FME
>           FPGA platform level management features. There shall be 1 FME
>           per DFL based FPGA device.
>
> +config FPGA_DFL_FME_MGR
> +       tristate "FPGA DFL FME Manager Driver"
> +       depends on FPGA_DFL_FME
> +       help
> +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
> +
>  config INTEL_FPGA_DFL_PCI
>         tristate "Intel FPGA DFL PCIe Device Driver"
>         depends on PCI && FPGA_DFL
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index cc75bb3..6378580 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
>  # FPGA Device Feature List Support
>  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
>  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
> +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
>
>  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
>
> diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
> new file mode 100644
> index 0000000..70356ce
> --- /dev/null
> +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
> @@ -0,0 +1,318 @@
> +/*
> + * FPGA Manager Driver for FPGA Management Engine (FME)
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Christopher Rauer <christopher.rauer@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2.
> + * SPDX-License-Identifier: GPL-2.0
> + */
> +
> +#include <linux/module.h>
> +#include <linux/iopoll.h>
> +#include <linux/fpga/fpga-mgr.h>
> +
> +#include "fpga-dfl.h"
> +#include "dfl-fme.h"
> +
> +#define PR_WAIT_TIMEOUT   8000000
> +#define PR_HOST_STATUS_IDLE    0
> +
> +struct fme_mgr_priv {
> +       void __iomem *ioaddr;
> +       u64 pr_error;
> +};
> +
> +static ssize_t interface_id_show(struct device *dev,
> +                                struct device_attribute *attr, char *buf)
> +{
> +       struct fpga_manager *mgr = dev_get_drvdata(dev);
> +       struct fme_mgr_priv *priv = mgr->priv;
> +       u64 intfc_id_l, intfc_id_h;
> +
> +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
> +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
> +
> +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> +                       (unsigned long long)intfc_id_h,
> +                       (unsigned long long)intfc_id_l);
> +}
> +static DEVICE_ATTR_RO(interface_id);
> +
> +static const struct attribute *fme_mgr_attrs[] = {
> +       &dev_attr_interface_id.attr,
> +       NULL,
> +};
> +
> +static u64 pr_error_to_mgr_status(u64 err)
> +{
> +       u64 status = 0;
> +
> +       if (err & FME_PR_ERR_OPERATION_ERR)
> +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
> +       if (err & FME_PR_ERR_CRC_ERR)
> +               status |= FPGA_MGR_STATUS_CRC_ERR;
> +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
> +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
> +       if (err & FME_PR_ERR_PROTOCOL_ERR)
> +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
> +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> +
> +       return status;
> +}
> +
> +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
> +{
> +       u64 pr_status, pr_error;
> +
> +       pr_status = readq(fme_pr + FME_PR_STS);
> +       if (!(pr_status & FME_PR_STS_PR_STS))
> +               return 0;
> +
> +       pr_error = readq(fme_pr + FME_PR_ERR);
> +       writeq(pr_error, fme_pr + FME_PR_ERR);
> +
> +       return pr_error;
> +}
> +
> +static int fme_mgr_write_init(struct fpga_manager *mgr,
> +                             struct fpga_image_info *info,
> +                             const char *buf, size_t count)
> +{
> +       struct device *dev = &mgr->dev;
> +       struct fme_mgr_priv *priv = mgr->priv;
> +       void __iomem *fme_pr = priv->ioaddr;
> +       u64 pr_ctrl, pr_status;
> +
> +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> +               dev_err(dev, "only support partial reconfiguration.\n");

supports

> +               return -EINVAL;
> +       }
> +
> +       dev_dbg(dev, "resetting PR before initiated PR\n");
> +
> +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> +       pr_ctrl |= FME_PR_CTRL_PR_RST;
> +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> +
> +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
> +                              PR_WAIT_TIMEOUT)) {
> +               dev_err(dev, "maximum PR timeout\n");

We have two dev_err's with the same "maximum PR timeout" so the user
would not be able to see at which point the timeout occurred.

I suggest to get rid of the word 'maximum' for both.  This one could
be 'reset ack timeout" or something similar.

> +               return -ETIMEDOUT;
> +       }
> +
> +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
> +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> +
> +       dev_dbg(dev,
> +               "waiting for PR resource in HW to be initialized and ready\n");
> +
> +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
> +                              (pr_status & FME_PR_STS_PR_STS) ==
> +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
> +               dev_err(dev, "maximum PR timeout\n");

"PR_STS timeout"?  Or something better.

> +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(dev, "check and clear previous PR error\n");
> +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> +       if (priv->pr_error)
> +               dev_dbg(dev, "previous PR error detected %llx\n",
> +                       (unsigned long long)priv->pr_error);
> +
> +       dev_dbg(dev, "set PR port ID\n");
> +
> +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
> +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
> +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> +
> +       return 0;
> +}
> +
> +static int fme_mgr_write(struct fpga_manager *mgr,
> +                        const char *buf, size_t count)
> +{
> +       struct device *dev = &mgr->dev;
> +       struct fme_mgr_priv *priv = mgr->priv;
> +       void __iomem *fme_pr = priv->ioaddr;
> +       u64 pr_ctrl, pr_status, pr_data;
> +       int delay = 0, pr_credit, i = 0;
> +
> +       dev_dbg(dev, "start request\n");
> +
> +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> +       pr_ctrl |= FME_PR_CTRL_PR_START;
> +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> +
> +       dev_dbg(dev, "pushing data from bitstream to HW\n");
> +
> +       pr_status = readq(fme_pr + FME_PR_STS);
> +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> +
> +       while (count > 0) {
> +               while (pr_credit <= 1) {
> +                       if (delay++ > PR_WAIT_TIMEOUT) {
> +                               dev_err(dev, "maximum try\n");

It looks like this is waiting for an entry in a queue and timing out
here.  Could you add a some comments for pr_credit above and this
loop?  Also a better error message, perhaps "PR credit timeout"?

> +                               return -ETIMEDOUT;
> +                       }
> +                       udelay(1);
> +
> +                       pr_status = readq(fme_pr + FME_PR_STS);
> +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> +               }
> +
> +               if (count >= 4) {
> +                       pr_data = 0;
> +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> +                                             *(((u32 *)buf) + i));
> +                       writeq(pr_data, fme_pr + FME_PR_DATA);
> +                       count -= 4;
> +                       pr_credit--;
> +                       i++;
> +               } else {
> +                       WARN_ON(1);
> +                       return -EINVAL;
> +               }
> +       }
> +
> +       return 0;
> +}
> +
> +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> +                                 struct fpga_image_info *info)
> +{
> +       struct device *dev = &mgr->dev;
> +       struct fme_mgr_priv *priv = mgr->priv;
> +       void __iomem *fme_pr = priv->ioaddr;
> +       u64 pr_ctrl;
> +
> +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
> +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> +
> +       dev_dbg(dev, "green bitstream push complete\n");
> +       dev_dbg(dev, "waiting for HW to release PR resource\n");
> +
> +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
> +                              PR_WAIT_TIMEOUT)) {
> +               dev_err(dev, "maximum try.\n");

Some message specific to here also, please.

> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(dev, "PR operation complete, checking status\n");
> +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> +       if (priv->pr_error) {
> +               dev_dbg(dev, "PR error detected %llx\n",
> +                       (unsigned long long)priv->pr_error);
> +               return -EIO;
> +       }
> +
> +       dev_dbg(dev, "PR done successfully\n");
> +
> +       return 0;
> +}
> +
> +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> +{
> +       return FPGA_MGR_STATE_UNKNOWN;
> +}
> +
> +static u64 fme_mgr_status(struct fpga_manager *mgr)
> +{
> +       struct fme_mgr_priv *priv = mgr->priv;
> +
> +       return pr_error_to_mgr_status(priv->pr_error);
> +}
> +
> +static const struct fpga_manager_ops fme_mgr_ops = {
> +       .write_init = fme_mgr_write_init,
> +       .write = fme_mgr_write,
> +       .write_complete = fme_mgr_write_complete,
> +       .state = fme_mgr_state,
> +       .status = fme_mgr_status,
> +};
> +
> +static int fme_mgr_probe(struct platform_device *pdev)
> +{
> +       struct device *dev = &pdev->dev;
> +       struct fme_mgr_priv *priv;
> +       struct fpga_manager *mgr;
> +       struct resource *res;
> +       int ret;
> +
> +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> +       if (!priv)
> +               return -ENOMEM;
> +
> +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> +       if (IS_ERR(priv->ioaddr))
> +               return PTR_ERR(priv->ioaddr);
> +
> +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> +       if (ret)
> +               return ret;
> +
> +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
> +       if (!mgr)
> +               goto sysfs_remove_exit;
> +
> +       mgr->name = "DFL FPGA Manager";
> +       mgr->mops = &fme_mgr_ops;
> +       mgr->priv = priv;
> +       mgr->parent = dev;
> +       platform_set_drvdata(pdev, mgr);
> +
> +       ret = fpga_mgr_register(mgr);
> +       if (ret) {
> +               dev_err(dev, "unable to register FPGA manager\n");
> +               goto sysfs_remove_exit;
> +       }
> +
> +       return 0;
> +
> +sysfs_remove_exit:
> +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> +       return ret;
> +}
> +
> +static int fme_mgr_remove(struct platform_device *pdev)
> +{
> +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
> +
> +       fpga_mgr_unregister(mgr);
> +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> +
> +       return 0;
> +}
> +
> +static struct platform_driver fme_mgr_driver = {
> +       .driver = {
> +               .name    = FPGA_DFL_FME_MGR,
> +       },
> +       .probe   = fme_mgr_probe,
> +       .remove  = fme_mgr_remove,
> +};
> +
> +module_platform_driver(fme_mgr_driver);
> +
> +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("GPL v2");
> +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
> diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> index e5a1094..d45eb82 100644
> --- a/drivers/fpga/fpga-dfl.h
> +++ b/drivers/fpga/fpga-dfl.h
> @@ -130,7 +130,44 @@
>
>  /* FME Partial Reconfiguration Sub Feature Register Set */
>  #define FME_PR_DFH             DFH
> -#define FME_PR_SIZE            DFH_SIZE
> +#define FME_PR_CTRL            0x8
> +#define FME_PR_STS             0x10
> +#define FME_PR_DATA            0x18
> +#define FME_PR_ERR             0x20
> +#define FME_PR_INTFC_ID_H      0xA8
> +#define FME_PR_INTFC_ID_L      0xB0
> +#define FME_PR_SIZE            0xB8
> +
> +/* FME PR Control Register Bitfield */
> +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
> +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
> +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
> +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
> +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
> +
> +/* FME PR Status Register Bitfield */
> +/* Number of available entries in HW queue inside the PR engine. */
> +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
> +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
> +#define FME_PR_STS_PR_STS_IDLE 0
> +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
> +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
> +
> +/* FME PR Data Register Bitfield */
> +/* PR data from the raw-binary file. */
> +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
> +
> +/* FME PR Error Register */
> +/* Previous PR Operation errors detected. */
> +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
> +/* CRC error detected. */
> +#define FME_PR_ERR_CRC_ERR             BIT(1)
> +/* Incompatible PR bitstream detected. */
> +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
> +/* PR data push protocol violated. */
> +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
> +/* PR data fifo overflow error detected */
> +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
>
>  /* FME HSSI Sub Feature Register Set */
>  #define FME_HSSI_DFH           DFH

I see fpga-dfl.h as enumeration code which is separate from any driver
implementation specifics other than what's required for the DFL
enumeration scheme.    These PR engine #defines should move to a .h
that is dedicated to this specific PR hardware device. If someone else
adds a different PR device to the framework, their PR driver would
also have its own .h.  Same for any other modules that aren't central
to DFL enumeration.

Thanks,
Alan

> --
> 1.8.3.1
>

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Wu Hao]

On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> A few comments below.   Besides that, looks good.
> 
> > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ----
> > v3: rename driver to dfl-fpga-fme-mgr
> >     implemented status callback for fpga manager
> >     rebased due to fpga api changes
> > ---
> >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> >  drivers/fpga/Kconfig                               |   6 +
> >  drivers/fpga/Makefile                              |   1 +
> >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> >  5 files changed, 371 insertions(+), 1 deletion(-)
> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> >
> > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > new file mode 100644
> > index 0000000..2d4f917
> > --- /dev/null
> > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > @@ -0,0 +1,8 @@
> > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> > +Date:          November 2017
> > +KernelVersion:  4.15
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read-only. It returns interface id of partial reconfiguration
> > +               hardware. Userspace could use this information to check if
> > +               current hardware is compatible with given image before FPGA
> > +               programming.
> 
> I'm a little confused by this.  I can understand that the PR bitstream
> has a dependency on the FPGA's static image, but I don't understand
> the dependency of the bistream on the hardware that is used to program
> the bitstream to the FPGA.

Sorry for the confusion, the interface_id is used to indicate the version of
the hardware for partial reconfiguration (it's part of the static image of
the FPGA device). Will improve the description on this.

> 
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > index 57da904..0171ecb 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -150,6 +150,12 @@ config FPGA_DFL_FME
> >           FPGA platform level management features. There shall be 1 FME
> >           per DFL based FPGA device.
> >
> > +config FPGA_DFL_FME_MGR
> > +       tristate "FPGA DFL FME Manager Driver"
> > +       depends on FPGA_DFL_FME
> > +       help
> > +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
> > +
> >  config INTEL_FPGA_DFL_PCI
> >         tristate "Intel FPGA DFL PCIe Device Driver"
> >         depends on PCI && FPGA_DFL
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index cc75bb3..6378580 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> >  # FPGA Device Feature List Support
> >  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
> >  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
> > +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
> >
> >  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
> >
> > diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
> > new file mode 100644
> > index 0000000..70356ce
> > --- /dev/null
> > +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
> > @@ -0,0 +1,318 @@
> > +/*
> > + * FPGA Manager Driver for FPGA Management Engine (FME)
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Joseph Grecco <joe.grecco@intel.com>
> > + *   Enno Luebbers <enno.luebbers@intel.com>
> > + *   Tim Whisonant <tim.whisonant@intel.com>
> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > + *   Christopher Rauer <christopher.rauer@intel.com>
> > + *   Henry Mitchel <henry.mitchel@intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2.
> > + * SPDX-License-Identifier: GPL-2.0
> > + */
> > +
> > +#include <linux/module.h>
> > +#include <linux/iopoll.h>
> > +#include <linux/fpga/fpga-mgr.h>
> > +
> > +#include "fpga-dfl.h"
> > +#include "dfl-fme.h"
> > +
> > +#define PR_WAIT_TIMEOUT   8000000
> > +#define PR_HOST_STATUS_IDLE    0
> > +
> > +struct fme_mgr_priv {
> > +       void __iomem *ioaddr;
> > +       u64 pr_error;
> > +};
> > +
> > +static ssize_t interface_id_show(struct device *dev,
> > +                                struct device_attribute *attr, char *buf)
> > +{
> > +       struct fpga_manager *mgr = dev_get_drvdata(dev);
> > +       struct fme_mgr_priv *priv = mgr->priv;
> > +       u64 intfc_id_l, intfc_id_h;
> > +
> > +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
> > +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > +                       (unsigned long long)intfc_id_h,
> > +                       (unsigned long long)intfc_id_l);
> > +}
> > +static DEVICE_ATTR_RO(interface_id);
> > +
> > +static const struct attribute *fme_mgr_attrs[] = {
> > +       &dev_attr_interface_id.attr,
> > +       NULL,
> > +};
> > +
> > +static u64 pr_error_to_mgr_status(u64 err)
> > +{
> > +       u64 status = 0;
> > +
> > +       if (err & FME_PR_ERR_OPERATION_ERR)
> > +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
> > +       if (err & FME_PR_ERR_CRC_ERR)
> > +               status |= FPGA_MGR_STATUS_CRC_ERR;
> > +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
> > +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
> > +       if (err & FME_PR_ERR_PROTOCOL_ERR)
> > +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> > +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
> > +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> > +
> > +       return status;
> > +}
> > +
> > +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
> > +{
> > +       u64 pr_status, pr_error;
> > +
> > +       pr_status = readq(fme_pr + FME_PR_STS);
> > +       if (!(pr_status & FME_PR_STS_PR_STS))
> > +               return 0;
> > +
> > +       pr_error = readq(fme_pr + FME_PR_ERR);
> > +       writeq(pr_error, fme_pr + FME_PR_ERR);
> > +
> > +       return pr_error;
> > +}
> > +
> > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> > +                             struct fpga_image_info *info,
> > +                             const char *buf, size_t count)
> > +{
> > +       struct device *dev = &mgr->dev;
> > +       struct fme_mgr_priv *priv = mgr->priv;
> > +       void __iomem *fme_pr = priv->ioaddr;
> > +       u64 pr_ctrl, pr_status;
> > +
> > +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> > +               dev_err(dev, "only support partial reconfiguration.\n");
> 
> supports
> 
> > +               return -EINVAL;
> > +       }
> > +
> > +       dev_dbg(dev, "resetting PR before initiated PR\n");
> > +
> > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > +       pr_ctrl |= FME_PR_CTRL_PR_RST;
> > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > +
> > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
> > +                              PR_WAIT_TIMEOUT)) {
> > +               dev_err(dev, "maximum PR timeout\n");
> 
> We have two dev_err's with the same "maximum PR timeout" so the user
> would not be able to see at which point the timeout occurred.
> 
> I suggest to get rid of the word 'maximum' for both.  This one could
> be 'reset ack timeout" or something similar.

Sure, will switch to a more specific message per your suggestion. Thanks.

> 
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
> > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > +
> > +       dev_dbg(dev,
> > +               "waiting for PR resource in HW to be initialized and ready\n");
> > +
> > +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
> > +                              (pr_status & FME_PR_STS_PR_STS) ==
> > +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
> > +               dev_err(dev, "maximum PR timeout\n");
> 
> "PR_STS timeout"?  Or something better.
> 
> > +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(dev, "check and clear previous PR error\n");
> > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > +       if (priv->pr_error)
> > +               dev_dbg(dev, "previous PR error detected %llx\n",
> > +                       (unsigned long long)priv->pr_error);
> > +
> > +       dev_dbg(dev, "set PR port ID\n");
> > +
> > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
> > +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
> > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > +
> > +       return 0;
> > +}
> > +
> > +static int fme_mgr_write(struct fpga_manager *mgr,
> > +                        const char *buf, size_t count)
> > +{
> > +       struct device *dev = &mgr->dev;
> > +       struct fme_mgr_priv *priv = mgr->priv;
> > +       void __iomem *fme_pr = priv->ioaddr;
> > +       u64 pr_ctrl, pr_status, pr_data;
> > +       int delay = 0, pr_credit, i = 0;
> > +
> > +       dev_dbg(dev, "start request\n");
> > +
> > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > +       pr_ctrl |= FME_PR_CTRL_PR_START;
> > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > +
> > +       dev_dbg(dev, "pushing data from bitstream to HW\n");
> > +
> > +       pr_status = readq(fme_pr + FME_PR_STS);
> > +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > +
> > +       while (count > 0) {
> > +               while (pr_credit <= 1) {
> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > +                               dev_err(dev, "maximum try\n");
> 
> It looks like this is waiting for an entry in a queue and timing out
> here.  Could you add a some comments for pr_credit above and this
> loop?  Also a better error message, perhaps "PR credit timeout"?

Driver needs to read the PR credit to know if it could push PR data
to hardware or not. I will add more description here on this PR credit,
and use "PR credit timeout" as error message.

> 
> > +                               return -ETIMEDOUT;
> > +                       }
> > +                       udelay(1);
> > +
> > +                       pr_status = readq(fme_pr + FME_PR_STS);
> > +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > +               }
> > +
> > +               if (count >= 4) {
> > +                       pr_data = 0;
> > +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> > +                                             *(((u32 *)buf) + i));
> > +                       writeq(pr_data, fme_pr + FME_PR_DATA);
> > +                       count -= 4;
> > +                       pr_credit--;
> > +                       i++;
> > +               } else {
> > +                       WARN_ON(1);
> > +                       return -EINVAL;
> > +               }
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> > +                                 struct fpga_image_info *info)
> > +{
> > +       struct device *dev = &mgr->dev;
> > +       struct fme_mgr_priv *priv = mgr->priv;
> > +       void __iomem *fme_pr = priv->ioaddr;
> > +       u64 pr_ctrl;
> > +
> > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
> > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > +
> > +       dev_dbg(dev, "green bitstream push complete\n");
> > +       dev_dbg(dev, "waiting for HW to release PR resource\n");
> > +
> > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
> > +                              PR_WAIT_TIMEOUT)) {
> > +               dev_err(dev, "maximum try.\n");
> 
> Some message specific to here also, please.
> 
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(dev, "PR operation complete, checking status\n");
> > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > +       if (priv->pr_error) {
> > +               dev_dbg(dev, "PR error detected %llx\n",
> > +                       (unsigned long long)priv->pr_error);
> > +               return -EIO;
> > +       }
> > +
> > +       dev_dbg(dev, "PR done successfully\n");
> > +
> > +       return 0;
> > +}
> > +
> > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> > +{
> > +       return FPGA_MGR_STATE_UNKNOWN;
> > +}
> > +
> > +static u64 fme_mgr_status(struct fpga_manager *mgr)
> > +{
> > +       struct fme_mgr_priv *priv = mgr->priv;
> > +
> > +       return pr_error_to_mgr_status(priv->pr_error);
> > +}
> > +
> > +static const struct fpga_manager_ops fme_mgr_ops = {
> > +       .write_init = fme_mgr_write_init,
> > +       .write = fme_mgr_write,
> > +       .write_complete = fme_mgr_write_complete,
> > +       .state = fme_mgr_state,
> > +       .status = fme_mgr_status,
> > +};
> > +
> > +static int fme_mgr_probe(struct platform_device *pdev)
> > +{
> > +       struct device *dev = &pdev->dev;
> > +       struct fme_mgr_priv *priv;
> > +       struct fpga_manager *mgr;
> > +       struct resource *res;
> > +       int ret;
> > +
> > +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> > +       if (!priv)
> > +               return -ENOMEM;
> > +
> > +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> > +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> > +       if (IS_ERR(priv->ioaddr))
> > +               return PTR_ERR(priv->ioaddr);
> > +
> > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> > +       if (ret)
> > +               return ret;
> > +
> > +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
> > +       if (!mgr)
> > +               goto sysfs_remove_exit;
> > +
> > +       mgr->name = "DFL FPGA Manager";
> > +       mgr->mops = &fme_mgr_ops;
> > +       mgr->priv = priv;
> > +       mgr->parent = dev;
> > +       platform_set_drvdata(pdev, mgr);
> > +
> > +       ret = fpga_mgr_register(mgr);
> > +       if (ret) {
> > +               dev_err(dev, "unable to register FPGA manager\n");
> > +               goto sysfs_remove_exit;
> > +       }
> > +
> > +       return 0;
> > +
> > +sysfs_remove_exit:
> > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > +       return ret;
> > +}
> > +
> > +static int fme_mgr_remove(struct platform_device *pdev)
> > +{
> > +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
> > +
> > +       fpga_mgr_unregister(mgr);
> > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > +
> > +       return 0;
> > +}
> > +
> > +static struct platform_driver fme_mgr_driver = {
> > +       .driver = {
> > +               .name    = FPGA_DFL_FME_MGR,
> > +       },
> > +       .probe   = fme_mgr_probe,
> > +       .remove  = fme_mgr_remove,
> > +};
> > +
> > +module_platform_driver(fme_mgr_driver);
> > +
> > +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
> > +MODULE_AUTHOR("Intel Corporation");
> > +MODULE_LICENSE("GPL v2");
> > +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
> > diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> > index e5a1094..d45eb82 100644
> > --- a/drivers/fpga/fpga-dfl.h
> > +++ b/drivers/fpga/fpga-dfl.h
> > @@ -130,7 +130,44 @@
> >
> >  /* FME Partial Reconfiguration Sub Feature Register Set */
> >  #define FME_PR_DFH             DFH
> > -#define FME_PR_SIZE            DFH_SIZE
> > +#define FME_PR_CTRL            0x8
> > +#define FME_PR_STS             0x10
> > +#define FME_PR_DATA            0x18
> > +#define FME_PR_ERR             0x20
> > +#define FME_PR_INTFC_ID_H      0xA8
> > +#define FME_PR_INTFC_ID_L      0xB0
> > +#define FME_PR_SIZE            0xB8
> > +
> > +/* FME PR Control Register Bitfield */
> > +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
> > +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
> > +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
> > +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
> > +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
> > +
> > +/* FME PR Status Register Bitfield */
> > +/* Number of available entries in HW queue inside the PR engine. */
> > +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
> > +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
> > +#define FME_PR_STS_PR_STS_IDLE 0
> > +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
> > +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
> > +
> > +/* FME PR Data Register Bitfield */
> > +/* PR data from the raw-binary file. */
> > +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
> > +
> > +/* FME PR Error Register */
> > +/* Previous PR Operation errors detected. */
> > +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
> > +/* CRC error detected. */
> > +#define FME_PR_ERR_CRC_ERR             BIT(1)
> > +/* Incompatible PR bitstream detected. */
> > +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
> > +/* PR data push protocol violated. */
> > +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
> > +/* PR data fifo overflow error detected */
> > +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
> >
> >  /* FME HSSI Sub Feature Register Set */
> >  #define FME_HSSI_DFH           DFH
> 
> I see fpga-dfl.h as enumeration code which is separate from any driver
> implementation specifics other than what's required for the DFL
> enumeration scheme.    These PR engine #defines should move to a .h
> that is dedicated to this specific PR hardware device. If someone else
> adds a different PR device to the framework, their PR driver would
> also have its own .h.  Same for any other modules that aren't central
> to DFL enumeration.

Sure, will move PR related register to a separated header file.
DFL enumeration related registers, will still be kept in fpga-dfl.h.

Thanks
Hao

> 
> Thanks,
> Alan
> 
> > --
> > 1.8.3.1
> >

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Luebbers, Enno]

Hi Hao, Alan,

On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > 
> > Hi Hao,
> > 
> > A few comments below.   Besides that, looks good.
> > 
> > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> > >
> > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > ----
> > > v3: rename driver to dfl-fpga-fme-mgr
> > >     implemented status callback for fpga manager
> > >     rebased due to fpga api changes
> > > ---
> > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> > >  drivers/fpga/Kconfig                               |   6 +
> > >  drivers/fpga/Makefile                              |   1 +
> > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> > >  5 files changed, 371 insertions(+), 1 deletion(-)
> > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> > >
> > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > new file mode 100644
> > > index 0000000..2d4f917
> > > --- /dev/null
> > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > @@ -0,0 +1,8 @@
> > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> > > +Date:          November 2017
> > > +KernelVersion:  4.15
> > > +Contact:       Wu Hao <hao.wu@intel.com>
> > > +Description:   Read-only. It returns interface id of partial reconfiguration
> > > +               hardware. Userspace could use this information to check if
> > > +               current hardware is compatible with given image before FPGA
> > > +               programming.
> > 
> > I'm a little confused by this.  I can understand that the PR bitstream
> > has a dependency on the FPGA's static image, but I don't understand
> > the dependency of the bistream on the hardware that is used to program
> > the bitstream to the FPGA.
> 
> Sorry for the confusion, the interface_id is used to indicate the version of
> the hardware for partial reconfiguration (it's part of the static image of
> the FPGA device). Will improve the description on this.
> 

The interface_id expresses the compatibility of the static region with PR
bitstreams generated for it. It changes every time a new static region is
generated.

Would it make more sense to have the interface_id exposed as part of the FME
device (which represents the static region)? I'm not sure - it kind of also
makes sense here, where you would have all the information in one place (if the
interface_id matches, I can use this component to program a bitstream).

Sorry for my limited understanding of the infrastructure - would this same
"fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
it would need to expose multiple interface_ids (or we'd have to track both
interface IDs and an identifier for the target PR region).

> > 
> > > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > > index 57da904..0171ecb 100644
> > > --- a/drivers/fpga/Kconfig
> > > +++ b/drivers/fpga/Kconfig
> > > @@ -150,6 +150,12 @@ config FPGA_DFL_FME
> > >           FPGA platform level management features. There shall be 1 FME
> > >           per DFL based FPGA device.
> > >
> > > +config FPGA_DFL_FME_MGR
> > > +       tristate "FPGA DFL FME Manager Driver"
> > > +       depends on FPGA_DFL_FME
> > > +       help
> > > +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
> > > +
> > >  config INTEL_FPGA_DFL_PCI
> > >         tristate "Intel FPGA DFL PCIe Device Driver"
> > >         depends on PCI && FPGA_DFL
> > > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > > index cc75bb3..6378580 100644
> > > --- a/drivers/fpga/Makefile
> > > +++ b/drivers/fpga/Makefile
> > > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> > >  # FPGA Device Feature List Support
> > >  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
> > >  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
> > > +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
> > >
> > >  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
> > >
> > > diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > new file mode 100644
> > > index 0000000..70356ce
> > > --- /dev/null
> > > +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > @@ -0,0 +1,318 @@
> > > +/*
> > > + * FPGA Manager Driver for FPGA Management Engine (FME)
> > > + *
> > > + * Copyright (C) 2017 Intel Corporation, Inc.
> > > + *
> > > + * Authors:
> > > + *   Kang Luwei <luwei.kang@intel.com>
> > > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > + *   Wu Hao <hao.wu@intel.com>
> > > + *   Joseph Grecco <joe.grecco@intel.com>
> > > + *   Enno Luebbers <enno.luebbers@intel.com>
> > > + *   Tim Whisonant <tim.whisonant@intel.com>
> > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > > + *   Christopher Rauer <christopher.rauer@intel.com>
> > > + *   Henry Mitchel <henry.mitchel@intel.com>
> > > + *
> > > + * This work is licensed under the terms of the GNU GPL version 2.
> > > + * SPDX-License-Identifier: GPL-2.0
> > > + */
> > > +
> > > +#include <linux/module.h>
> > > +#include <linux/iopoll.h>
> > > +#include <linux/fpga/fpga-mgr.h>
> > > +
> > > +#include "fpga-dfl.h"
> > > +#include "dfl-fme.h"
> > > +
> > > +#define PR_WAIT_TIMEOUT   8000000
> > > +#define PR_HOST_STATUS_IDLE    0
> > > +
> > > +struct fme_mgr_priv {
> > > +       void __iomem *ioaddr;
> > > +       u64 pr_error;
> > > +};
> > > +
> > > +static ssize_t interface_id_show(struct device *dev,
> > > +                                struct device_attribute *attr, char *buf)
> > > +{
> > > +       struct fpga_manager *mgr = dev_get_drvdata(dev);
> > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > +       u64 intfc_id_l, intfc_id_h;
> > > +
> > > +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
> > > +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
> > > +
> > > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > > +                       (unsigned long long)intfc_id_h,
> > > +                       (unsigned long long)intfc_id_l);
> > > +}
> > > +static DEVICE_ATTR_RO(interface_id);
> > > +
> > > +static const struct attribute *fme_mgr_attrs[] = {
> > > +       &dev_attr_interface_id.attr,
> > > +       NULL,
> > > +};
> > > +
> > > +static u64 pr_error_to_mgr_status(u64 err)
> > > +{
> > > +       u64 status = 0;
> > > +
> > > +       if (err & FME_PR_ERR_OPERATION_ERR)
> > > +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
> > > +       if (err & FME_PR_ERR_CRC_ERR)
> > > +               status |= FPGA_MGR_STATUS_CRC_ERR;
> > > +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
> > > +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
> > > +       if (err & FME_PR_ERR_PROTOCOL_ERR)
> > > +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> > > +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
> > > +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> > > +
> > > +       return status;
> > > +}
> > > +
> > > +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
> > > +{
> > > +       u64 pr_status, pr_error;
> > > +
> > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > +       if (!(pr_status & FME_PR_STS_PR_STS))
> > > +               return 0;
> > > +
> > > +       pr_error = readq(fme_pr + FME_PR_ERR);
> > > +       writeq(pr_error, fme_pr + FME_PR_ERR);
> > > +
> > > +       return pr_error;
> > > +}
> > > +
> > > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> > > +                             struct fpga_image_info *info,
> > > +                             const char *buf, size_t count)
> > > +{
> > > +       struct device *dev = &mgr->dev;
> > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > +       void __iomem *fme_pr = priv->ioaddr;
> > > +       u64 pr_ctrl, pr_status;
> > > +
> > > +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> > > +               dev_err(dev, "only support partial reconfiguration.\n");
> > 
> > supports
> > 
> > > +               return -EINVAL;
> > > +       }
> > > +
> > > +       dev_dbg(dev, "resetting PR before initiated PR\n");
> > > +
> > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > +       pr_ctrl |= FME_PR_CTRL_PR_RST;
> > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > +
> > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
> > > +                              PR_WAIT_TIMEOUT)) {
> > > +               dev_err(dev, "maximum PR timeout\n");
> > 
> > We have two dev_err's with the same "maximum PR timeout" so the user
> > would not be able to see at which point the timeout occurred.
> > 
> > I suggest to get rid of the word 'maximum' for both.  This one could
> > be 'reset ack timeout" or something similar.
> 
> Sure, will switch to a more specific message per your suggestion. Thanks.
> 
> > 
> > > +               return -ETIMEDOUT;
> > > +       }
> > > +
> > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
> > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > +
> > > +       dev_dbg(dev,
> > > +               "waiting for PR resource in HW to be initialized and ready\n");
> > > +
> > > +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
> > > +                              (pr_status & FME_PR_STS_PR_STS) ==
> > > +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
> > > +               dev_err(dev, "maximum PR timeout\n");
> > 
> > "PR_STS timeout"?  Or something better.
> > 
> > > +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > +               return -ETIMEDOUT;
> > > +       }
> > > +
> > > +       dev_dbg(dev, "check and clear previous PR error\n");
> > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > +       if (priv->pr_error)
> > > +               dev_dbg(dev, "previous PR error detected %llx\n",
> > > +                       (unsigned long long)priv->pr_error);
> > > +
> > > +       dev_dbg(dev, "set PR port ID\n");
> > > +
> > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
> > > +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
> > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > +
> > > +       return 0;
> > > +}
> > > +
> > > +static int fme_mgr_write(struct fpga_manager *mgr,
> > > +                        const char *buf, size_t count)
> > > +{
> > > +       struct device *dev = &mgr->dev;
> > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > +       void __iomem *fme_pr = priv->ioaddr;
> > > +       u64 pr_ctrl, pr_status, pr_data;
> > > +       int delay = 0, pr_credit, i = 0;
> > > +
> > > +       dev_dbg(dev, "start request\n");
> > > +
> > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > +       pr_ctrl |= FME_PR_CTRL_PR_START;
> > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > +
> > > +       dev_dbg(dev, "pushing data from bitstream to HW\n");
> > > +
> > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > +
> > > +       while (count > 0) {
> > > +               while (pr_credit <= 1) {
> > > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > > +                               dev_err(dev, "maximum try\n");
> > 
> > It looks like this is waiting for an entry in a queue and timing out
> > here.  Could you add a some comments for pr_credit above and this
> > loop?  Also a better error message, perhaps "PR credit timeout"?
> 
> Driver needs to read the PR credit to know if it could push PR data
> to hardware or not. I will add more description here on this PR credit,
> and use "PR credit timeout" as error message.
> 
> > 
> > > +                               return -ETIMEDOUT;
> > > +                       }
> > > +                       udelay(1);
> > > +
> > > +                       pr_status = readq(fme_pr + FME_PR_STS);
> > > +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > +               }
> > > +
> > > +               if (count >= 4) {
> > > +                       pr_data = 0;
> > > +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> > > +                                             *(((u32 *)buf) + i));
> > > +                       writeq(pr_data, fme_pr + FME_PR_DATA);
> > > +                       count -= 4;
> > > +                       pr_credit--;
> > > +                       i++;
> > > +               } else {
> > > +                       WARN_ON(1);
> > > +                       return -EINVAL;
> > > +               }
> > > +       }
> > > +
> > > +       return 0;
> > > +}
> > > +
> > > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> > > +                                 struct fpga_image_info *info)
> > > +{
> > > +       struct device *dev = &mgr->dev;
> > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > +       void __iomem *fme_pr = priv->ioaddr;
> > > +       u64 pr_ctrl;
> > > +
> > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
> > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > +
> > > +       dev_dbg(dev, "green bitstream push complete\n");
> > > +       dev_dbg(dev, "waiting for HW to release PR resource\n");
> > > +
> > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
> > > +                              PR_WAIT_TIMEOUT)) {
> > > +               dev_err(dev, "maximum try.\n");
> > 
> > Some message specific to here also, please.
> > 
> > > +               return -ETIMEDOUT;
> > > +       }
> > > +
> > > +       dev_dbg(dev, "PR operation complete, checking status\n");
> > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > +       if (priv->pr_error) {
> > > +               dev_dbg(dev, "PR error detected %llx\n",
> > > +                       (unsigned long long)priv->pr_error);
> > > +               return -EIO;
> > > +       }
> > > +
> > > +       dev_dbg(dev, "PR done successfully\n");
> > > +
> > > +       return 0;
> > > +}
> > > +
> > > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> > > +{
> > > +       return FPGA_MGR_STATE_UNKNOWN;
> > > +}
> > > +
> > > +static u64 fme_mgr_status(struct fpga_manager *mgr)
> > > +{
> > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > +
> > > +       return pr_error_to_mgr_status(priv->pr_error);
> > > +}
> > > +
> > > +static const struct fpga_manager_ops fme_mgr_ops = {
> > > +       .write_init = fme_mgr_write_init,
> > > +       .write = fme_mgr_write,
> > > +       .write_complete = fme_mgr_write_complete,
> > > +       .state = fme_mgr_state,
> > > +       .status = fme_mgr_status,
> > > +};
> > > +
> > > +static int fme_mgr_probe(struct platform_device *pdev)
> > > +{
> > > +       struct device *dev = &pdev->dev;
> > > +       struct fme_mgr_priv *priv;
> > > +       struct fpga_manager *mgr;
> > > +       struct resource *res;
> > > +       int ret;
> > > +
> > > +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> > > +       if (!priv)
> > > +               return -ENOMEM;
> > > +
> > > +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> > > +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> > > +       if (IS_ERR(priv->ioaddr))
> > > +               return PTR_ERR(priv->ioaddr);
> > > +
> > > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > +       if (ret)
> > > +               return ret;
> > > +
> > > +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
> > > +       if (!mgr)
> > > +               goto sysfs_remove_exit;
> > > +
> > > +       mgr->name = "DFL FPGA Manager";
> > > +       mgr->mops = &fme_mgr_ops;
> > > +       mgr->priv = priv;
> > > +       mgr->parent = dev;
> > > +       platform_set_drvdata(pdev, mgr);
> > > +
> > > +       ret = fpga_mgr_register(mgr);
> > > +       if (ret) {
> > > +               dev_err(dev, "unable to register FPGA manager\n");
> > > +               goto sysfs_remove_exit;
> > > +       }
> > > +
> > > +       return 0;
> > > +
> > > +sysfs_remove_exit:
> > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > +       return ret;
> > > +}
> > > +
> > > +static int fme_mgr_remove(struct platform_device *pdev)
> > > +{
> > > +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
> > > +
> > > +       fpga_mgr_unregister(mgr);
> > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > +
> > > +       return 0;
> > > +}
> > > +
> > > +static struct platform_driver fme_mgr_driver = {
> > > +       .driver = {
> > > +               .name    = FPGA_DFL_FME_MGR,
> > > +       },
> > > +       .probe   = fme_mgr_probe,
> > > +       .remove  = fme_mgr_remove,
> > > +};
> > > +
> > > +module_platform_driver(fme_mgr_driver);
> > > +
> > > +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
> > > +MODULE_AUTHOR("Intel Corporation");
> > > +MODULE_LICENSE("GPL v2");
> > > +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
> > > diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> > > index e5a1094..d45eb82 100644
> > > --- a/drivers/fpga/fpga-dfl.h
> > > +++ b/drivers/fpga/fpga-dfl.h
> > > @@ -130,7 +130,44 @@
> > >
> > >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > >  #define FME_PR_DFH             DFH
> > > -#define FME_PR_SIZE            DFH_SIZE
> > > +#define FME_PR_CTRL            0x8
> > > +#define FME_PR_STS             0x10
> > > +#define FME_PR_DATA            0x18
> > > +#define FME_PR_ERR             0x20
> > > +#define FME_PR_INTFC_ID_H      0xA8
> > > +#define FME_PR_INTFC_ID_L      0xB0
> > > +#define FME_PR_SIZE            0xB8
> > > +
> > > +/* FME PR Control Register Bitfield */
> > > +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
> > > +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
> > > +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
> > > +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
> > > +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
> > > +
> > > +/* FME PR Status Register Bitfield */
> > > +/* Number of available entries in HW queue inside the PR engine. */
> > > +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
> > > +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
> > > +#define FME_PR_STS_PR_STS_IDLE 0
> > > +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
> > > +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
> > > +
> > > +/* FME PR Data Register Bitfield */
> > > +/* PR data from the raw-binary file. */
> > > +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
> > > +
> > > +/* FME PR Error Register */
> > > +/* Previous PR Operation errors detected. */
> > > +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
> > > +/* CRC error detected. */
> > > +#define FME_PR_ERR_CRC_ERR             BIT(1)
> > > +/* Incompatible PR bitstream detected. */
> > > +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
> > > +/* PR data push protocol violated. */
> > > +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
> > > +/* PR data fifo overflow error detected */
> > > +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
> > >
> > >  /* FME HSSI Sub Feature Register Set */
> > >  #define FME_HSSI_DFH           DFH
> > 
> > I see fpga-dfl.h as enumeration code which is separate from any driver
> > implementation specifics other than what's required for the DFL
> > enumeration scheme.    These PR engine #defines should move to a .h
> > that is dedicated to this specific PR hardware device. If someone else
> > adds a different PR device to the framework, their PR driver would
> > also have its own .h.  Same for any other modules that aren't central
> > to DFL enumeration.
> 
> Sure, will move PR related register to a separated header file.
> DFL enumeration related registers, will still be kept in fpga-dfl.h.
> 
> Thanks
> Hao
> 
> > 
> > Thanks,
> > Alan
> > 
> > > --
> > > 1.8.3.1
> > >

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Moritz Fischer]

[-- Attachment #1: Type: text/plain, Size: 22982 bytes --]

Hi Hao,

On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> Hi Hao, Alan,
> 
> On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > > 
> > > Hi Hao,
> > > 
> > > A few comments below.   Besides that, looks good.
> > > 
> > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> > > >
> > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > > ----
> > > > v3: rename driver to dfl-fpga-fme-mgr
> > > >     implemented status callback for fpga manager
> > > >     rebased due to fpga api changes
> > > > ---
> > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> > > >  drivers/fpga/Kconfig                               |   6 +
> > > >  drivers/fpga/Makefile                              |   1 +
> > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> > > >
> > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > new file mode 100644
> > > > index 0000000..2d4f917
> > > > --- /dev/null
> > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > @@ -0,0 +1,8 @@
> > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> > > > +Date:          November 2017
> > > > +KernelVersion:  4.15
> > > > +Contact:       Wu Hao <hao.wu@intel.com>
> > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> > > > +               hardware. Userspace could use this information to check if
> > > > +               current hardware is compatible with given image before FPGA
> > > > +               programming.
> > > 
> > > I'm a little confused by this.  I can understand that the PR bitstream
> > > has a dependency on the FPGA's static image, but I don't understand
> > > the dependency of the bistream on the hardware that is used to program
> > > the bitstream to the FPGA.
> > 
> > Sorry for the confusion, the interface_id is used to indicate the version of
> > the hardware for partial reconfiguration (it's part of the static image of
> > the FPGA device). Will improve the description on this.

I'm not sure userland should be making the call on whether what you're
trying to load is compatible or not. Isn't there a way to check this in
your PR reconfiguration handler in-kernel?

> > 
> 
> The interface_id expresses the compatibility of the static region with PR
> bitstreams generated for it. It changes every time a new static region is
> generated.
> 
> Would it make more sense to have the interface_id exposed as part of the FME
> device (which represents the static region)? I'm not sure - it kind of also
> makes sense here, where you would have all the information in one place (if the
> interface_id matches, I can use this component to program a bitstream).
> 
> Sorry for my limited understanding of the infrastructure - would this same
> "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
> it would need to expose multiple interface_ids (or we'd have to track both
> interface IDs and an identifier for the target PR region).
> 
> > > 
> > > > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > > > index 57da904..0171ecb 100644
> > > > --- a/drivers/fpga/Kconfig
> > > > +++ b/drivers/fpga/Kconfig
> > > > @@ -150,6 +150,12 @@ config FPGA_DFL_FME
> > > >           FPGA platform level management features. There shall be 1 FME
> > > >           per DFL based FPGA device.
> > > >
> > > > +config FPGA_DFL_FME_MGR
> > > > +       tristate "FPGA DFL FME Manager Driver"
> > > > +       depends on FPGA_DFL_FME
> > > > +       help
> > > > +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
> > > > +
> > > >  config INTEL_FPGA_DFL_PCI
> > > >         tristate "Intel FPGA DFL PCIe Device Driver"
> > > >         depends on PCI && FPGA_DFL
> > > > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > > > index cc75bb3..6378580 100644
> > > > --- a/drivers/fpga/Makefile
> > > > +++ b/drivers/fpga/Makefile
> > > > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> > > >  # FPGA Device Feature List Support
> > > >  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
> > > >  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
> > > > +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
> > > >
> > > >  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
> > > >
> > > > diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > new file mode 100644
> > > > index 0000000..70356ce
> > > > --- /dev/null
> > > > +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > @@ -0,0 +1,318 @@
> > > > +/*
> > > > + * FPGA Manager Driver for FPGA Management Engine (FME)
> > > > + *
> > > > + * Copyright (C) 2017 Intel Corporation, Inc.
> > > > + *
> > > > + * Authors:
> > > > + *   Kang Luwei <luwei.kang@intel.com>
> > > > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > + *   Wu Hao <hao.wu@intel.com>
> > > > + *   Joseph Grecco <joe.grecco@intel.com>
> > > > + *   Enno Luebbers <enno.luebbers@intel.com>
> > > > + *   Tim Whisonant <tim.whisonant@intel.com>
> > > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > > > + *   Christopher Rauer <christopher.rauer@intel.com>
> > > > + *   Henry Mitchel <henry.mitchel@intel.com>
> > > > + *
> > > > + * This work is licensed under the terms of the GNU GPL version 2.
> > > > + * SPDX-License-Identifier: GPL-2.0
> > > > + */
> > > > +
> > > > +#include <linux/module.h>
> > > > +#include <linux/iopoll.h>
> > > > +#include <linux/fpga/fpga-mgr.h>
> > > > +
> > > > +#include "fpga-dfl.h"
> > > > +#include "dfl-fme.h"
> > > > +
> > > > +#define PR_WAIT_TIMEOUT   8000000
> > > > +#define PR_HOST_STATUS_IDLE    0
> > > > +
> > > > +struct fme_mgr_priv {
> > > > +       void __iomem *ioaddr;
> > > > +       u64 pr_error;
> > > > +};
> > > > +
> > > > +static ssize_t interface_id_show(struct device *dev,
> > > > +                                struct device_attribute *attr, char *buf)
> > > > +{
> > > > +       struct fpga_manager *mgr = dev_get_drvdata(dev);
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +       u64 intfc_id_l, intfc_id_h;
> > > > +
> > > > +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
> > > > +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
> > > > +
> > > > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > > > +                       (unsigned long long)intfc_id_h,
> > > > +                       (unsigned long long)intfc_id_l);
> > > > +}
> > > > +static DEVICE_ATTR_RO(interface_id);
> > > > +
> > > > +static const struct attribute *fme_mgr_attrs[] = {
> > > > +       &dev_attr_interface_id.attr,
> > > > +       NULL,
> > > > +};
> > > > +
> > > > +static u64 pr_error_to_mgr_status(u64 err)
> > > > +{
> > > > +       u64 status = 0;
> > > > +
> > > > +       if (err & FME_PR_ERR_OPERATION_ERR)
> > > > +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
> > > > +       if (err & FME_PR_ERR_CRC_ERR)
> > > > +               status |= FPGA_MGR_STATUS_CRC_ERR;
> > > > +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
> > > > +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
> > > > +       if (err & FME_PR_ERR_PROTOCOL_ERR)
> > > > +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> > > > +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
> > > > +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> > > > +
> > > > +       return status;
> > > > +}
> > > > +
> > > > +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
> > > > +{
> > > > +       u64 pr_status, pr_error;
> > > > +
> > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > +       if (!(pr_status & FME_PR_STS_PR_STS))
> > > > +               return 0;
> > > > +
> > > > +       pr_error = readq(fme_pr + FME_PR_ERR);
> > > > +       writeq(pr_error, fme_pr + FME_PR_ERR);
> > > > +
> > > > +       return pr_error;
> > > > +}
> > > > +
> > > > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> > > > +                             struct fpga_image_info *info,
> > > > +                             const char *buf, size_t count)
> > > > +{
> > > > +       struct device *dev = &mgr->dev;
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > +       u64 pr_ctrl, pr_status;
> > > > +
> > > > +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> > > > +               dev_err(dev, "only support partial reconfiguration.\n");
> > > 
> > > supports
> > > 
> > > > +               return -EINVAL;
> > > > +       }
> > > > +
> > > > +       dev_dbg(dev, "resetting PR before initiated PR\n");
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl |= FME_PR_CTRL_PR_RST;
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
> > > > +                              PR_WAIT_TIMEOUT)) {
> > > > +               dev_err(dev, "maximum PR timeout\n");
> > > 
> > > We have two dev_err's with the same "maximum PR timeout" so the user
> > > would not be able to see at which point the timeout occurred.
> > > 
> > > I suggest to get rid of the word 'maximum' for both.  This one could
> > > be 'reset ack timeout" or something similar.
> > 
> > Sure, will switch to a more specific message per your suggestion. Thanks.
> > 
> > > 
> > > > +               return -ETIMEDOUT;
> > > > +       }
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       dev_dbg(dev,
> > > > +               "waiting for PR resource in HW to be initialized and ready\n");
> > > > +
> > > > +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
> > > > +                              (pr_status & FME_PR_STS_PR_STS) ==
> > > > +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
> > > > +               dev_err(dev, "maximum PR timeout\n");
> > > 
> > > "PR_STS timeout"?  Or something better.
> > > 
> > > > +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > +               return -ETIMEDOUT;
> > > > +       }
> > > > +
> > > > +       dev_dbg(dev, "check and clear previous PR error\n");
> > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > +       if (priv->pr_error)
> > > > +               dev_dbg(dev, "previous PR error detected %llx\n",
> > > > +                       (unsigned long long)priv->pr_error);
> > > > +
> > > > +       dev_dbg(dev, "set PR port ID\n");
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
> > > > +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +
> > > > +static int fme_mgr_write(struct fpga_manager *mgr,
> > > > +                        const char *buf, size_t count)
> > > > +{
> > > > +       struct device *dev = &mgr->dev;
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > +       u64 pr_ctrl, pr_status, pr_data;
> > > > +       int delay = 0, pr_credit, i = 0;
> > > > +
> > > > +       dev_dbg(dev, "start request\n");
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl |= FME_PR_CTRL_PR_START;
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       dev_dbg(dev, "pushing data from bitstream to HW\n");
> > > > +
> > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > +
> > > > +       while (count > 0) {
> > > > +               while (pr_credit <= 1) {
> > > > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > > > +                               dev_err(dev, "maximum try\n");
> > > 
> > > It looks like this is waiting for an entry in a queue and timing out
> > > here.  Could you add a some comments for pr_credit above and this
> > > loop?  Also a better error message, perhaps "PR credit timeout"?
> > 
> > Driver needs to read the PR credit to know if it could push PR data
> > to hardware or not. I will add more description here on this PR credit,
> > and use "PR credit timeout" as error message.
> > 
> > > 
> > > > +                               return -ETIMEDOUT;
> > > > +                       }
> > > > +                       udelay(1);
> > > > +
> > > > +                       pr_status = readq(fme_pr + FME_PR_STS);
> > > > +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > +               }
> > > > +
> > > > +               if (count >= 4) {
> > > > +                       pr_data = 0;
> > > > +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> > > > +                                             *(((u32 *)buf) + i));
> > > > +                       writeq(pr_data, fme_pr + FME_PR_DATA);
> > > > +                       count -= 4;
> > > > +                       pr_credit--;
> > > > +                       i++;
> > > > +               } else {
> > > > +                       WARN_ON(1);
> > > > +                       return -EINVAL;
> > > > +               }
> > > > +       }
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +
> > > > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> > > > +                                 struct fpga_image_info *info)
> > > > +{
> > > > +       struct device *dev = &mgr->dev;
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > +       u64 pr_ctrl;
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       dev_dbg(dev, "green bitstream push complete\n");
> > > > +       dev_dbg(dev, "waiting for HW to release PR resource\n");
> > > > +
> > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
> > > > +                              PR_WAIT_TIMEOUT)) {
> > > > +               dev_err(dev, "maximum try.\n");
> > > 
> > > Some message specific to here also, please.
> > > 
> > > > +               return -ETIMEDOUT;
> > > > +       }
> > > > +
> > > > +       dev_dbg(dev, "PR operation complete, checking status\n");
> > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > +       if (priv->pr_error) {
> > > > +               dev_dbg(dev, "PR error detected %llx\n",
> > > > +                       (unsigned long long)priv->pr_error);
> > > > +               return -EIO;
> > > > +       }
> > > > +
> > > > +       dev_dbg(dev, "PR done successfully\n");
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +
> > > > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> > > > +{
> > > > +       return FPGA_MGR_STATE_UNKNOWN;
> > > > +}
> > > > +
> > > > +static u64 fme_mgr_status(struct fpga_manager *mgr)
> > > > +{
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +
> > > > +       return pr_error_to_mgr_status(priv->pr_error);
> > > > +}
> > > > +
> > > > +static const struct fpga_manager_ops fme_mgr_ops = {
> > > > +       .write_init = fme_mgr_write_init,
> > > > +       .write = fme_mgr_write,
> > > > +       .write_complete = fme_mgr_write_complete,
> > > > +       .state = fme_mgr_state,
> > > > +       .status = fme_mgr_status,
> > > > +};
> > > > +
> > > > +static int fme_mgr_probe(struct platform_device *pdev)
> > > > +{
> > > > +       struct device *dev = &pdev->dev;
> > > > +       struct fme_mgr_priv *priv;
> > > > +       struct fpga_manager *mgr;
> > > > +       struct resource *res;
> > > > +       int ret;
> > > > +
> > > > +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> > > > +       if (!priv)
> > > > +               return -ENOMEM;
> > > > +
> > > > +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> > > > +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> > > > +       if (IS_ERR(priv->ioaddr))
> > > > +               return PTR_ERR(priv->ioaddr);
> > > > +
> > > > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > +       if (ret)
> > > > +               return ret;
> > > > +
> > > > +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
> > > > +       if (!mgr)
> > > > +               goto sysfs_remove_exit;
> > > > +
> > > > +       mgr->name = "DFL FPGA Manager";
> > > > +       mgr->mops = &fme_mgr_ops;
> > > > +       mgr->priv = priv;
> > > > +       mgr->parent = dev;
> > > > +       platform_set_drvdata(pdev, mgr);
> > > > +
> > > > +       ret = fpga_mgr_register(mgr);
> > > > +       if (ret) {
> > > > +               dev_err(dev, "unable to register FPGA manager\n");
> > > > +               goto sysfs_remove_exit;
> > > > +       }
> > > > +
> > > > +       return 0;
> > > > +
> > > > +sysfs_remove_exit:
> > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > +       return ret;
> > > > +}
> > > > +
> > > > +static int fme_mgr_remove(struct platform_device *pdev)
> > > > +{
> > > > +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
> > > > +
> > > > +       fpga_mgr_unregister(mgr);
> > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +
> > > > +static struct platform_driver fme_mgr_driver = {
> > > > +       .driver = {
> > > > +               .name    = FPGA_DFL_FME_MGR,
> > > > +       },
> > > > +       .probe   = fme_mgr_probe,
> > > > +       .remove  = fme_mgr_remove,
> > > > +};
> > > > +
> > > > +module_platform_driver(fme_mgr_driver);
> > > > +
> > > > +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
> > > > +MODULE_AUTHOR("Intel Corporation");
> > > > +MODULE_LICENSE("GPL v2");
> > > > +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
> > > > diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> > > > index e5a1094..d45eb82 100644
> > > > --- a/drivers/fpga/fpga-dfl.h
> > > > +++ b/drivers/fpga/fpga-dfl.h
> > > > @@ -130,7 +130,44 @@
> > > >
> > > >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > > >  #define FME_PR_DFH             DFH
> > > > -#define FME_PR_SIZE            DFH_SIZE
> > > > +#define FME_PR_CTRL            0x8
> > > > +#define FME_PR_STS             0x10
> > > > +#define FME_PR_DATA            0x18
> > > > +#define FME_PR_ERR             0x20
> > > > +#define FME_PR_INTFC_ID_H      0xA8
> > > > +#define FME_PR_INTFC_ID_L      0xB0
> > > > +#define FME_PR_SIZE            0xB8
> > > > +
> > > > +/* FME PR Control Register Bitfield */
> > > > +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
> > > > +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
> > > > +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
> > > > +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
> > > > +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
> > > > +
> > > > +/* FME PR Status Register Bitfield */
> > > > +/* Number of available entries in HW queue inside the PR engine. */
> > > > +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
> > > > +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
> > > > +#define FME_PR_STS_PR_STS_IDLE 0
> > > > +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
> > > > +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
> > > > +
> > > > +/* FME PR Data Register Bitfield */
> > > > +/* PR data from the raw-binary file. */
> > > > +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
> > > > +
> > > > +/* FME PR Error Register */
> > > > +/* Previous PR Operation errors detected. */
> > > > +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
> > > > +/* CRC error detected. */
> > > > +#define FME_PR_ERR_CRC_ERR             BIT(1)
> > > > +/* Incompatible PR bitstream detected. */
> > > > +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
> > > > +/* PR data push protocol violated. */
> > > > +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
> > > > +/* PR data fifo overflow error detected */
> > > > +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
> > > >
> > > >  /* FME HSSI Sub Feature Register Set */
> > > >  #define FME_HSSI_DFH           DFH
> > > 
> > > I see fpga-dfl.h as enumeration code which is separate from any driver
> > > implementation specifics other than what's required for the DFL
> > > enumeration scheme.    These PR engine #defines should move to a .h
> > > that is dedicated to this specific PR hardware device. If someone else
> > > adds a different PR device to the framework, their PR driver would
> > > also have its own .h.  Same for any other modules that aren't central
> > > to DFL enumeration.
> > 
> > Sure, will move PR related register to a separated header file.
> > DFL enumeration related registers, will still be kept in fpga-dfl.h.
> > 
> > Thanks
> > Hao
> > 
> > > 
> > > Thanks,
> > > Alan
> > > 
> > > > --
> > > > 1.8.3.1
> > > >

Thanks,
Moritz

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Wu Hao]

On Sat, Feb 03, 2018 at 11:41:24AM +0100, Moritz Fischer wrote:
> Hi Hao,
> 
> On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> > Hi Hao, Alan,
> > 
> > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > > > 
> > > > Hi Hao,
> > > > 
> > > > A few comments below.   Besides that, looks good.
> > > > 
> > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> > > > >
> > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > > > ----
> > > > > v3: rename driver to dfl-fpga-fme-mgr
> > > > >     implemented status callback for fpga manager
> > > > >     rebased due to fpga api changes
> > > > > ---
> > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> > > > >  drivers/fpga/Kconfig                               |   6 +
> > > > >  drivers/fpga/Makefile                              |   1 +
> > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> > > > >
> > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > > new file mode 100644
> > > > > index 0000000..2d4f917
> > > > > --- /dev/null
> > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > > @@ -0,0 +1,8 @@
> > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> > > > > +Date:          November 2017
> > > > > +KernelVersion:  4.15
> > > > > +Contact:       Wu Hao <hao.wu@intel.com>
> > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> > > > > +               hardware. Userspace could use this information to check if
> > > > > +               current hardware is compatible with given image before FPGA
> > > > > +               programming.
> > > > 
> > > > I'm a little confused by this.  I can understand that the PR bitstream
> > > > has a dependency on the FPGA's static image, but I don't understand
> > > > the dependency of the bistream on the hardware that is used to program
> > > > the bitstream to the FPGA.
> > > 
> > > Sorry for the confusion, the interface_id is used to indicate the version of
> > > the hardware for partial reconfiguration (it's part of the static image of
> > > the FPGA device). Will improve the description on this.
> 
> I'm not sure userland should be making the call on whether what you're
> trying to load is compatible or not. Isn't there a way to check this in
> your PR reconfiguration handler in-kernel?

Hi Moritz

Actually with current driver interface, doing a partial reconfiguration with an
incompatible image, then driver will report PR failure with error code
FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR as hardware checks it, but anyway user
needs to know hardware interface_id information to find or re-generated correct
images. I think it's more flexible to leave it to userspace on using this
information exposed by driver. : )

Thanks
Hao

> 
> > > 
> > 
> > The interface_id expresses the compatibility of the static region with PR
> > bitstreams generated for it. It changes every time a new static region is
> > generated.
> > 
> > Would it make more sense to have the interface_id exposed as part of the FME
> > device (which represents the static region)? I'm not sure - it kind of also
> > makes sense here, where you would have all the information in one place (if the
> > interface_id matches, I can use this component to program a bitstream).
> > 
> > Sorry for my limited understanding of the infrastructure - would this same
> > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
> > it would need to expose multiple interface_ids (or we'd have to track both
> > interface IDs and an identifier for the target PR region).
> > 
> > > > 
> > > > > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > > > > index 57da904..0171ecb 100644
> > > > > --- a/drivers/fpga/Kconfig
> > > > > +++ b/drivers/fpga/Kconfig
> > > > > @@ -150,6 +150,12 @@ config FPGA_DFL_FME
> > > > >           FPGA platform level management features. There shall be 1 FME
> > > > >           per DFL based FPGA device.
> > > > >
> > > > > +config FPGA_DFL_FME_MGR
> > > > > +       tristate "FPGA DFL FME Manager Driver"
> > > > > +       depends on FPGA_DFL_FME
> > > > > +       help
> > > > > +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
> > > > > +
> > > > >  config INTEL_FPGA_DFL_PCI
> > > > >         tristate "Intel FPGA DFL PCIe Device Driver"
> > > > >         depends on PCI && FPGA_DFL
> > > > > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > > > > index cc75bb3..6378580 100644
> > > > > --- a/drivers/fpga/Makefile
> > > > > +++ b/drivers/fpga/Makefile
> > > > > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> > > > >  # FPGA Device Feature List Support
> > > > >  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
> > > > >  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
> > > > > +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
> > > > >
> > > > >  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
> > > > >
> > > > > diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > > new file mode 100644
> > > > > index 0000000..70356ce
> > > > > --- /dev/null
> > > > > +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > > @@ -0,0 +1,318 @@
> > > > > +/*
> > > > > + * FPGA Manager Driver for FPGA Management Engine (FME)
> > > > > + *
> > > > > + * Copyright (C) 2017 Intel Corporation, Inc.
> > > > > + *
> > > > > + * Authors:
> > > > > + *   Kang Luwei <luwei.kang@intel.com>
> > > > > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > > + *   Wu Hao <hao.wu@intel.com>
> > > > > + *   Joseph Grecco <joe.grecco@intel.com>
> > > > > + *   Enno Luebbers <enno.luebbers@intel.com>
> > > > > + *   Tim Whisonant <tim.whisonant@intel.com>
> > > > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > > > > + *   Christopher Rauer <christopher.rauer@intel.com>
> > > > > + *   Henry Mitchel <henry.mitchel@intel.com>
> > > > > + *
> > > > > + * This work is licensed under the terms of the GNU GPL version 2.
> > > > > + * SPDX-License-Identifier: GPL-2.0
> > > > > + */
> > > > > +
> > > > > +#include <linux/module.h>
> > > > > +#include <linux/iopoll.h>
> > > > > +#include <linux/fpga/fpga-mgr.h>
> > > > > +
> > > > > +#include "fpga-dfl.h"
> > > > > +#include "dfl-fme.h"
> > > > > +
> > > > > +#define PR_WAIT_TIMEOUT   8000000
> > > > > +#define PR_HOST_STATUS_IDLE    0
> > > > > +
> > > > > +struct fme_mgr_priv {
> > > > > +       void __iomem *ioaddr;
> > > > > +       u64 pr_error;
> > > > > +};
> > > > > +
> > > > > +static ssize_t interface_id_show(struct device *dev,
> > > > > +                                struct device_attribute *attr, char *buf)
> > > > > +{
> > > > > +       struct fpga_manager *mgr = dev_get_drvdata(dev);
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +       u64 intfc_id_l, intfc_id_h;
> > > > > +
> > > > > +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
> > > > > +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
> > > > > +
> > > > > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > > > > +                       (unsigned long long)intfc_id_h,
> > > > > +                       (unsigned long long)intfc_id_l);
> > > > > +}
> > > > > +static DEVICE_ATTR_RO(interface_id);
> > > > > +
> > > > > +static const struct attribute *fme_mgr_attrs[] = {
> > > > > +       &dev_attr_interface_id.attr,
> > > > > +       NULL,
> > > > > +};
> > > > > +
> > > > > +static u64 pr_error_to_mgr_status(u64 err)
> > > > > +{
> > > > > +       u64 status = 0;
> > > > > +
> > > > > +       if (err & FME_PR_ERR_OPERATION_ERR)
> > > > > +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
> > > > > +       if (err & FME_PR_ERR_CRC_ERR)
> > > > > +               status |= FPGA_MGR_STATUS_CRC_ERR;
> > > > > +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
> > > > > +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
> > > > > +       if (err & FME_PR_ERR_PROTOCOL_ERR)
> > > > > +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> > > > > +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
> > > > > +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> > > > > +
> > > > > +       return status;
> > > > > +}
> > > > > +
> > > > > +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
> > > > > +{
> > > > > +       u64 pr_status, pr_error;
> > > > > +
> > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > > +       if (!(pr_status & FME_PR_STS_PR_STS))
> > > > > +               return 0;
> > > > > +
> > > > > +       pr_error = readq(fme_pr + FME_PR_ERR);
> > > > > +       writeq(pr_error, fme_pr + FME_PR_ERR);
> > > > > +
> > > > > +       return pr_error;
> > > > > +}
> > > > > +
> > > > > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> > > > > +                             struct fpga_image_info *info,
> > > > > +                             const char *buf, size_t count)
> > > > > +{
> > > > > +       struct device *dev = &mgr->dev;
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > > +       u64 pr_ctrl, pr_status;
> > > > > +
> > > > > +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> > > > > +               dev_err(dev, "only support partial reconfiguration.\n");
> > > > 
> > > > supports
> > > > 
> > > > > +               return -EINVAL;
> > > > > +       }
> > > > > +
> > > > > +       dev_dbg(dev, "resetting PR before initiated PR\n");
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_RST;
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > > +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
> > > > > +                              PR_WAIT_TIMEOUT)) {
> > > > > +               dev_err(dev, "maximum PR timeout\n");
> > > > 
> > > > We have two dev_err's with the same "maximum PR timeout" so the user
> > > > would not be able to see at which point the timeout occurred.
> > > > 
> > > > I suggest to get rid of the word 'maximum' for both.  This one could
> > > > be 'reset ack timeout" or something similar.
> > > 
> > > Sure, will switch to a more specific message per your suggestion. Thanks.
> > > 
> > > > 
> > > > > +               return -ETIMEDOUT;
> > > > > +       }
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       dev_dbg(dev,
> > > > > +               "waiting for PR resource in HW to be initialized and ready\n");
> > > > > +
> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
> > > > > +                              (pr_status & FME_PR_STS_PR_STS) ==
> > > > > +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
> > > > > +               dev_err(dev, "maximum PR timeout\n");
> > > > 
> > > > "PR_STS timeout"?  Or something better.
> > > > 
> > > > > +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > > +               return -ETIMEDOUT;
> > > > > +       }
> > > > > +
> > > > > +       dev_dbg(dev, "check and clear previous PR error\n");
> > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > > +       if (priv->pr_error)
> > > > > +               dev_dbg(dev, "previous PR error detected %llx\n",
> > > > > +                       (unsigned long long)priv->pr_error);
> > > > > +
> > > > > +       dev_dbg(dev, "set PR port ID\n");
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
> > > > > +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +
> > > > > +static int fme_mgr_write(struct fpga_manager *mgr,
> > > > > +                        const char *buf, size_t count)
> > > > > +{
> > > > > +       struct device *dev = &mgr->dev;
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > > +       u64 pr_ctrl, pr_status, pr_data;
> > > > > +       int delay = 0, pr_credit, i = 0;
> > > > > +
> > > > > +       dev_dbg(dev, "start request\n");
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_START;
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       dev_dbg(dev, "pushing data from bitstream to HW\n");
> > > > > +
> > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > > +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > > +
> > > > > +       while (count > 0) {
> > > > > +               while (pr_credit <= 1) {
> > > > > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > > > > +                               dev_err(dev, "maximum try\n");
> > > > 
> > > > It looks like this is waiting for an entry in a queue and timing out
> > > > here.  Could you add a some comments for pr_credit above and this
> > > > loop?  Also a better error message, perhaps "PR credit timeout"?
> > > 
> > > Driver needs to read the PR credit to know if it could push PR data
> > > to hardware or not. I will add more description here on this PR credit,
> > > and use "PR credit timeout" as error message.
> > > 
> > > > 
> > > > > +                               return -ETIMEDOUT;
> > > > > +                       }
> > > > > +                       udelay(1);
> > > > > +
> > > > > +                       pr_status = readq(fme_pr + FME_PR_STS);
> > > > > +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > > +               }
> > > > > +
> > > > > +               if (count >= 4) {
> > > > > +                       pr_data = 0;
> > > > > +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> > > > > +                                             *(((u32 *)buf) + i));
> > > > > +                       writeq(pr_data, fme_pr + FME_PR_DATA);
> > > > > +                       count -= 4;
> > > > > +                       pr_credit--;
> > > > > +                       i++;
> > > > > +               } else {
> > > > > +                       WARN_ON(1);
> > > > > +                       return -EINVAL;
> > > > > +               }
> > > > > +       }
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +
> > > > > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> > > > > +                                 struct fpga_image_info *info)
> > > > > +{
> > > > > +       struct device *dev = &mgr->dev;
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > > +       u64 pr_ctrl;
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       dev_dbg(dev, "green bitstream push complete\n");
> > > > > +       dev_dbg(dev, "waiting for HW to release PR resource\n");
> > > > > +
> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > > +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
> > > > > +                              PR_WAIT_TIMEOUT)) {
> > > > > +               dev_err(dev, "maximum try.\n");
> > > > 
> > > > Some message specific to here also, please.
> > > > 
> > > > > +               return -ETIMEDOUT;
> > > > > +       }
> > > > > +
> > > > > +       dev_dbg(dev, "PR operation complete, checking status\n");
> > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > > +       if (priv->pr_error) {
> > > > > +               dev_dbg(dev, "PR error detected %llx\n",
> > > > > +                       (unsigned long long)priv->pr_error);
> > > > > +               return -EIO;
> > > > > +       }
> > > > > +
> > > > > +       dev_dbg(dev, "PR done successfully\n");
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +
> > > > > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> > > > > +{
> > > > > +       return FPGA_MGR_STATE_UNKNOWN;
> > > > > +}
> > > > > +
> > > > > +static u64 fme_mgr_status(struct fpga_manager *mgr)
> > > > > +{
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +
> > > > > +       return pr_error_to_mgr_status(priv->pr_error);
> > > > > +}
> > > > > +
> > > > > +static const struct fpga_manager_ops fme_mgr_ops = {
> > > > > +       .write_init = fme_mgr_write_init,
> > > > > +       .write = fme_mgr_write,
> > > > > +       .write_complete = fme_mgr_write_complete,
> > > > > +       .state = fme_mgr_state,
> > > > > +       .status = fme_mgr_status,
> > > > > +};
> > > > > +
> > > > > +static int fme_mgr_probe(struct platform_device *pdev)
> > > > > +{
> > > > > +       struct device *dev = &pdev->dev;
> > > > > +       struct fme_mgr_priv *priv;
> > > > > +       struct fpga_manager *mgr;
> > > > > +       struct resource *res;
> > > > > +       int ret;
> > > > > +
> > > > > +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> > > > > +       if (!priv)
> > > > > +               return -ENOMEM;
> > > > > +
> > > > > +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> > > > > +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> > > > > +       if (IS_ERR(priv->ioaddr))
> > > > > +               return PTR_ERR(priv->ioaddr);
> > > > > +
> > > > > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > > +       if (ret)
> > > > > +               return ret;
> > > > > +
> > > > > +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
> > > > > +       if (!mgr)
> > > > > +               goto sysfs_remove_exit;
> > > > > +
> > > > > +       mgr->name = "DFL FPGA Manager";
> > > > > +       mgr->mops = &fme_mgr_ops;
> > > > > +       mgr->priv = priv;
> > > > > +       mgr->parent = dev;
> > > > > +       platform_set_drvdata(pdev, mgr);
> > > > > +
> > > > > +       ret = fpga_mgr_register(mgr);
> > > > > +       if (ret) {
> > > > > +               dev_err(dev, "unable to register FPGA manager\n");
> > > > > +               goto sysfs_remove_exit;
> > > > > +       }
> > > > > +
> > > > > +       return 0;
> > > > > +
> > > > > +sysfs_remove_exit:
> > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > > +       return ret;
> > > > > +}
> > > > > +
> > > > > +static int fme_mgr_remove(struct platform_device *pdev)
> > > > > +{
> > > > > +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
> > > > > +
> > > > > +       fpga_mgr_unregister(mgr);
> > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +
> > > > > +static struct platform_driver fme_mgr_driver = {
> > > > > +       .driver = {
> > > > > +               .name    = FPGA_DFL_FME_MGR,
> > > > > +       },
> > > > > +       .probe   = fme_mgr_probe,
> > > > > +       .remove  = fme_mgr_remove,
> > > > > +};
> > > > > +
> > > > > +module_platform_driver(fme_mgr_driver);
> > > > > +
> > > > > +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
> > > > > +MODULE_AUTHOR("Intel Corporation");
> > > > > +MODULE_LICENSE("GPL v2");
> > > > > +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
> > > > > diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> > > > > index e5a1094..d45eb82 100644
> > > > > --- a/drivers/fpga/fpga-dfl.h
> > > > > +++ b/drivers/fpga/fpga-dfl.h
> > > > > @@ -130,7 +130,44 @@
> > > > >
> > > > >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > > > >  #define FME_PR_DFH             DFH
> > > > > -#define FME_PR_SIZE            DFH_SIZE
> > > > > +#define FME_PR_CTRL            0x8
> > > > > +#define FME_PR_STS             0x10
> > > > > +#define FME_PR_DATA            0x18
> > > > > +#define FME_PR_ERR             0x20
> > > > > +#define FME_PR_INTFC_ID_H      0xA8
> > > > > +#define FME_PR_INTFC_ID_L      0xB0
> > > > > +#define FME_PR_SIZE            0xB8
> > > > > +
> > > > > +/* FME PR Control Register Bitfield */
> > > > > +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
> > > > > +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
> > > > > +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
> > > > > +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
> > > > > +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
> > > > > +
> > > > > +/* FME PR Status Register Bitfield */
> > > > > +/* Number of available entries in HW queue inside the PR engine. */
> > > > > +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
> > > > > +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
> > > > > +#define FME_PR_STS_PR_STS_IDLE 0
> > > > > +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
> > > > > +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
> > > > > +
> > > > > +/* FME PR Data Register Bitfield */
> > > > > +/* PR data from the raw-binary file. */
> > > > > +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
> > > > > +
> > > > > +/* FME PR Error Register */
> > > > > +/* Previous PR Operation errors detected. */
> > > > > +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
> > > > > +/* CRC error detected. */
> > > > > +#define FME_PR_ERR_CRC_ERR             BIT(1)
> > > > > +/* Incompatible PR bitstream detected. */
> > > > > +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
> > > > > +/* PR data push protocol violated. */
> > > > > +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
> > > > > +/* PR data fifo overflow error detected */
> > > > > +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
> > > > >
> > > > >  /* FME HSSI Sub Feature Register Set */
> > > > >  #define FME_HSSI_DFH           DFH
> > > > 
> > > > I see fpga-dfl.h as enumeration code which is separate from any driver
> > > > implementation specifics other than what's required for the DFL
> > > > enumeration scheme.    These PR engine #defines should move to a .h
> > > > that is dedicated to this specific PR hardware device. If someone else
> > > > adds a different PR device to the framework, their PR driver would
> > > > also have its own .h.  Same for any other modules that aren't central
> > > > to DFL enumeration.
> > > 
> > > Sure, will move PR related register to a separated header file.
> > > DFL enumeration related registers, will still be kept in fpga-dfl.h.
> > > 
> > > Thanks
> > > Hao
> > > 
> > > > 
> > > > Thanks,
> > > > Alan
> > > > 
> > > > > --
> > > > > 1.8.3.1
> > > > >
> 
> Thanks,
> Moritz

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Alan Tull]

On Sun, Feb 4, 2018 at 4:05 AM, Wu Hao <hao.wu@intel.com> wrote:
> On Sat, Feb 03, 2018 at 11:41:24AM +0100, Moritz Fischer wrote:
>> Hi Hao,
>>
>> On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
>> > Hi Hao, Alan,
>> >
>> > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
>> > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
>> > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
>> > > >
>> > > > Hi Hao,
>> > > >
>> > > > A few comments below.   Besides that, looks good.
>> > > >
>> > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
>> > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
>> > > > >
>> > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > > > > ----
>> > > > > v3: rename driver to dfl-fpga-fme-mgr
>> > > > >     implemented status callback for fpga manager
>> > > > >     rebased due to fpga api changes
>> > > > > ---
>> > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
>> > > > >  drivers/fpga/Kconfig                               |   6 +
>> > > > >  drivers/fpga/Makefile                              |   1 +
>> > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
>> > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
>> > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
>> > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
>> > > > >
>> > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> > > > > new file mode 100644
>> > > > > index 0000000..2d4f917
>> > > > > --- /dev/null
>> > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> > > > > @@ -0,0 +1,8 @@
>> > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
>> > > > > +Date:          November 2017
>> > > > > +KernelVersion:  4.15
>> > > > > +Contact:       Wu Hao <hao.wu@intel.com>
>> > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
>> > > > > +               hardware. Userspace could use this information to check if
>> > > > > +               current hardware is compatible with given image before FPGA
>> > > > > +               programming.
>> > > >
>> > > > I'm a little confused by this.  I can understand that the PR bitstream
>> > > > has a dependency on the FPGA's static image, but I don't understand
>> > > > the dependency of the bistream on the hardware that is used to program
>> > > > the bitstream to the FPGA.
>> > >
>> > > Sorry for the confusion, the interface_id is used to indicate the version of
>> > > the hardware for partial reconfiguration (it's part of the static image of
>> > > the FPGA device). Will improve the description on this.
>>
>> I'm not sure userland should be making the call on whether what you're
>> trying to load is compatible or not.

Could you explain more about what your concern was about this (unless
Hao has covered it below)?

It makes sense to me in this use case at least since userspace has a
pile of images and is choosing which one to load.

>> Isn't there a way to check this in
>> your PR reconfiguration handler in-kernel?
>
> Hi Moritz
>
> Actually with current driver interface, doing a partial reconfiguration with an
> incompatible image, then driver will report PR failure with error code
> FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR as hardware checks it, but anyway user
> needs to know hardware interface_id information to find or re-generated correct
> images. I think it's more flexible to leave it to userspace on using this
> information exposed by driver. : )
>
> Thanks
> Hao
>
>>
>> > >
>> >
>> > The interface_id expresses the compatibility of the static region with PR
>> > bitstreams generated for it. It changes every time a new static region is
>> > generated.

In the near future the DFL framework will be used with SoC's that have
a hard FPGA PR manager (that's not part of the static region).  The
hard FPGA manager driver won't know anything about the static region.

>> >
>> > Would it make more sense to have the interface_id exposed as part of the FME
>> > device (which represents the static region)? I'm not sure - it kind of also
>> > makes sense here, where you would have all the information in one place (if the
>> > interface_id matches, I can use this component to program a bitstream).

According to the intel-fpga.txt document, the identifier for the
static image is at

/sys/class/fpga_region/regionX/fpga-dfl-fme.n/bitstream_id

>> >
>> > Sorry for my limited understanding of the infrastructure - would this same
>> > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
>> > it would need to expose multiple interface_ids (or we'd have to track both
>> > interface IDs and an identifier for the target PR region).

interface_id sounds like it's trying to give some per-PR region
information.  That could support the real case where a FPGA static
region has multiple PR regions and userspace has a bunch of images.
The images are keyed to certain regions.  The reason the images are
region-specific could be that the regions have different connections
(possible but I think that's unlikely.  but possible) or because the
FPGA doesn't support relocatable PR images (true of most FPGAs that
support PR, but not a problem with Stratix10).

Each interface_ids could be exposed per fpga-region (each fpga-region
maps to a PR region).  That wouldn't be hard to implement, driver-wise
I think.

>> >
>> > > >
>> > > > > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
>> > > > > index 57da904..0171ecb 100644
>> > > > > --- a/drivers/fpga/Kconfig
>> > > > > +++ b/drivers/fpga/Kconfig
>> > > > > @@ -150,6 +150,12 @@ config FPGA_DFL_FME
>> > > > >           FPGA platform level management features. There shall be 1 FME
>> > > > >           per DFL based FPGA device.
>> > > > >
>> > > > > +config FPGA_DFL_FME_MGR
>> > > > > +       tristate "FPGA DFL FME Manager Driver"
>> > > > > +       depends on FPGA_DFL_FME
>> > > > > +       help
>> > > > > +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
>> > > > > +
>> > > > >  config INTEL_FPGA_DFL_PCI
>> > > > >         tristate "Intel FPGA DFL PCIe Device Driver"
>> > > > >         depends on PCI && FPGA_DFL
>> > > > > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
>> > > > > index cc75bb3..6378580 100644
>> > > > > --- a/drivers/fpga/Makefile
>> > > > > +++ b/drivers/fpga/Makefile
>> > > > > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
>> > > > >  # FPGA Device Feature List Support
>> > > > >  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
>> > > > >  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
>> > > > > +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
>> > > > >
>> > > > >  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
>> > > > >
>> > > > > diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
>> > > > > new file mode 100644
>> > > > > index 0000000..70356ce
>> > > > > --- /dev/null
>> > > > > +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
>> > > > > @@ -0,0 +1,318 @@
>> > > > > +/*
>> > > > > + * FPGA Manager Driver for FPGA Management Engine (FME)
>> > > > > + *
>> > > > > + * Copyright (C) 2017 Intel Corporation, Inc.
>> > > > > + *
>> > > > > + * Authors:
>> > > > > + *   Kang Luwei <luwei.kang@intel.com>
>> > > > > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > > > > + *   Wu Hao <hao.wu@intel.com>
>> > > > > + *   Joseph Grecco <joe.grecco@intel.com>
>> > > > > + *   Enno Luebbers <enno.luebbers@intel.com>
>> > > > > + *   Tim Whisonant <tim.whisonant@intel.com>
>> > > > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
>> > > > > + *   Christopher Rauer <christopher.rauer@intel.com>
>> > > > > + *   Henry Mitchel <henry.mitchel@intel.com>
>> > > > > + *
>> > > > > + * This work is licensed under the terms of the GNU GPL version 2.
>> > > > > + * SPDX-License-Identifier: GPL-2.0
>> > > > > + */
>> > > > > +
>> > > > > +#include <linux/module.h>
>> > > > > +#include <linux/iopoll.h>
>> > > > > +#include <linux/fpga/fpga-mgr.h>
>> > > > > +
>> > > > > +#include "fpga-dfl.h"
>> > > > > +#include "dfl-fme.h"
>> > > > > +
>> > > > > +#define PR_WAIT_TIMEOUT   8000000
>> > > > > +#define PR_HOST_STATUS_IDLE    0
>> > > > > +
>> > > > > +struct fme_mgr_priv {
>> > > > > +       void __iomem *ioaddr;
>> > > > > +       u64 pr_error;
>> > > > > +};
>> > > > > +
>> > > > > +static ssize_t interface_id_show(struct device *dev,
>> > > > > +                                struct device_attribute *attr, char *buf)
>> > > > > +{
>> > > > > +       struct fpga_manager *mgr = dev_get_drvdata(dev);
>> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
>> > > > > +       u64 intfc_id_l, intfc_id_h;
>> > > > > +
>> > > > > +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
>> > > > > +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
>> > > > > +
>> > > > > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
>> > > > > +                       (unsigned long long)intfc_id_h,
>> > > > > +                       (unsigned long long)intfc_id_l);
>> > > > > +}
>> > > > > +static DEVICE_ATTR_RO(interface_id);
>> > > > > +
>> > > > > +static const struct attribute *fme_mgr_attrs[] = {
>> > > > > +       &dev_attr_interface_id.attr,
>> > > > > +       NULL,
>> > > > > +};
>> > > > > +
>> > > > > +static u64 pr_error_to_mgr_status(u64 err)
>> > > > > +{
>> > > > > +       u64 status = 0;
>> > > > > +
>> > > > > +       if (err & FME_PR_ERR_OPERATION_ERR)
>> > > > > +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
>> > > > > +       if (err & FME_PR_ERR_CRC_ERR)
>> > > > > +               status |= FPGA_MGR_STATUS_CRC_ERR;
>> > > > > +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
>> > > > > +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
>> > > > > +       if (err & FME_PR_ERR_PROTOCOL_ERR)
>> > > > > +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
>> > > > > +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
>> > > > > +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
>> > > > > +
>> > > > > +       return status;
>> > > > > +}
>> > > > > +
>> > > > > +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
>> > > > > +{
>> > > > > +       u64 pr_status, pr_error;
>> > > > > +
>> > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
>> > > > > +       if (!(pr_status & FME_PR_STS_PR_STS))
>> > > > > +               return 0;
>> > > > > +
>> > > > > +       pr_error = readq(fme_pr + FME_PR_ERR);
>> > > > > +       writeq(pr_error, fme_pr + FME_PR_ERR);
>> > > > > +
>> > > > > +       return pr_error;
>> > > > > +}
>> > > > > +
>> > > > > +static int fme_mgr_write_init(struct fpga_manager *mgr,
>> > > > > +                             struct fpga_image_info *info,
>> > > > > +                             const char *buf, size_t count)
>> > > > > +{
>> > > > > +       struct device *dev = &mgr->dev;
>> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
>> > > > > +       void __iomem *fme_pr = priv->ioaddr;
>> > > > > +       u64 pr_ctrl, pr_status;
>> > > > > +
>> > > > > +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
>> > > > > +               dev_err(dev, "only support partial reconfiguration.\n");
>> > > >
>> > > > supports
>> > > >
>> > > > > +               return -EINVAL;
>> > > > > +       }
>> > > > > +
>> > > > > +       dev_dbg(dev, "resetting PR before initiated PR\n");
>> > > > > +
>> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
>> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_RST;
>> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
>> > > > > +
>> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
>> > > > > +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
>> > > > > +                              PR_WAIT_TIMEOUT)) {
>> > > > > +               dev_err(dev, "maximum PR timeout\n");
>> > > >
>> > > > We have two dev_err's with the same "maximum PR timeout" so the user
>> > > > would not be able to see at which point the timeout occurred.
>> > > >
>> > > > I suggest to get rid of the word 'maximum' for both.  This one could
>> > > > be 'reset ack timeout" or something similar.
>> > >
>> > > Sure, will switch to a more specific message per your suggestion. Thanks.
>> > >
>> > > >
>> > > > > +               return -ETIMEDOUT;
>> > > > > +       }
>> > > > > +
>> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
>> > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
>> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
>> > > > > +
>> > > > > +       dev_dbg(dev,
>> > > > > +               "waiting for PR resource in HW to be initialized and ready\n");
>> > > > > +
>> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
>> > > > > +                              (pr_status & FME_PR_STS_PR_STS) ==
>> > > > > +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
>> > > > > +               dev_err(dev, "maximum PR timeout\n");
>> > > >
>> > > > "PR_STS timeout"?  Or something better.
>> > > >
>> > > > > +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
>> > > > > +               return -ETIMEDOUT;
>> > > > > +       }
>> > > > > +
>> > > > > +       dev_dbg(dev, "check and clear previous PR error\n");
>> > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
>> > > > > +       if (priv->pr_error)
>> > > > > +               dev_dbg(dev, "previous PR error detected %llx\n",
>> > > > > +                       (unsigned long long)priv->pr_error);
>> > > > > +
>> > > > > +       dev_dbg(dev, "set PR port ID\n");
>> > > > > +
>> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
>> > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
>> > > > > +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
>> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
>> > > > > +
>> > > > > +       return 0;
>> > > > > +}
>> > > > > +
>> > > > > +static int fme_mgr_write(struct fpga_manager *mgr,
>> > > > > +                        const char *buf, size_t count)
>> > > > > +{
>> > > > > +       struct device *dev = &mgr->dev;
>> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
>> > > > > +       void __iomem *fme_pr = priv->ioaddr;
>> > > > > +       u64 pr_ctrl, pr_status, pr_data;
>> > > > > +       int delay = 0, pr_credit, i = 0;
>> > > > > +
>> > > > > +       dev_dbg(dev, "start request\n");
>> > > > > +
>> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
>> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_START;
>> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
>> > > > > +
>> > > > > +       dev_dbg(dev, "pushing data from bitstream to HW\n");
>> > > > > +
>> > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
>> > > > > +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
>> > > > > +
>> > > > > +       while (count > 0) {
>> > > > > +               while (pr_credit <= 1) {
>> > > > > +                       if (delay++ > PR_WAIT_TIMEOUT) {
>> > > > > +                               dev_err(dev, "maximum try\n");
>> > > >
>> > > > It looks like this is waiting for an entry in a queue and timing out
>> > > > here.  Could you add a some comments for pr_credit above and this
>> > > > loop?  Also a better error message, perhaps "PR credit timeout"?
>> > >
>> > > Driver needs to read the PR credit to know if it could push PR data
>> > > to hardware or not. I will add more description here on this PR credit,
>> > > and use "PR credit timeout" as error message.
>> > >
>> > > >
>> > > > > +                               return -ETIMEDOUT;
>> > > > > +                       }
>> > > > > +                       udelay(1);
>> > > > > +
>> > > > > +                       pr_status = readq(fme_pr + FME_PR_STS);
>> > > > > +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
>> > > > > +               }
>> > > > > +
>> > > > > +               if (count >= 4) {
>> > > > > +                       pr_data = 0;
>> > > > > +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
>> > > > > +                                             *(((u32 *)buf) + i));
>> > > > > +                       writeq(pr_data, fme_pr + FME_PR_DATA);
>> > > > > +                       count -= 4;
>> > > > > +                       pr_credit--;
>> > > > > +                       i++;
>> > > > > +               } else {
>> > > > > +                       WARN_ON(1);
>> > > > > +                       return -EINVAL;
>> > > > > +               }
>> > > > > +       }
>> > > > > +
>> > > > > +       return 0;
>> > > > > +}
>> > > > > +
>> > > > > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
>> > > > > +                                 struct fpga_image_info *info)
>> > > > > +{
>> > > > > +       struct device *dev = &mgr->dev;
>> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
>> > > > > +       void __iomem *fme_pr = priv->ioaddr;
>> > > > > +       u64 pr_ctrl;
>> > > > > +
>> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
>> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
>> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
>> > > > > +
>> > > > > +       dev_dbg(dev, "green bitstream push complete\n");
>> > > > > +       dev_dbg(dev, "waiting for HW to release PR resource\n");
>> > > > > +
>> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
>> > > > > +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
>> > > > > +                              PR_WAIT_TIMEOUT)) {
>> > > > > +               dev_err(dev, "maximum try.\n");
>> > > >
>> > > > Some message specific to here also, please.
>> > > >
>> > > > > +               return -ETIMEDOUT;
>> > > > > +       }
>> > > > > +
>> > > > > +       dev_dbg(dev, "PR operation complete, checking status\n");
>> > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
>> > > > > +       if (priv->pr_error) {
>> > > > > +               dev_dbg(dev, "PR error detected %llx\n",
>> > > > > +                       (unsigned long long)priv->pr_error);
>> > > > > +               return -EIO;
>> > > > > +       }
>> > > > > +
>> > > > > +       dev_dbg(dev, "PR done successfully\n");
>> > > > > +
>> > > > > +       return 0;
>> > > > > +}
>> > > > > +
>> > > > > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
>> > > > > +{
>> > > > > +       return FPGA_MGR_STATE_UNKNOWN;
>> > > > > +}
>> > > > > +
>> > > > > +static u64 fme_mgr_status(struct fpga_manager *mgr)
>> > > > > +{
>> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
>> > > > > +
>> > > > > +       return pr_error_to_mgr_status(priv->pr_error);
>> > > > > +}
>> > > > > +
>> > > > > +static const struct fpga_manager_ops fme_mgr_ops = {
>> > > > > +       .write_init = fme_mgr_write_init,
>> > > > > +       .write = fme_mgr_write,
>> > > > > +       .write_complete = fme_mgr_write_complete,
>> > > > > +       .state = fme_mgr_state,
>> > > > > +       .status = fme_mgr_status,
>> > > > > +};
>> > > > > +
>> > > > > +static int fme_mgr_probe(struct platform_device *pdev)
>> > > > > +{
>> > > > > +       struct device *dev = &pdev->dev;
>> > > > > +       struct fme_mgr_priv *priv;
>> > > > > +       struct fpga_manager *mgr;
>> > > > > +       struct resource *res;
>> > > > > +       int ret;
>> > > > > +
>> > > > > +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
>> > > > > +       if (!priv)
>> > > > > +               return -ENOMEM;
>> > > > > +
>> > > > > +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> > > > > +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
>> > > > > +       if (IS_ERR(priv->ioaddr))
>> > > > > +               return PTR_ERR(priv->ioaddr);
>> > > > > +
>> > > > > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
>> > > > > +       if (ret)
>> > > > > +               return ret;
>> > > > > +
>> > > > > +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
>> > > > > +       if (!mgr)
>> > > > > +               goto sysfs_remove_exit;
>> > > > > +
>> > > > > +       mgr->name = "DFL FPGA Manager";
>> > > > > +       mgr->mops = &fme_mgr_ops;
>> > > > > +       mgr->priv = priv;
>> > > > > +       mgr->parent = dev;
>> > > > > +       platform_set_drvdata(pdev, mgr);
>> > > > > +
>> > > > > +       ret = fpga_mgr_register(mgr);
>> > > > > +       if (ret) {
>> > > > > +               dev_err(dev, "unable to register FPGA manager\n");
>> > > > > +               goto sysfs_remove_exit;
>> > > > > +       }
>> > > > > +
>> > > > > +       return 0;
>> > > > > +
>> > > > > +sysfs_remove_exit:
>> > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
>> > > > > +       return ret;
>> > > > > +}
>> > > > > +
>> > > > > +static int fme_mgr_remove(struct platform_device *pdev)
>> > > > > +{
>> > > > > +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
>> > > > > +
>> > > > > +       fpga_mgr_unregister(mgr);
>> > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
>> > > > > +
>> > > > > +       return 0;
>> > > > > +}
>> > > > > +
>> > > > > +static struct platform_driver fme_mgr_driver = {
>> > > > > +       .driver = {
>> > > > > +               .name    = FPGA_DFL_FME_MGR,
>> > > > > +       },
>> > > > > +       .probe   = fme_mgr_probe,
>> > > > > +       .remove  = fme_mgr_remove,
>> > > > > +};
>> > > > > +
>> > > > > +module_platform_driver(fme_mgr_driver);
>> > > > > +
>> > > > > +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
>> > > > > +MODULE_AUTHOR("Intel Corporation");
>> > > > > +MODULE_LICENSE("GPL v2");
>> > > > > +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
>> > > > > diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
>> > > > > index e5a1094..d45eb82 100644
>> > > > > --- a/drivers/fpga/fpga-dfl.h
>> > > > > +++ b/drivers/fpga/fpga-dfl.h
>> > > > > @@ -130,7 +130,44 @@
>> > > > >
>> > > > >  /* FME Partial Reconfiguration Sub Feature Register Set */
>> > > > >  #define FME_PR_DFH             DFH
>> > > > > -#define FME_PR_SIZE            DFH_SIZE
>> > > > > +#define FME_PR_CTRL            0x8
>> > > > > +#define FME_PR_STS             0x10
>> > > > > +#define FME_PR_DATA            0x18
>> > > > > +#define FME_PR_ERR             0x20
>> > > > > +#define FME_PR_INTFC_ID_H      0xA8
>> > > > > +#define FME_PR_INTFC_ID_L      0xB0
>> > > > > +#define FME_PR_SIZE            0xB8
>> > > > > +
>> > > > > +/* FME PR Control Register Bitfield */
>> > > > > +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
>> > > > > +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
>> > > > > +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
>> > > > > +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
>> > > > > +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
>> > > > > +
>> > > > > +/* FME PR Status Register Bitfield */
>> > > > > +/* Number of available entries in HW queue inside the PR engine. */
>> > > > > +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
>> > > > > +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
>> > > > > +#define FME_PR_STS_PR_STS_IDLE 0
>> > > > > +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
>> > > > > +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
>> > > > > +
>> > > > > +/* FME PR Data Register Bitfield */
>> > > > > +/* PR data from the raw-binary file. */
>> > > > > +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
>> > > > > +
>> > > > > +/* FME PR Error Register */
>> > > > > +/* Previous PR Operation errors detected. */
>> > > > > +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
>> > > > > +/* CRC error detected. */
>> > > > > +#define FME_PR_ERR_CRC_ERR             BIT(1)
>> > > > > +/* Incompatible PR bitstream detected. */
>> > > > > +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
>> > > > > +/* PR data push protocol violated. */
>> > > > > +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
>> > > > > +/* PR data fifo overflow error detected */
>> > > > > +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
>> > > > >
>> > > > >  /* FME HSSI Sub Feature Register Set */
>> > > > >  #define FME_HSSI_DFH           DFH
>> > > >
>> > > > I see fpga-dfl.h as enumeration code which is separate from any driver
>> > > > implementation specifics other than what's required for the DFL
>> > > > enumeration scheme.    These PR engine #defines should move to a .h
>> > > > that is dedicated to this specific PR hardware device. If someone else
>> > > > adds a different PR device to the framework, their PR driver would
>> > > > also have its own .h.  Same for any other modules that aren't central
>> > > > to DFL enumeration.
>> > >
>> > > Sure, will move PR related register to a separated header file.
>> > > DFL enumeration related registers, will still be kept in fpga-dfl.h.
>> > >
>> > > Thanks
>> > > Hao
>> > >
>> > > >
>> > > > Thanks,
>> > > > Alan
>> > > >
>> > > > > --
>> > > > > 1.8.3.1
>> > > > >
>>
>> Thanks,
>> Moritz
>
>

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Wu Hao]

On Mon, Feb 05, 2018 at 11:21:52AM -0600, Alan Tull wrote:
> On Sun, Feb 4, 2018 at 4:05 AM, Wu Hao <hao.wu@intel.com> wrote:
> > On Sat, Feb 03, 2018 at 11:41:24AM +0100, Moritz Fischer wrote:
> >> Hi Hao,
> >>
> >> On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> >> > Hi Hao, Alan,
> >> >
> >> > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> >> > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> >> > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> > > >
> >> > > > Hi Hao,
> >> > > >
> >> > > > A few comments below.   Besides that, looks good.
> >> > > >
> >> > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> >> > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> >> > > > >
> >> > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> >> > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >> > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> >> > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> >> > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> >> > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> >> > > > > ----
> >> > > > > v3: rename driver to dfl-fpga-fme-mgr
> >> > > > >     implemented status callback for fpga manager
> >> > > > >     rebased due to fpga api changes
> >> > > > > ---
> >> > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> >> > > > >  drivers/fpga/Kconfig                               |   6 +
> >> > > > >  drivers/fpga/Makefile                              |   1 +
> >> > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> >> > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> >> > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> >> > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> >> > > > >
> >> > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> > > > > new file mode 100644
> >> > > > > index 0000000..2d4f917
> >> > > > > --- /dev/null
> >> > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> > > > > @@ -0,0 +1,8 @@
> >> > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> >> > > > > +Date:          November 2017
> >> > > > > +KernelVersion:  4.15
> >> > > > > +Contact:       Wu Hao <hao.wu@intel.com>
> >> > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> >> > > > > +               hardware. Userspace could use this information to check if
> >> > > > > +               current hardware is compatible with given image before FPGA
> >> > > > > +               programming.
> >> > > >
> >> > > > I'm a little confused by this.  I can understand that the PR bitstream
> >> > > > has a dependency on the FPGA's static image, but I don't understand
> >> > > > the dependency of the bistream on the hardware that is used to program
> >> > > > the bitstream to the FPGA.
> >> > >
> >> > > Sorry for the confusion, the interface_id is used to indicate the version of
> >> > > the hardware for partial reconfiguration (it's part of the static image of
> >> > > the FPGA device). Will improve the description on this.
> >>
> >> I'm not sure userland should be making the call on whether what you're
> >> trying to load is compatible or not.
> 
> Could you explain more about what your concern was about this (unless
> Hao has covered it below)?
> 
> It makes sense to me in this use case at least since userspace has a
> pile of images and is choosing which one to load.
> 
> >> Isn't there a way to check this in
> >> your PR reconfiguration handler in-kernel?
> >
> > Hi Moritz
> >
> > Actually with current driver interface, doing a partial reconfiguration with an
> > incompatible image, then driver will report PR failure with error code
> > FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR as hardware checks it, but anyway user
> > needs to know hardware interface_id information to find or re-generated correct
> > images. I think it's more flexible to leave it to userspace on using this
> > information exposed by driver. : )
> >
> > Thanks
> > Hao
> >
> >>
> >> > >
> >> >
> >> > The interface_id expresses the compatibility of the static region with PR
> >> > bitstreams generated for it. It changes every time a new static region is
> >> > generated.
> 
> In the near future the DFL framework will be used with SoC's that have
> a hard FPGA PR manager (that's not part of the static region).  The
> hard FPGA manager driver won't know anything about the static region.
> 
> >> >
> >> > Would it make more sense to have the interface_id exposed as part of the FME
> >> > device (which represents the static region)? I'm not sure - it kind of also
> >> > makes sense here, where you would have all the information in one place (if the
> >> > interface_id matches, I can use this component to program a bitstream).
> 
> According to the intel-fpga.txt document, the identifier for the
> static image is at
> 
> /sys/class/fpga_region/regionX/fpga-dfl-fme.n/bitstream_id

Hi Alan

This bitstream_id refects the full static region version. As you know, PR is
only a sub feature of the FME functional unit, it's possible that we have
different static region (different bitstream_id) but it has the exact same
PR sub feature under the FME, only changes on other sub features or function
units.

The interface_id is used to indicate the PR hardware version and express the
compatibility of the PR hardware with PR image generated for it. It updates
whenever PR hardware has been changed. 

So we should use interface_id which is from the PR hardware Register for PR
compatibility check, not the bitstream_id of the full static region.

> 
> >> >
> >> > Sorry for my limited understanding of the infrastructure - would this same
> >> > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
> >> > it would need to expose multiple interface_ids (or we'd have to track both
> >> > interface IDs and an identifier for the target PR region).
> 
> interface_id sounds like it's trying to give some per-PR region
> information.  That could support the real case where a FPGA static
> region has multiple PR regions and userspace has a bunch of images.
> The images are keyed to certain regions.  The reason the images are
> region-specific could be that the regions have different connections
> (possible but I think that's unlikely.  but possible) or because the
> FPGA doesn't support relocatable PR images (true of most FPGAs that
> support PR, but not a problem with Stratix10).
> 
> Each interface_ids could be exposed per fpga-region (each fpga-region
> maps to a PR region).  That wouldn't be hard to implement, driver-wise
> I think.

I think that wouldn't be hard to implement, but actually in our case, FME PR
sub feature supports PR for multiple regions, only provide one interface id
register, so it's not per region interface_id for sure. But anyway, we could
add it whenever needed. :)

Thanks
Hao

> 
> >> >
> >> > > >
> >> > > > > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> >> > > > > index 57da904..0171ecb 100644
> >> > > > > --- a/drivers/fpga/Kconfig
> >> > > > > +++ b/drivers/fpga/Kconfig
> >> > > > > @@ -150,6 +150,12 @@ config FPGA_DFL_FME
> >> > > > >           FPGA platform level management features. There shall be 1 FME
> >> > > > >           per DFL based FPGA device.
> >> > > > >
> >> > > > > +config FPGA_DFL_FME_MGR
> >> > > > > +       tristate "FPGA DFL FME Manager Driver"
> >> > > > > +       depends on FPGA_DFL_FME
> >> > > > > +       help
> >> > > > > +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
> >> > > > > +
> >> > > > >  config INTEL_FPGA_DFL_PCI
> >> > > > >         tristate "Intel FPGA DFL PCIe Device Driver"
> >> > > > >         depends on PCI && FPGA_DFL
> >> > > > > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> >> > > > > index cc75bb3..6378580 100644
> >> > > > > --- a/drivers/fpga/Makefile
> >> > > > > +++ b/drivers/fpga/Makefile
> >> > > > > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> >> > > > >  # FPGA Device Feature List Support
> >> > > > >  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
> >> > > > >  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
> >> > > > > +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
> >> > > > >
> >> > > > >  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
> >> > > > >
> >> > > > > diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
> >> > > > > new file mode 100644
> >> > > > > index 0000000..70356ce
> >> > > > > --- /dev/null
> >> > > > > +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
> >> > > > > @@ -0,0 +1,318 @@
> >> > > > > +/*
> >> > > > > + * FPGA Manager Driver for FPGA Management Engine (FME)
> >> > > > > + *
> >> > > > > + * Copyright (C) 2017 Intel Corporation, Inc.
> >> > > > > + *
> >> > > > > + * Authors:
> >> > > > > + *   Kang Luwei <luwei.kang@intel.com>
> >> > > > > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> > > > > + *   Wu Hao <hao.wu@intel.com>
> >> > > > > + *   Joseph Grecco <joe.grecco@intel.com>
> >> > > > > + *   Enno Luebbers <enno.luebbers@intel.com>
> >> > > > > + *   Tim Whisonant <tim.whisonant@intel.com>
> >> > > > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> >> > > > > + *   Christopher Rauer <christopher.rauer@intel.com>
> >> > > > > + *   Henry Mitchel <henry.mitchel@intel.com>
> >> > > > > + *
> >> > > > > + * This work is licensed under the terms of the GNU GPL version 2.
> >> > > > > + * SPDX-License-Identifier: GPL-2.0
> >> > > > > + */
> >> > > > > +
> >> > > > > +#include <linux/module.h>
> >> > > > > +#include <linux/iopoll.h>
> >> > > > > +#include <linux/fpga/fpga-mgr.h>
> >> > > > > +
> >> > > > > +#include "fpga-dfl.h"
> >> > > > > +#include "dfl-fme.h"
> >> > > > > +
> >> > > > > +#define PR_WAIT_TIMEOUT   8000000
> >> > > > > +#define PR_HOST_STATUS_IDLE    0
> >> > > > > +
> >> > > > > +struct fme_mgr_priv {
> >> > > > > +       void __iomem *ioaddr;
> >> > > > > +       u64 pr_error;
> >> > > > > +};
> >> > > > > +
> >> > > > > +static ssize_t interface_id_show(struct device *dev,
> >> > > > > +                                struct device_attribute *attr, char *buf)
> >> > > > > +{
> >> > > > > +       struct fpga_manager *mgr = dev_get_drvdata(dev);
> >> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> >> > > > > +       u64 intfc_id_l, intfc_id_h;
> >> > > > > +
> >> > > > > +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
> >> > > > > +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
> >> > > > > +
> >> > > > > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> >> > > > > +                       (unsigned long long)intfc_id_h,
> >> > > > > +                       (unsigned long long)intfc_id_l);
> >> > > > > +}
> >> > > > > +static DEVICE_ATTR_RO(interface_id);
> >> > > > > +
> >> > > > > +static const struct attribute *fme_mgr_attrs[] = {
> >> > > > > +       &dev_attr_interface_id.attr,
> >> > > > > +       NULL,
> >> > > > > +};
> >> > > > > +
> >> > > > > +static u64 pr_error_to_mgr_status(u64 err)
> >> > > > > +{
> >> > > > > +       u64 status = 0;
> >> > > > > +
> >> > > > > +       if (err & FME_PR_ERR_OPERATION_ERR)
> >> > > > > +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
> >> > > > > +       if (err & FME_PR_ERR_CRC_ERR)
> >> > > > > +               status |= FPGA_MGR_STATUS_CRC_ERR;
> >> > > > > +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
> >> > > > > +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
> >> > > > > +       if (err & FME_PR_ERR_PROTOCOL_ERR)
> >> > > > > +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> >> > > > > +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
> >> > > > > +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> >> > > > > +
> >> > > > > +       return status;
> >> > > > > +}
> >> > > > > +
> >> > > > > +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
> >> > > > > +{
> >> > > > > +       u64 pr_status, pr_error;
> >> > > > > +
> >> > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> >> > > > > +       if (!(pr_status & FME_PR_STS_PR_STS))
> >> > > > > +               return 0;
> >> > > > > +
> >> > > > > +       pr_error = readq(fme_pr + FME_PR_ERR);
> >> > > > > +       writeq(pr_error, fme_pr + FME_PR_ERR);
> >> > > > > +
> >> > > > > +       return pr_error;
> >> > > > > +}
> >> > > > > +
> >> > > > > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> >> > > > > +                             struct fpga_image_info *info,
> >> > > > > +                             const char *buf, size_t count)
> >> > > > > +{
> >> > > > > +       struct device *dev = &mgr->dev;
> >> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> >> > > > > +       void __iomem *fme_pr = priv->ioaddr;
> >> > > > > +       u64 pr_ctrl, pr_status;
> >> > > > > +
> >> > > > > +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> >> > > > > +               dev_err(dev, "only support partial reconfiguration.\n");
> >> > > >
> >> > > > supports
> >> > > >
> >> > > > > +               return -EINVAL;
> >> > > > > +       }
> >> > > > > +
> >> > > > > +       dev_dbg(dev, "resetting PR before initiated PR\n");
> >> > > > > +
> >> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> >> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_RST;
> >> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> >> > > > > +
> >> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> >> > > > > +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
> >> > > > > +                              PR_WAIT_TIMEOUT)) {
> >> > > > > +               dev_err(dev, "maximum PR timeout\n");
> >> > > >
> >> > > > We have two dev_err's with the same "maximum PR timeout" so the user
> >> > > > would not be able to see at which point the timeout occurred.
> >> > > >
> >> > > > I suggest to get rid of the word 'maximum' for both.  This one could
> >> > > > be 'reset ack timeout" or something similar.
> >> > >
> >> > > Sure, will switch to a more specific message per your suggestion. Thanks.
> >> > >
> >> > > >
> >> > > > > +               return -ETIMEDOUT;
> >> > > > > +       }
> >> > > > > +
> >> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> >> > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
> >> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> >> > > > > +
> >> > > > > +       dev_dbg(dev,
> >> > > > > +               "waiting for PR resource in HW to be initialized and ready\n");
> >> > > > > +
> >> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
> >> > > > > +                              (pr_status & FME_PR_STS_PR_STS) ==
> >> > > > > +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
> >> > > > > +               dev_err(dev, "maximum PR timeout\n");
> >> > > >
> >> > > > "PR_STS timeout"?  Or something better.
> >> > > >
> >> > > > > +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> >> > > > > +               return -ETIMEDOUT;
> >> > > > > +       }
> >> > > > > +
> >> > > > > +       dev_dbg(dev, "check and clear previous PR error\n");
> >> > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> >> > > > > +       if (priv->pr_error)
> >> > > > > +               dev_dbg(dev, "previous PR error detected %llx\n",
> >> > > > > +                       (unsigned long long)priv->pr_error);
> >> > > > > +
> >> > > > > +       dev_dbg(dev, "set PR port ID\n");
> >> > > > > +
> >> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> >> > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
> >> > > > > +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
> >> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> >> > > > > +
> >> > > > > +       return 0;
> >> > > > > +}
> >> > > > > +
> >> > > > > +static int fme_mgr_write(struct fpga_manager *mgr,
> >> > > > > +                        const char *buf, size_t count)
> >> > > > > +{
> >> > > > > +       struct device *dev = &mgr->dev;
> >> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> >> > > > > +       void __iomem *fme_pr = priv->ioaddr;
> >> > > > > +       u64 pr_ctrl, pr_status, pr_data;
> >> > > > > +       int delay = 0, pr_credit, i = 0;
> >> > > > > +
> >> > > > > +       dev_dbg(dev, "start request\n");
> >> > > > > +
> >> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> >> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_START;
> >> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> >> > > > > +
> >> > > > > +       dev_dbg(dev, "pushing data from bitstream to HW\n");
> >> > > > > +
> >> > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> >> > > > > +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> >> > > > > +
> >> > > > > +       while (count > 0) {
> >> > > > > +               while (pr_credit <= 1) {
> >> > > > > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> >> > > > > +                               dev_err(dev, "maximum try\n");
> >> > > >
> >> > > > It looks like this is waiting for an entry in a queue and timing out
> >> > > > here.  Could you add a some comments for pr_credit above and this
> >> > > > loop?  Also a better error message, perhaps "PR credit timeout"?
> >> > >
> >> > > Driver needs to read the PR credit to know if it could push PR data
> >> > > to hardware or not. I will add more description here on this PR credit,
> >> > > and use "PR credit timeout" as error message.
> >> > >
> >> > > >
> >> > > > > +                               return -ETIMEDOUT;
> >> > > > > +                       }
> >> > > > > +                       udelay(1);
> >> > > > > +
> >> > > > > +                       pr_status = readq(fme_pr + FME_PR_STS);
> >> > > > > +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> >> > > > > +               }
> >> > > > > +
> >> > > > > +               if (count >= 4) {
> >> > > > > +                       pr_data = 0;
> >> > > > > +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> >> > > > > +                                             *(((u32 *)buf) + i));
> >> > > > > +                       writeq(pr_data, fme_pr + FME_PR_DATA);
> >> > > > > +                       count -= 4;
> >> > > > > +                       pr_credit--;
> >> > > > > +                       i++;
> >> > > > > +               } else {
> >> > > > > +                       WARN_ON(1);
> >> > > > > +                       return -EINVAL;
> >> > > > > +               }
> >> > > > > +       }
> >> > > > > +
> >> > > > > +       return 0;
> >> > > > > +}
> >> > > > > +
> >> > > > > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> >> > > > > +                                 struct fpga_image_info *info)
> >> > > > > +{
> >> > > > > +       struct device *dev = &mgr->dev;
> >> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> >> > > > > +       void __iomem *fme_pr = priv->ioaddr;
> >> > > > > +       u64 pr_ctrl;
> >> > > > > +
> >> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> >> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
> >> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> >> > > > > +
> >> > > > > +       dev_dbg(dev, "green bitstream push complete\n");
> >> > > > > +       dev_dbg(dev, "waiting for HW to release PR resource\n");
> >> > > > > +
> >> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> >> > > > > +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
> >> > > > > +                              PR_WAIT_TIMEOUT)) {
> >> > > > > +               dev_err(dev, "maximum try.\n");
> >> > > >
> >> > > > Some message specific to here also, please.
> >> > > >
> >> > > > > +               return -ETIMEDOUT;
> >> > > > > +       }
> >> > > > > +
> >> > > > > +       dev_dbg(dev, "PR operation complete, checking status\n");
> >> > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> >> > > > > +       if (priv->pr_error) {
> >> > > > > +               dev_dbg(dev, "PR error detected %llx\n",
> >> > > > > +                       (unsigned long long)priv->pr_error);
> >> > > > > +               return -EIO;
> >> > > > > +       }
> >> > > > > +
> >> > > > > +       dev_dbg(dev, "PR done successfully\n");
> >> > > > > +
> >> > > > > +       return 0;
> >> > > > > +}
> >> > > > > +
> >> > > > > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> >> > > > > +{
> >> > > > > +       return FPGA_MGR_STATE_UNKNOWN;
> >> > > > > +}
> >> > > > > +
> >> > > > > +static u64 fme_mgr_status(struct fpga_manager *mgr)
> >> > > > > +{
> >> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> >> > > > > +
> >> > > > > +       return pr_error_to_mgr_status(priv->pr_error);
> >> > > > > +}
> >> > > > > +
> >> > > > > +static const struct fpga_manager_ops fme_mgr_ops = {
> >> > > > > +       .write_init = fme_mgr_write_init,
> >> > > > > +       .write = fme_mgr_write,
> >> > > > > +       .write_complete = fme_mgr_write_complete,
> >> > > > > +       .state = fme_mgr_state,
> >> > > > > +       .status = fme_mgr_status,
> >> > > > > +};
> >> > > > > +
> >> > > > > +static int fme_mgr_probe(struct platform_device *pdev)
> >> > > > > +{
> >> > > > > +       struct device *dev = &pdev->dev;
> >> > > > > +       struct fme_mgr_priv *priv;
> >> > > > > +       struct fpga_manager *mgr;
> >> > > > > +       struct resource *res;
> >> > > > > +       int ret;
> >> > > > > +
> >> > > > > +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> >> > > > > +       if (!priv)
> >> > > > > +               return -ENOMEM;
> >> > > > > +
> >> > > > > +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> >> > > > > +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> >> > > > > +       if (IS_ERR(priv->ioaddr))
> >> > > > > +               return PTR_ERR(priv->ioaddr);
> >> > > > > +
> >> > > > > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> >> > > > > +       if (ret)
> >> > > > > +               return ret;
> >> > > > > +
> >> > > > > +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
> >> > > > > +       if (!mgr)
> >> > > > > +               goto sysfs_remove_exit;
> >> > > > > +
> >> > > > > +       mgr->name = "DFL FPGA Manager";
> >> > > > > +       mgr->mops = &fme_mgr_ops;
> >> > > > > +       mgr->priv = priv;
> >> > > > > +       mgr->parent = dev;
> >> > > > > +       platform_set_drvdata(pdev, mgr);
> >> > > > > +
> >> > > > > +       ret = fpga_mgr_register(mgr);
> >> > > > > +       if (ret) {
> >> > > > > +               dev_err(dev, "unable to register FPGA manager\n");
> >> > > > > +               goto sysfs_remove_exit;
> >> > > > > +       }
> >> > > > > +
> >> > > > > +       return 0;
> >> > > > > +
> >> > > > > +sysfs_remove_exit:
> >> > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> >> > > > > +       return ret;
> >> > > > > +}
> >> > > > > +
> >> > > > > +static int fme_mgr_remove(struct platform_device *pdev)
> >> > > > > +{
> >> > > > > +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
> >> > > > > +
> >> > > > > +       fpga_mgr_unregister(mgr);
> >> > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> >> > > > > +
> >> > > > > +       return 0;
> >> > > > > +}
> >> > > > > +
> >> > > > > +static struct platform_driver fme_mgr_driver = {
> >> > > > > +       .driver = {
> >> > > > > +               .name    = FPGA_DFL_FME_MGR,
> >> > > > > +       },
> >> > > > > +       .probe   = fme_mgr_probe,
> >> > > > > +       .remove  = fme_mgr_remove,
> >> > > > > +};
> >> > > > > +
> >> > > > > +module_platform_driver(fme_mgr_driver);
> >> > > > > +
> >> > > > > +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
> >> > > > > +MODULE_AUTHOR("Intel Corporation");
> >> > > > > +MODULE_LICENSE("GPL v2");
> >> > > > > +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
> >> > > > > diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> >> > > > > index e5a1094..d45eb82 100644
> >> > > > > --- a/drivers/fpga/fpga-dfl.h
> >> > > > > +++ b/drivers/fpga/fpga-dfl.h
> >> > > > > @@ -130,7 +130,44 @@
> >> > > > >
> >> > > > >  /* FME Partial Reconfiguration Sub Feature Register Set */
> >> > > > >  #define FME_PR_DFH             DFH
> >> > > > > -#define FME_PR_SIZE            DFH_SIZE
> >> > > > > +#define FME_PR_CTRL            0x8
> >> > > > > +#define FME_PR_STS             0x10
> >> > > > > +#define FME_PR_DATA            0x18
> >> > > > > +#define FME_PR_ERR             0x20
> >> > > > > +#define FME_PR_INTFC_ID_H      0xA8
> >> > > > > +#define FME_PR_INTFC_ID_L      0xB0
> >> > > > > +#define FME_PR_SIZE            0xB8
> >> > > > > +
> >> > > > > +/* FME PR Control Register Bitfield */
> >> > > > > +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
> >> > > > > +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
> >> > > > > +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
> >> > > > > +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
> >> > > > > +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
> >> > > > > +
> >> > > > > +/* FME PR Status Register Bitfield */
> >> > > > > +/* Number of available entries in HW queue inside the PR engine. */
> >> > > > > +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
> >> > > > > +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
> >> > > > > +#define FME_PR_STS_PR_STS_IDLE 0
> >> > > > > +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
> >> > > > > +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
> >> > > > > +
> >> > > > > +/* FME PR Data Register Bitfield */
> >> > > > > +/* PR data from the raw-binary file. */
> >> > > > > +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
> >> > > > > +
> >> > > > > +/* FME PR Error Register */
> >> > > > > +/* Previous PR Operation errors detected. */
> >> > > > > +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
> >> > > > > +/* CRC error detected. */
> >> > > > > +#define FME_PR_ERR_CRC_ERR             BIT(1)
> >> > > > > +/* Incompatible PR bitstream detected. */
> >> > > > > +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
> >> > > > > +/* PR data push protocol violated. */
> >> > > > > +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
> >> > > > > +/* PR data fifo overflow error detected */
> >> > > > > +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
> >> > > > >
> >> > > > >  /* FME HSSI Sub Feature Register Set */
> >> > > > >  #define FME_HSSI_DFH           DFH
> >> > > >
> >> > > > I see fpga-dfl.h as enumeration code which is separate from any driver
> >> > > > implementation specifics other than what's required for the DFL
> >> > > > enumeration scheme.    These PR engine #defines should move to a .h
> >> > > > that is dedicated to this specific PR hardware device. If someone else
> >> > > > adds a different PR device to the framework, their PR driver would
> >> > > > also have its own .h.  Same for any other modules that aren't central
> >> > > > to DFL enumeration.
> >> > >
> >> > > Sure, will move PR related register to a separated header file.
> >> > > DFL enumeration related registers, will still be kept in fpga-dfl.h.
> >> > >
> >> > > Thanks
> >> > > Hao
> >> > >
> >> > > >
> >> > > > Thanks,
> >> > > > Alan
> >> > > >
> >> > > > > --
> >> > > > > 1.8.3.1
> >> > > > >
> >>
> >> Thanks,
> >> Moritz
> >
> >

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Alan Tull]

On Mon, Feb 5, 2018 at 8:17 PM, Wu Hao <hao.wu@intel.com> wrote:
> On Mon, Feb 05, 2018 at 11:21:52AM -0600, Alan Tull wrote:
>> On Sun, Feb 4, 2018 at 4:05 AM, Wu Hao <hao.wu@intel.com> wrote:
>> > On Sat, Feb 03, 2018 at 11:41:24AM +0100, Moritz Fischer wrote:
>> >> Hi Hao,
>> >>
>> >> On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
>> >> > Hi Hao, Alan,
>> >> >
>> >> > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
>> >> > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
>> >> > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
>> >> > > >
>> >> > > > Hi Hao,
>> >> > > >
>> >> > > > A few comments below.   Besides that, looks good.
>> >> > > >
>> >> > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
>> >> > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
>> >> > > > >
>> >> > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> >> > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> >> > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> >> > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> >> > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> >> > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> >> > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> >> > > > > ----
>> >> > > > > v3: rename driver to dfl-fpga-fme-mgr
>> >> > > > >     implemented status callback for fpga manager
>> >> > > > >     rebased due to fpga api changes
>> >> > > > > ---
>> >> > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
>> >> > > > >  drivers/fpga/Kconfig                               |   6 +
>> >> > > > >  drivers/fpga/Makefile                              |   1 +
>> >> > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
>> >> > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
>> >> > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
>> >> > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> >> > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
>> >> > > > >
>> >> > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> >> > > > > new file mode 100644
>> >> > > > > index 0000000..2d4f917
>> >> > > > > --- /dev/null
>> >> > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> >> > > > > @@ -0,0 +1,8 @@
>> >> > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
>> >> > > > > +Date:          November 2017
>> >> > > > > +KernelVersion:  4.15
>> >> > > > > +Contact:       Wu Hao <hao.wu@intel.com>
>> >> > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
>> >> > > > > +               hardware. Userspace could use this information to check if
>> >> > > > > +               current hardware is compatible with given image before FPGA
>> >> > > > > +               programming.
>> >> > > >
>> >> > > > I'm a little confused by this.  I can understand that the PR bitstream
>> >> > > > has a dependency on the FPGA's static image, but I don't understand
>> >> > > > the dependency of the bistream on the hardware that is used to program
>> >> > > > the bitstream to the FPGA.
>> >> > >
>> >> > > Sorry for the confusion, the interface_id is used to indicate the version of
>> >> > > the hardware for partial reconfiguration (it's part of the static image of
>> >> > > the FPGA device). Will improve the description on this.
>> >>
>> >> I'm not sure userland should be making the call on whether what you're
>> >> trying to load is compatible or not.
>>
>> Could you explain more about what your concern was about this (unless
>> Hao has covered it below)?
>>
>> It makes sense to me in this use case at least since userspace has a
>> pile of images and is choosing which one to load.
>>
>> >> Isn't there a way to check this in
>> >> your PR reconfiguration handler in-kernel?
>> >
>> > Hi Moritz
>> >
>> > Actually with current driver interface, doing a partial reconfiguration with an
>> > incompatible image, then driver will report PR failure with error code
>> > FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR as hardware checks it, but anyway user
>> > needs to know hardware interface_id information to find or re-generated correct
>> > images. I think it's more flexible to leave it to userspace on using this
>> > information exposed by driver. : )
>> >
>> > Thanks
>> > Hao
>> >
>> >>
>> >> > >
>> >> >
>> >> > The interface_id expresses the compatibility of the static region with PR
>> >> > bitstreams generated for it. It changes every time a new static region is
>> >> > generated.
>>
>> In the near future the DFL framework will be used with SoC's that have
>> a hard FPGA PR manager (that's not part of the static region).  The
>> hard FPGA manager driver won't know anything about the static region.
>>
>> >> >
>> >> > Would it make more sense to have the interface_id exposed as part of the FME
>> >> > device (which represents the static region)? I'm not sure - it kind of also
>> >> > makes sense here, where you would have all the information in one place (if the
>> >> > interface_id matches, I can use this component to program a bitstream).
>>
>> According to the intel-fpga.txt document, the identifier for the
>> static image is at
>>
>> /sys/class/fpga_region/regionX/fpga-dfl-fme.n/bitstream_id
>
> Hi Alan

Hi Hao,

>
> This bitstream_id refects the full static region version. As you know, PR is
> only a sub feature of the FME functional unit, it's possible that we have
> different static region (different bitstream_id) but it has the exact same
> PR sub feature under the FME, only changes on other sub features or function
> units.

OK, thanks for that explanation.  That makes sense but could have
easily been different.  Please document this somewhere.

Alan

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Wu Hao]

On Mon, Feb 05, 2018 at 10:25:27PM -0600, Alan Tull wrote:
> On Mon, Feb 5, 2018 at 8:17 PM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Feb 05, 2018 at 11:21:52AM -0600, Alan Tull wrote:
> >> On Sun, Feb 4, 2018 at 4:05 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> > On Sat, Feb 03, 2018 at 11:41:24AM +0100, Moritz Fischer wrote:
> >> >> Hi Hao,
> >> >>
> >> >> On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> >> >> > Hi Hao, Alan,
> >> >> >
> >> >> > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> >> >> > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> >> >> > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> >> > > >
> >> >> > > > Hi Hao,
> >> >> > > >
> >> >> > > > A few comments below.   Besides that, looks good.
> >> >> > > >
> >> >> > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> >> >> > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> >> >> > > > >
> >> >> > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> >> >> > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >> >> > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> >> >> > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> >> >> > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> >> >> > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> >> > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> >> >> > > > > ----
> >> >> > > > > v3: rename driver to dfl-fpga-fme-mgr
> >> >> > > > >     implemented status callback for fpga manager
> >> >> > > > >     rebased due to fpga api changes
> >> >> > > > > ---
> >> >> > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> >> >> > > > >  drivers/fpga/Kconfig                               |   6 +
> >> >> > > > >  drivers/fpga/Makefile                              |   1 +
> >> >> > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> >> >> > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> >> >> > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> >> >> > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> >> > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> >> >> > > > >
> >> >> > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> >> > > > > new file mode 100644
> >> >> > > > > index 0000000..2d4f917
> >> >> > > > > --- /dev/null
> >> >> > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> >> > > > > @@ -0,0 +1,8 @@
> >> >> > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> >> >> > > > > +Date:          November 2017
> >> >> > > > > +KernelVersion:  4.15
> >> >> > > > > +Contact:       Wu Hao <hao.wu@intel.com>
> >> >> > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> >> >> > > > > +               hardware. Userspace could use this information to check if
> >> >> > > > > +               current hardware is compatible with given image before FPGA
> >> >> > > > > +               programming.
> >> >> > > >
> >> >> > > > I'm a little confused by this.  I can understand that the PR bitstream
> >> >> > > > has a dependency on the FPGA's static image, but I don't understand
> >> >> > > > the dependency of the bistream on the hardware that is used to program
> >> >> > > > the bitstream to the FPGA.
> >> >> > >
> >> >> > > Sorry for the confusion, the interface_id is used to indicate the version of
> >> >> > > the hardware for partial reconfiguration (it's part of the static image of
> >> >> > > the FPGA device). Will improve the description on this.
> >> >>
> >> >> I'm not sure userland should be making the call on whether what you're
> >> >> trying to load is compatible or not.
> >>
> >> Could you explain more about what your concern was about this (unless
> >> Hao has covered it below)?
> >>
> >> It makes sense to me in this use case at least since userspace has a
> >> pile of images and is choosing which one to load.
> >>
> >> >> Isn't there a way to check this in
> >> >> your PR reconfiguration handler in-kernel?
> >> >
> >> > Hi Moritz
> >> >
> >> > Actually with current driver interface, doing a partial reconfiguration with an
> >> > incompatible image, then driver will report PR failure with error code
> >> > FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR as hardware checks it, but anyway user
> >> > needs to know hardware interface_id information to find or re-generated correct
> >> > images. I think it's more flexible to leave it to userspace on using this
> >> > information exposed by driver. : )
> >> >
> >> > Thanks
> >> > Hao
> >> >
> >> >>
> >> >> > >
> >> >> >
> >> >> > The interface_id expresses the compatibility of the static region with PR
> >> >> > bitstreams generated for it. It changes every time a new static region is
> >> >> > generated.
> >>
> >> In the near future the DFL framework will be used with SoC's that have
> >> a hard FPGA PR manager (that's not part of the static region).  The
> >> hard FPGA manager driver won't know anything about the static region.
> >>
> >> >> >
> >> >> > Would it make more sense to have the interface_id exposed as part of the FME
> >> >> > device (which represents the static region)? I'm not sure - it kind of also
> >> >> > makes sense here, where you would have all the information in one place (if the
> >> >> > interface_id matches, I can use this component to program a bitstream).
> >>
> >> According to the intel-fpga.txt document, the identifier for the
> >> static image is at
> >>
> >> /sys/class/fpga_region/regionX/fpga-dfl-fme.n/bitstream_id
> >
> > Hi Alan
> 
> Hi Hao,
> 
> >
> > This bitstream_id refects the full static region version. As you know, PR is
> > only a sub feature of the FME functional unit, it's possible that we have
> > different static region (different bitstream_id) but it has the exact same
> > PR sub feature under the FME, only changes on other sub features or function
> > units.
> 
> OK, thanks for that explanation.  That makes sense but could have
> easily been different.  Please document this somewhere.

Sure, will do.

Thanks
Hao

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Moritz Fischer]

On Mon, Feb 05, 2018 at 10:25:27PM -0600, Alan Tull wrote:
> On Mon, Feb 5, 2018 at 8:17 PM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Feb 05, 2018 at 11:21:52AM -0600, Alan Tull wrote:
> >> On Sun, Feb 4, 2018 at 4:05 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> > On Sat, Feb 03, 2018 at 11:41:24AM +0100, Moritz Fischer wrote:
> >> >> Hi Hao,
> >> >>
> >> >> On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> >> >> > Hi Hao, Alan,
> >> >> >
> >> >> > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> >> >> > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> >> >> > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> >> > > >
> >> >> > > > Hi Hao,
> >> >> > > >
> >> >> > > > A few comments below.   Besides that, looks good.
> >> >> > > >
> >> >> > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> >> >> > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> >> >> > > > >
> >> >> > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> >> >> > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >> >> > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> >> >> > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> >> >> > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> >> >> > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> >> > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> >> >> > > > > ----
> >> >> > > > > v3: rename driver to dfl-fpga-fme-mgr
> >> >> > > > >     implemented status callback for fpga manager
> >> >> > > > >     rebased due to fpga api changes
> >> >> > > > > ---
> >> >> > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> >> >> > > > >  drivers/fpga/Kconfig                               |   6 +
> >> >> > > > >  drivers/fpga/Makefile                              |   1 +
> >> >> > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> >> >> > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> >> >> > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> >> >> > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> >> > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> >> >> > > > >
> >> >> > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> >> > > > > new file mode 100644
> >> >> > > > > index 0000000..2d4f917
> >> >> > > > > --- /dev/null
> >> >> > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> >> > > > > @@ -0,0 +1,8 @@
> >> >> > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> >> >> > > > > +Date:          November 2017
> >> >> > > > > +KernelVersion:  4.15
> >> >> > > > > +Contact:       Wu Hao <hao.wu@intel.com>
> >> >> > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> >> >> > > > > +               hardware. Userspace could use this information to check if
> >> >> > > > > +               current hardware is compatible with given image before FPGA
> >> >> > > > > +               programming.
> >> >> > > >
> >> >> > > > I'm a little confused by this.  I can understand that the PR bitstream
> >> >> > > > has a dependency on the FPGA's static image, but I don't understand
> >> >> > > > the dependency of the bistream on the hardware that is used to program
> >> >> > > > the bitstream to the FPGA.
> >> >> > >
> >> >> > > Sorry for the confusion, the interface_id is used to indicate the version of
> >> >> > > the hardware for partial reconfiguration (it's part of the static image of
> >> >> > > the FPGA device). Will improve the description on this.
> >> >>
> >> >> I'm not sure userland should be making the call on whether what you're
> >> >> trying to load is compatible or not.

I guess I was worried that you might take down a PR region that you
could've avoided. On second thought unless the kernel could extract that
info somehow from the bitstream before programming this is probably
anyway not possible, since the only one that knows the id -> bitstream
mapping is userland.

Hao's comments further down outline well enough that the hardware can
deal with incompatible bitstreams. So ignore my comments ;-)

> >>
> >> Could you explain more about what your concern was about this (unless
> >> Hao has covered it below)?
> >>
> >> It makes sense to me in this use case at least since userspace has a
> >> pile of images and is choosing which one to load.
> >>
> >> >> Isn't there a way to check this in
> >> >> your PR reconfiguration handler in-kernel?
> >> >
> >> > Hi Moritz
> >> >
> >> > Actually with current driver interface, doing a partial reconfiguration with an
> >> > incompatible image, then driver will report PR failure with error code
> >> > FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR as hardware checks it, but anyway user
> >> > needs to know hardware interface_id information to find or re-generated correct
> >> > images. I think it's more flexible to leave it to userspace on using this
> >> > information exposed by driver. : )
> >> >
> >> > Thanks
> >> > Hao
> >> >
> >> >>
> >> >> > >
> >> >> >
> >> >> > The interface_id expresses the compatibility of the static region with PR
> >> >> > bitstreams generated for it. It changes every time a new static region is
> >> >> > generated.
> >>
> >> In the near future the DFL framework will be used with SoC's that have
> >> a hard FPGA PR manager (that's not part of the static region).  The
> >> hard FPGA manager driver won't know anything about the static region.
> >>
> >> >> >
> >> >> > Would it make more sense to have the interface_id exposed as part of the FME
> >> >> > device (which represents the static region)? I'm not sure - it kind of also
> >> >> > makes sense here, where you would have all the information in one place (if the
> >> >> > interface_id matches, I can use this component to program a bitstream).
> >>
> >> According to the intel-fpga.txt document, the identifier for the
> >> static image is at
> >>
> >> /sys/class/fpga_region/regionX/fpga-dfl-fme.n/bitstream_id
> >
> > Hi Alan
> 
> Hi Hao,
> 
> >
> > This bitstream_id refects the full static region version. As you know, PR is
> > only a sub feature of the FME functional unit, it's possible that we have
> > different static region (different bitstream_id) but it has the exact same
> > PR sub feature under the FME, only changes on other sub features or function
> > units.
> 
> OK, thanks for that explanation.  That makes sense but could have
> easily been different.  Please document this somewhere.
> 
> Alan

Moritz

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Wu Hao]

On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> Hi Hao, Alan,
> 
> On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > > 
> > > Hi Hao,
> > > 
> > > A few comments below.   Besides that, looks good.
> > > 
> > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> > > >
> > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > > ----
> > > > v3: rename driver to dfl-fpga-fme-mgr
> > > >     implemented status callback for fpga manager
> > > >     rebased due to fpga api changes
> > > > ---
> > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> > > >  drivers/fpga/Kconfig                               |   6 +
> > > >  drivers/fpga/Makefile                              |   1 +
> > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> > > >
> > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > new file mode 100644
> > > > index 0000000..2d4f917
> > > > --- /dev/null
> > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > @@ -0,0 +1,8 @@
> > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> > > > +Date:          November 2017
> > > > +KernelVersion:  4.15
> > > > +Contact:       Wu Hao <hao.wu@intel.com>
> > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> > > > +               hardware. Userspace could use this information to check if
> > > > +               current hardware is compatible with given image before FPGA
> > > > +               programming.
> > > 
> > > I'm a little confused by this.  I can understand that the PR bitstream
> > > has a dependency on the FPGA's static image, but I don't understand
> > > the dependency of the bistream on the hardware that is used to program
> > > the bitstream to the FPGA.
> > 
> > Sorry for the confusion, the interface_id is used to indicate the version of
> > the hardware for partial reconfiguration (it's part of the static image of
> > the FPGA device). Will improve the description on this.
> > 
> 
> The interface_id expresses the compatibility of the static region with PR
> bitstreams generated for it. It changes every time a new static region is
> generated.
> 
> Would it make more sense to have the interface_id exposed as part of the FME
> device (which represents the static region)? I'm not sure - it kind of also
> makes sense here, where you would have all the information in one place (if the
> interface_id matches, I can use this component to program a bitstream).

Hi Enno

Yes, this interface is under fpga-dfl-fme-mgr.0, and fpga-dfl-fme-mgr.0 is
under fpga-dfl-fme.0. It's part of the FME device for sure. From another
point of view, it means if anyone wants to do PR on this Intel FPGA device,
he needs to find the FME device firstly, and then check if any fpga manager
created under this FME device, if yes, check the interface_id before PR via
the FME device node ioctl.

> 
> Sorry for my limited understanding of the infrastructure - would this same
> "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
> it would need to expose multiple interface_ids (or we'd have to track both
> interface IDs and an identifier for the target PR region).

Yes, the fpga manager could be shared with different PR regions.

Sorry, I'm not sure where we need to expose multiple interface_ids and why.

Thanks
Hao

> 
> > > 
> > > > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > > > index 57da904..0171ecb 100644
> > > > --- a/drivers/fpga/Kconfig
> > > > +++ b/drivers/fpga/Kconfig
> > > > @@ -150,6 +150,12 @@ config FPGA_DFL_FME
> > > >           FPGA platform level management features. There shall be 1 FME
> > > >           per DFL based FPGA device.
> > > >
> > > > +config FPGA_DFL_FME_MGR
> > > > +       tristate "FPGA DFL FME Manager Driver"
> > > > +       depends on FPGA_DFL_FME
> > > > +       help
> > > > +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
> > > > +
> > > >  config INTEL_FPGA_DFL_PCI
> > > >         tristate "Intel FPGA DFL PCIe Device Driver"
> > > >         depends on PCI && FPGA_DFL
> > > > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > > > index cc75bb3..6378580 100644
> > > > --- a/drivers/fpga/Makefile
> > > > +++ b/drivers/fpga/Makefile
> > > > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> > > >  # FPGA Device Feature List Support
> > > >  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
> > > >  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
> > > > +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
> > > >
> > > >  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
> > > >
> > > > diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > new file mode 100644
> > > > index 0000000..70356ce
> > > > --- /dev/null
> > > > +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > @@ -0,0 +1,318 @@
> > > > +/*
> > > > + * FPGA Manager Driver for FPGA Management Engine (FME)
> > > > + *
> > > > + * Copyright (C) 2017 Intel Corporation, Inc.
> > > > + *
> > > > + * Authors:
> > > > + *   Kang Luwei <luwei.kang@intel.com>
> > > > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > + *   Wu Hao <hao.wu@intel.com>
> > > > + *   Joseph Grecco <joe.grecco@intel.com>
> > > > + *   Enno Luebbers <enno.luebbers@intel.com>
> > > > + *   Tim Whisonant <tim.whisonant@intel.com>
> > > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > > > + *   Christopher Rauer <christopher.rauer@intel.com>
> > > > + *   Henry Mitchel <henry.mitchel@intel.com>
> > > > + *
> > > > + * This work is licensed under the terms of the GNU GPL version 2.
> > > > + * SPDX-License-Identifier: GPL-2.0
> > > > + */
> > > > +
> > > > +#include <linux/module.h>
> > > > +#include <linux/iopoll.h>
> > > > +#include <linux/fpga/fpga-mgr.h>
> > > > +
> > > > +#include "fpga-dfl.h"
> > > > +#include "dfl-fme.h"
> > > > +
> > > > +#define PR_WAIT_TIMEOUT   8000000
> > > > +#define PR_HOST_STATUS_IDLE    0
> > > > +
> > > > +struct fme_mgr_priv {
> > > > +       void __iomem *ioaddr;
> > > > +       u64 pr_error;
> > > > +};
> > > > +
> > > > +static ssize_t interface_id_show(struct device *dev,
> > > > +                                struct device_attribute *attr, char *buf)
> > > > +{
> > > > +       struct fpga_manager *mgr = dev_get_drvdata(dev);
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +       u64 intfc_id_l, intfc_id_h;
> > > > +
> > > > +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
> > > > +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
> > > > +
> > > > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > > > +                       (unsigned long long)intfc_id_h,
> > > > +                       (unsigned long long)intfc_id_l);
> > > > +}
> > > > +static DEVICE_ATTR_RO(interface_id);
> > > > +
> > > > +static const struct attribute *fme_mgr_attrs[] = {
> > > > +       &dev_attr_interface_id.attr,
> > > > +       NULL,
> > > > +};
> > > > +
> > > > +static u64 pr_error_to_mgr_status(u64 err)
> > > > +{
> > > > +       u64 status = 0;
> > > > +
> > > > +       if (err & FME_PR_ERR_OPERATION_ERR)
> > > > +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
> > > > +       if (err & FME_PR_ERR_CRC_ERR)
> > > > +               status |= FPGA_MGR_STATUS_CRC_ERR;
> > > > +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
> > > > +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
> > > > +       if (err & FME_PR_ERR_PROTOCOL_ERR)
> > > > +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> > > > +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
> > > > +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> > > > +
> > > > +       return status;
> > > > +}
> > > > +
> > > > +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
> > > > +{
> > > > +       u64 pr_status, pr_error;
> > > > +
> > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > +       if (!(pr_status & FME_PR_STS_PR_STS))
> > > > +               return 0;
> > > > +
> > > > +       pr_error = readq(fme_pr + FME_PR_ERR);
> > > > +       writeq(pr_error, fme_pr + FME_PR_ERR);
> > > > +
> > > > +       return pr_error;
> > > > +}
> > > > +
> > > > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> > > > +                             struct fpga_image_info *info,
> > > > +                             const char *buf, size_t count)
> > > > +{
> > > > +       struct device *dev = &mgr->dev;
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > +       u64 pr_ctrl, pr_status;
> > > > +
> > > > +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> > > > +               dev_err(dev, "only support partial reconfiguration.\n");
> > > 
> > > supports
> > > 
> > > > +               return -EINVAL;
> > > > +       }
> > > > +
> > > > +       dev_dbg(dev, "resetting PR before initiated PR\n");
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl |= FME_PR_CTRL_PR_RST;
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
> > > > +                              PR_WAIT_TIMEOUT)) {
> > > > +               dev_err(dev, "maximum PR timeout\n");
> > > 
> > > We have two dev_err's with the same "maximum PR timeout" so the user
> > > would not be able to see at which point the timeout occurred.
> > > 
> > > I suggest to get rid of the word 'maximum' for both.  This one could
> > > be 'reset ack timeout" or something similar.
> > 
> > Sure, will switch to a more specific message per your suggestion. Thanks.
> > 
> > > 
> > > > +               return -ETIMEDOUT;
> > > > +       }
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       dev_dbg(dev,
> > > > +               "waiting for PR resource in HW to be initialized and ready\n");
> > > > +
> > > > +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
> > > > +                              (pr_status & FME_PR_STS_PR_STS) ==
> > > > +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
> > > > +               dev_err(dev, "maximum PR timeout\n");
> > > 
> > > "PR_STS timeout"?  Or something better.
> > > 
> > > > +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > +               return -ETIMEDOUT;
> > > > +       }
> > > > +
> > > > +       dev_dbg(dev, "check and clear previous PR error\n");
> > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > +       if (priv->pr_error)
> > > > +               dev_dbg(dev, "previous PR error detected %llx\n",
> > > > +                       (unsigned long long)priv->pr_error);
> > > > +
> > > > +       dev_dbg(dev, "set PR port ID\n");
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
> > > > +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +
> > > > +static int fme_mgr_write(struct fpga_manager *mgr,
> > > > +                        const char *buf, size_t count)
> > > > +{
> > > > +       struct device *dev = &mgr->dev;
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > +       u64 pr_ctrl, pr_status, pr_data;
> > > > +       int delay = 0, pr_credit, i = 0;
> > > > +
> > > > +       dev_dbg(dev, "start request\n");
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl |= FME_PR_CTRL_PR_START;
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       dev_dbg(dev, "pushing data from bitstream to HW\n");
> > > > +
> > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > +
> > > > +       while (count > 0) {
> > > > +               while (pr_credit <= 1) {
> > > > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > > > +                               dev_err(dev, "maximum try\n");
> > > 
> > > It looks like this is waiting for an entry in a queue and timing out
> > > here.  Could you add a some comments for pr_credit above and this
> > > loop?  Also a better error message, perhaps "PR credit timeout"?
> > 
> > Driver needs to read the PR credit to know if it could push PR data
> > to hardware or not. I will add more description here on this PR credit,
> > and use "PR credit timeout" as error message.
> > 
> > > 
> > > > +                               return -ETIMEDOUT;
> > > > +                       }
> > > > +                       udelay(1);
> > > > +
> > > > +                       pr_status = readq(fme_pr + FME_PR_STS);
> > > > +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > +               }
> > > > +
> > > > +               if (count >= 4) {
> > > > +                       pr_data = 0;
> > > > +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> > > > +                                             *(((u32 *)buf) + i));
> > > > +                       writeq(pr_data, fme_pr + FME_PR_DATA);
> > > > +                       count -= 4;
> > > > +                       pr_credit--;
> > > > +                       i++;
> > > > +               } else {
> > > > +                       WARN_ON(1);
> > > > +                       return -EINVAL;
> > > > +               }
> > > > +       }
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +
> > > > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> > > > +                                 struct fpga_image_info *info)
> > > > +{
> > > > +       struct device *dev = &mgr->dev;
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > +       u64 pr_ctrl;
> > > > +
> > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
> > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > +
> > > > +       dev_dbg(dev, "green bitstream push complete\n");
> > > > +       dev_dbg(dev, "waiting for HW to release PR resource\n");
> > > > +
> > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
> > > > +                              PR_WAIT_TIMEOUT)) {
> > > > +               dev_err(dev, "maximum try.\n");
> > > 
> > > Some message specific to here also, please.
> > > 
> > > > +               return -ETIMEDOUT;
> > > > +       }
> > > > +
> > > > +       dev_dbg(dev, "PR operation complete, checking status\n");
> > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > +       if (priv->pr_error) {
> > > > +               dev_dbg(dev, "PR error detected %llx\n",
> > > > +                       (unsigned long long)priv->pr_error);
> > > > +               return -EIO;
> > > > +       }
> > > > +
> > > > +       dev_dbg(dev, "PR done successfully\n");
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +
> > > > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> > > > +{
> > > > +       return FPGA_MGR_STATE_UNKNOWN;
> > > > +}
> > > > +
> > > > +static u64 fme_mgr_status(struct fpga_manager *mgr)
> > > > +{
> > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > +
> > > > +       return pr_error_to_mgr_status(priv->pr_error);
> > > > +}
> > > > +
> > > > +static const struct fpga_manager_ops fme_mgr_ops = {
> > > > +       .write_init = fme_mgr_write_init,
> > > > +       .write = fme_mgr_write,
> > > > +       .write_complete = fme_mgr_write_complete,
> > > > +       .state = fme_mgr_state,
> > > > +       .status = fme_mgr_status,
> > > > +};
> > > > +
> > > > +static int fme_mgr_probe(struct platform_device *pdev)
> > > > +{
> > > > +       struct device *dev = &pdev->dev;
> > > > +       struct fme_mgr_priv *priv;
> > > > +       struct fpga_manager *mgr;
> > > > +       struct resource *res;
> > > > +       int ret;
> > > > +
> > > > +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> > > > +       if (!priv)
> > > > +               return -ENOMEM;
> > > > +
> > > > +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> > > > +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> > > > +       if (IS_ERR(priv->ioaddr))
> > > > +               return PTR_ERR(priv->ioaddr);
> > > > +
> > > > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > +       if (ret)
> > > > +               return ret;
> > > > +
> > > > +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
> > > > +       if (!mgr)
> > > > +               goto sysfs_remove_exit;
> > > > +
> > > > +       mgr->name = "DFL FPGA Manager";
> > > > +       mgr->mops = &fme_mgr_ops;
> > > > +       mgr->priv = priv;
> > > > +       mgr->parent = dev;
> > > > +       platform_set_drvdata(pdev, mgr);
> > > > +
> > > > +       ret = fpga_mgr_register(mgr);
> > > > +       if (ret) {
> > > > +               dev_err(dev, "unable to register FPGA manager\n");
> > > > +               goto sysfs_remove_exit;
> > > > +       }
> > > > +
> > > > +       return 0;
> > > > +
> > > > +sysfs_remove_exit:
> > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > +       return ret;
> > > > +}
> > > > +
> > > > +static int fme_mgr_remove(struct platform_device *pdev)
> > > > +{
> > > > +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
> > > > +
> > > > +       fpga_mgr_unregister(mgr);
> > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +
> > > > +static struct platform_driver fme_mgr_driver = {
> > > > +       .driver = {
> > > > +               .name    = FPGA_DFL_FME_MGR,
> > > > +       },
> > > > +       .probe   = fme_mgr_probe,
> > > > +       .remove  = fme_mgr_remove,
> > > > +};
> > > > +
> > > > +module_platform_driver(fme_mgr_driver);
> > > > +
> > > > +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
> > > > +MODULE_AUTHOR("Intel Corporation");
> > > > +MODULE_LICENSE("GPL v2");
> > > > +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
> > > > diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> > > > index e5a1094..d45eb82 100644
> > > > --- a/drivers/fpga/fpga-dfl.h
> > > > +++ b/drivers/fpga/fpga-dfl.h
> > > > @@ -130,7 +130,44 @@
> > > >
> > > >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > > >  #define FME_PR_DFH             DFH
> > > > -#define FME_PR_SIZE            DFH_SIZE
> > > > +#define FME_PR_CTRL            0x8
> > > > +#define FME_PR_STS             0x10
> > > > +#define FME_PR_DATA            0x18
> > > > +#define FME_PR_ERR             0x20
> > > > +#define FME_PR_INTFC_ID_H      0xA8
> > > > +#define FME_PR_INTFC_ID_L      0xB0
> > > > +#define FME_PR_SIZE            0xB8
> > > > +
> > > > +/* FME PR Control Register Bitfield */
> > > > +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
> > > > +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
> > > > +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
> > > > +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
> > > > +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
> > > > +
> > > > +/* FME PR Status Register Bitfield */
> > > > +/* Number of available entries in HW queue inside the PR engine. */
> > > > +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
> > > > +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
> > > > +#define FME_PR_STS_PR_STS_IDLE 0
> > > > +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
> > > > +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
> > > > +
> > > > +/* FME PR Data Register Bitfield */
> > > > +/* PR data from the raw-binary file. */
> > > > +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
> > > > +
> > > > +/* FME PR Error Register */
> > > > +/* Previous PR Operation errors detected. */
> > > > +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
> > > > +/* CRC error detected. */
> > > > +#define FME_PR_ERR_CRC_ERR             BIT(1)
> > > > +/* Incompatible PR bitstream detected. */
> > > > +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
> > > > +/* PR data push protocol violated. */
> > > > +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
> > > > +/* PR data fifo overflow error detected */
> > > > +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
> > > >
> > > >  /* FME HSSI Sub Feature Register Set */
> > > >  #define FME_HSSI_DFH           DFH
> > > 
> > > I see fpga-dfl.h as enumeration code which is separate from any driver
> > > implementation specifics other than what's required for the DFL
> > > enumeration scheme.    These PR engine #defines should move to a .h
> > > that is dedicated to this specific PR hardware device. If someone else
> > > adds a different PR device to the framework, their PR driver would
> > > also have its own .h.  Same for any other modules that aren't central
> > > to DFL enumeration.
> > 
> > Sure, will move PR related register to a separated header file.
> > DFL enumeration related registers, will still be kept in fpga-dfl.h.
> > 
> > Thanks
> > Hao
> > 
> > > 
> > > Thanks,
> > > Alan
> > > 
> > > > --
> > > > 1.8.3.1
> > > >

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Luebbers, Enno]

Hi Hao,

On Sun, Feb 04, 2018 at 05:37:06PM +0800, Wu Hao wrote:
> On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> > Hi Hao, Alan,
> > 
> > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > > > 
> > > > Hi Hao,
> > > > 
> > > > A few comments below.   Besides that, looks good.
> > > > 
> > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> > > > >
> > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > > > ----
> > > > > v3: rename driver to dfl-fpga-fme-mgr
> > > > >     implemented status callback for fpga manager
> > > > >     rebased due to fpga api changes
> > > > > ---
> > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> > > > >  drivers/fpga/Kconfig                               |   6 +
> > > > >  drivers/fpga/Makefile                              |   1 +
> > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> > > > >
> > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > > new file mode 100644
> > > > > index 0000000..2d4f917
> > > > > --- /dev/null
> > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > > @@ -0,0 +1,8 @@
> > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> > > > > +Date:          November 2017
> > > > > +KernelVersion:  4.15
> > > > > +Contact:       Wu Hao <hao.wu@intel.com>
> > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> > > > > +               hardware. Userspace could use this information to check if
> > > > > +               current hardware is compatible with given image before FPGA
> > > > > +               programming.
> > > > 
> > > > I'm a little confused by this.  I can understand that the PR bitstream
> > > > has a dependency on the FPGA's static image, but I don't understand
> > > > the dependency of the bistream on the hardware that is used to program
> > > > the bitstream to the FPGA.
> > > 
> > > Sorry for the confusion, the interface_id is used to indicate the version of
> > > the hardware for partial reconfiguration (it's part of the static image of
> > > the FPGA device). Will improve the description on this.
> > > 
> > 
> > The interface_id expresses the compatibility of the static region with PR
> > bitstreams generated for it. It changes every time a new static region is
> > generated.
> > 
> > Would it make more sense to have the interface_id exposed as part of the FME
> > device (which represents the static region)? I'm not sure - it kind of also
> > makes sense here, where you would have all the information in one place (if the
> > interface_id matches, I can use this component to program a bitstream).
> 
> Hi Enno
> 
> Yes, this interface is under fpga-dfl-fme-mgr.0, and fpga-dfl-fme-mgr.0 is
> under fpga-dfl-fme.0. It's part of the FME device for sure. From another
> point of view, it means if anyone wants to do PR on this Intel FPGA device,
> he needs to find the FME device firstly, and then check if any fpga manager
> created under this FME device, if yes, check the interface_id before PR via
> the FME device node ioctl.

That sounds good, thank you!

> 
> > 
> > Sorry for my limited understanding of the infrastructure - would this same
> > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
> > it would need to expose multiple interface_ids (or we'd have to track both
> > interface IDs and an identifier for the target PR region).
> 
> Yes, the fpga manager could be shared with different PR regions.
> 
> Sorry, I'm not sure where we need to expose multiple interface_ids and why.

It's basically a question of how to determine bitstream compatibility - either,
there's a separate interface_id per reconfigurable region, or there is a single
interface_id for the entire device. Both make sense from a certain perspective.

If there are multiple interface_ids per device (one per region), the driver
would need to expose all of them. If there's only a single one, the driver only
exposes that one ID - compatibility would be determined by looking at both that
single interface_id _and_ the identifier/number of the targeted region.

I would prefer a separate interface_id per region - it seems more generic and
flexible.

Thanks
- Enno

> 
> Thanks
> Hao
> 
> > 
> > > > 
> > > > > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > > > > index 57da904..0171ecb 100644
> > > > > --- a/drivers/fpga/Kconfig
> > > > > +++ b/drivers/fpga/Kconfig
> > > > > @@ -150,6 +150,12 @@ config FPGA_DFL_FME
> > > > >           FPGA platform level management features. There shall be 1 FME
> > > > >           per DFL based FPGA device.
> > > > >
> > > > > +config FPGA_DFL_FME_MGR
> > > > > +       tristate "FPGA DFL FME Manager Driver"
> > > > > +       depends on FPGA_DFL_FME
> > > > > +       help
> > > > > +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
> > > > > +
> > > > >  config INTEL_FPGA_DFL_PCI
> > > > >         tristate "Intel FPGA DFL PCIe Device Driver"
> > > > >         depends on PCI && FPGA_DFL
> > > > > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > > > > index cc75bb3..6378580 100644
> > > > > --- a/drivers/fpga/Makefile
> > > > > +++ b/drivers/fpga/Makefile
> > > > > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> > > > >  # FPGA Device Feature List Support
> > > > >  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
> > > > >  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
> > > > > +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
> > > > >
> > > > >  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
> > > > >
> > > > > diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > > new file mode 100644
> > > > > index 0000000..70356ce
> > > > > --- /dev/null
> > > > > +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > > @@ -0,0 +1,318 @@
> > > > > +/*
> > > > > + * FPGA Manager Driver for FPGA Management Engine (FME)
> > > > > + *
> > > > > + * Copyright (C) 2017 Intel Corporation, Inc.
> > > > > + *
> > > > > + * Authors:
> > > > > + *   Kang Luwei <luwei.kang@intel.com>
> > > > > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > > + *   Wu Hao <hao.wu@intel.com>
> > > > > + *   Joseph Grecco <joe.grecco@intel.com>
> > > > > + *   Enno Luebbers <enno.luebbers@intel.com>
> > > > > + *   Tim Whisonant <tim.whisonant@intel.com>
> > > > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > > > > + *   Christopher Rauer <christopher.rauer@intel.com>
> > > > > + *   Henry Mitchel <henry.mitchel@intel.com>
> > > > > + *
> > > > > + * This work is licensed under the terms of the GNU GPL version 2.
> > > > > + * SPDX-License-Identifier: GPL-2.0
> > > > > + */
> > > > > +
> > > > > +#include <linux/module.h>
> > > > > +#include <linux/iopoll.h>
> > > > > +#include <linux/fpga/fpga-mgr.h>
> > > > > +
> > > > > +#include "fpga-dfl.h"
> > > > > +#include "dfl-fme.h"
> > > > > +
> > > > > +#define PR_WAIT_TIMEOUT   8000000
> > > > > +#define PR_HOST_STATUS_IDLE    0
> > > > > +
> > > > > +struct fme_mgr_priv {
> > > > > +       void __iomem *ioaddr;
> > > > > +       u64 pr_error;
> > > > > +};
> > > > > +
> > > > > +static ssize_t interface_id_show(struct device *dev,
> > > > > +                                struct device_attribute *attr, char *buf)
> > > > > +{
> > > > > +       struct fpga_manager *mgr = dev_get_drvdata(dev);
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +       u64 intfc_id_l, intfc_id_h;
> > > > > +
> > > > > +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
> > > > > +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
> > > > > +
> > > > > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > > > > +                       (unsigned long long)intfc_id_h,
> > > > > +                       (unsigned long long)intfc_id_l);
> > > > > +}
> > > > > +static DEVICE_ATTR_RO(interface_id);
> > > > > +
> > > > > +static const struct attribute *fme_mgr_attrs[] = {
> > > > > +       &dev_attr_interface_id.attr,
> > > > > +       NULL,
> > > > > +};
> > > > > +
> > > > > +static u64 pr_error_to_mgr_status(u64 err)
> > > > > +{
> > > > > +       u64 status = 0;
> > > > > +
> > > > > +       if (err & FME_PR_ERR_OPERATION_ERR)
> > > > > +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
> > > > > +       if (err & FME_PR_ERR_CRC_ERR)
> > > > > +               status |= FPGA_MGR_STATUS_CRC_ERR;
> > > > > +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
> > > > > +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
> > > > > +       if (err & FME_PR_ERR_PROTOCOL_ERR)
> > > > > +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> > > > > +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
> > > > > +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> > > > > +
> > > > > +       return status;
> > > > > +}
> > > > > +
> > > > > +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
> > > > > +{
> > > > > +       u64 pr_status, pr_error;
> > > > > +
> > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > > +       if (!(pr_status & FME_PR_STS_PR_STS))
> > > > > +               return 0;
> > > > > +
> > > > > +       pr_error = readq(fme_pr + FME_PR_ERR);
> > > > > +       writeq(pr_error, fme_pr + FME_PR_ERR);
> > > > > +
> > > > > +       return pr_error;
> > > > > +}
> > > > > +
> > > > > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> > > > > +                             struct fpga_image_info *info,
> > > > > +                             const char *buf, size_t count)
> > > > > +{
> > > > > +       struct device *dev = &mgr->dev;
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > > +       u64 pr_ctrl, pr_status;
> > > > > +
> > > > > +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> > > > > +               dev_err(dev, "only support partial reconfiguration.\n");
> > > > 
> > > > supports
> > > > 
> > > > > +               return -EINVAL;
> > > > > +       }
> > > > > +
> > > > > +       dev_dbg(dev, "resetting PR before initiated PR\n");
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_RST;
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > > +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
> > > > > +                              PR_WAIT_TIMEOUT)) {
> > > > > +               dev_err(dev, "maximum PR timeout\n");
> > > > 
> > > > We have two dev_err's with the same "maximum PR timeout" so the user
> > > > would not be able to see at which point the timeout occurred.
> > > > 
> > > > I suggest to get rid of the word 'maximum' for both.  This one could
> > > > be 'reset ack timeout" or something similar.
> > > 
> > > Sure, will switch to a more specific message per your suggestion. Thanks.
> > > 
> > > > 
> > > > > +               return -ETIMEDOUT;
> > > > > +       }
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       dev_dbg(dev,
> > > > > +               "waiting for PR resource in HW to be initialized and ready\n");
> > > > > +
> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
> > > > > +                              (pr_status & FME_PR_STS_PR_STS) ==
> > > > > +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
> > > > > +               dev_err(dev, "maximum PR timeout\n");
> > > > 
> > > > "PR_STS timeout"?  Or something better.
> > > > 
> > > > > +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > > +               return -ETIMEDOUT;
> > > > > +       }
> > > > > +
> > > > > +       dev_dbg(dev, "check and clear previous PR error\n");
> > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > > +       if (priv->pr_error)
> > > > > +               dev_dbg(dev, "previous PR error detected %llx\n",
> > > > > +                       (unsigned long long)priv->pr_error);
> > > > > +
> > > > > +       dev_dbg(dev, "set PR port ID\n");
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
> > > > > +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +
> > > > > +static int fme_mgr_write(struct fpga_manager *mgr,
> > > > > +                        const char *buf, size_t count)
> > > > > +{
> > > > > +       struct device *dev = &mgr->dev;
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > > +       u64 pr_ctrl, pr_status, pr_data;
> > > > > +       int delay = 0, pr_credit, i = 0;
> > > > > +
> > > > > +       dev_dbg(dev, "start request\n");
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_START;
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       dev_dbg(dev, "pushing data from bitstream to HW\n");
> > > > > +
> > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > > +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > > +
> > > > > +       while (count > 0) {
> > > > > +               while (pr_credit <= 1) {
> > > > > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > > > > +                               dev_err(dev, "maximum try\n");
> > > > 
> > > > It looks like this is waiting for an entry in a queue and timing out
> > > > here.  Could you add a some comments for pr_credit above and this
> > > > loop?  Also a better error message, perhaps "PR credit timeout"?
> > > 
> > > Driver needs to read the PR credit to know if it could push PR data
> > > to hardware or not. I will add more description here on this PR credit,
> > > and use "PR credit timeout" as error message.
> > > 
> > > > 
> > > > > +                               return -ETIMEDOUT;
> > > > > +                       }
> > > > > +                       udelay(1);
> > > > > +
> > > > > +                       pr_status = readq(fme_pr + FME_PR_STS);
> > > > > +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > > +               }
> > > > > +
> > > > > +               if (count >= 4) {
> > > > > +                       pr_data = 0;
> > > > > +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> > > > > +                                             *(((u32 *)buf) + i));
> > > > > +                       writeq(pr_data, fme_pr + FME_PR_DATA);
> > > > > +                       count -= 4;
> > > > > +                       pr_credit--;
> > > > > +                       i++;
> > > > > +               } else {
> > > > > +                       WARN_ON(1);
> > > > > +                       return -EINVAL;
> > > > > +               }
> > > > > +       }
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +
> > > > > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> > > > > +                                 struct fpga_image_info *info)
> > > > > +{
> > > > > +       struct device *dev = &mgr->dev;
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > > +       u64 pr_ctrl;
> > > > > +
> > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
> > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > +
> > > > > +       dev_dbg(dev, "green bitstream push complete\n");
> > > > > +       dev_dbg(dev, "waiting for HW to release PR resource\n");
> > > > > +
> > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > > +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
> > > > > +                              PR_WAIT_TIMEOUT)) {
> > > > > +               dev_err(dev, "maximum try.\n");
> > > > 
> > > > Some message specific to here also, please.
> > > > 
> > > > > +               return -ETIMEDOUT;
> > > > > +       }
> > > > > +
> > > > > +       dev_dbg(dev, "PR operation complete, checking status\n");
> > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > > +       if (priv->pr_error) {
> > > > > +               dev_dbg(dev, "PR error detected %llx\n",
> > > > > +                       (unsigned long long)priv->pr_error);
> > > > > +               return -EIO;
> > > > > +       }
> > > > > +
> > > > > +       dev_dbg(dev, "PR done successfully\n");
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +
> > > > > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> > > > > +{
> > > > > +       return FPGA_MGR_STATE_UNKNOWN;
> > > > > +}
> > > > > +
> > > > > +static u64 fme_mgr_status(struct fpga_manager *mgr)
> > > > > +{
> > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > +
> > > > > +       return pr_error_to_mgr_status(priv->pr_error);
> > > > > +}
> > > > > +
> > > > > +static const struct fpga_manager_ops fme_mgr_ops = {
> > > > > +       .write_init = fme_mgr_write_init,
> > > > > +       .write = fme_mgr_write,
> > > > > +       .write_complete = fme_mgr_write_complete,
> > > > > +       .state = fme_mgr_state,
> > > > > +       .status = fme_mgr_status,
> > > > > +};
> > > > > +
> > > > > +static int fme_mgr_probe(struct platform_device *pdev)
> > > > > +{
> > > > > +       struct device *dev = &pdev->dev;
> > > > > +       struct fme_mgr_priv *priv;
> > > > > +       struct fpga_manager *mgr;
> > > > > +       struct resource *res;
> > > > > +       int ret;
> > > > > +
> > > > > +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> > > > > +       if (!priv)
> > > > > +               return -ENOMEM;
> > > > > +
> > > > > +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> > > > > +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> > > > > +       if (IS_ERR(priv->ioaddr))
> > > > > +               return PTR_ERR(priv->ioaddr);
> > > > > +
> > > > > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > > +       if (ret)
> > > > > +               return ret;
> > > > > +
> > > > > +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
> > > > > +       if (!mgr)
> > > > > +               goto sysfs_remove_exit;
> > > > > +
> > > > > +       mgr->name = "DFL FPGA Manager";
> > > > > +       mgr->mops = &fme_mgr_ops;
> > > > > +       mgr->priv = priv;
> > > > > +       mgr->parent = dev;
> > > > > +       platform_set_drvdata(pdev, mgr);
> > > > > +
> > > > > +       ret = fpga_mgr_register(mgr);
> > > > > +       if (ret) {
> > > > > +               dev_err(dev, "unable to register FPGA manager\n");
> > > > > +               goto sysfs_remove_exit;
> > > > > +       }
> > > > > +
> > > > > +       return 0;
> > > > > +
> > > > > +sysfs_remove_exit:
> > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > > +       return ret;
> > > > > +}
> > > > > +
> > > > > +static int fme_mgr_remove(struct platform_device *pdev)
> > > > > +{
> > > > > +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
> > > > > +
> > > > > +       fpga_mgr_unregister(mgr);
> > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +
> > > > > +static struct platform_driver fme_mgr_driver = {
> > > > > +       .driver = {
> > > > > +               .name    = FPGA_DFL_FME_MGR,
> > > > > +       },
> > > > > +       .probe   = fme_mgr_probe,
> > > > > +       .remove  = fme_mgr_remove,
> > > > > +};
> > > > > +
> > > > > +module_platform_driver(fme_mgr_driver);
> > > > > +
> > > > > +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
> > > > > +MODULE_AUTHOR("Intel Corporation");
> > > > > +MODULE_LICENSE("GPL v2");
> > > > > +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
> > > > > diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> > > > > index e5a1094..d45eb82 100644
> > > > > --- a/drivers/fpga/fpga-dfl.h
> > > > > +++ b/drivers/fpga/fpga-dfl.h
> > > > > @@ -130,7 +130,44 @@
> > > > >
> > > > >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > > > >  #define FME_PR_DFH             DFH
> > > > > -#define FME_PR_SIZE            DFH_SIZE
> > > > > +#define FME_PR_CTRL            0x8
> > > > > +#define FME_PR_STS             0x10
> > > > > +#define FME_PR_DATA            0x18
> > > > > +#define FME_PR_ERR             0x20
> > > > > +#define FME_PR_INTFC_ID_H      0xA8
> > > > > +#define FME_PR_INTFC_ID_L      0xB0
> > > > > +#define FME_PR_SIZE            0xB8
> > > > > +
> > > > > +/* FME PR Control Register Bitfield */
> > > > > +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
> > > > > +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
> > > > > +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
> > > > > +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
> > > > > +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
> > > > > +
> > > > > +/* FME PR Status Register Bitfield */
> > > > > +/* Number of available entries in HW queue inside the PR engine. */
> > > > > +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
> > > > > +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
> > > > > +#define FME_PR_STS_PR_STS_IDLE 0
> > > > > +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
> > > > > +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
> > > > > +
> > > > > +/* FME PR Data Register Bitfield */
> > > > > +/* PR data from the raw-binary file. */
> > > > > +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
> > > > > +
> > > > > +/* FME PR Error Register */
> > > > > +/* Previous PR Operation errors detected. */
> > > > > +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
> > > > > +/* CRC error detected. */
> > > > > +#define FME_PR_ERR_CRC_ERR             BIT(1)
> > > > > +/* Incompatible PR bitstream detected. */
> > > > > +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
> > > > > +/* PR data push protocol violated. */
> > > > > +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
> > > > > +/* PR data fifo overflow error detected */
> > > > > +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
> > > > >
> > > > >  /* FME HSSI Sub Feature Register Set */
> > > > >  #define FME_HSSI_DFH           DFH
> > > > 
> > > > I see fpga-dfl.h as enumeration code which is separate from any driver
> > > > implementation specifics other than what's required for the DFL
> > > > enumeration scheme.    These PR engine #defines should move to a .h
> > > > that is dedicated to this specific PR hardware device. If someone else
> > > > adds a different PR device to the framework, their PR driver would
> > > > also have its own .h.  Same for any other modules that aren't central
> > > > to DFL enumeration.
> > > 
> > > Sure, will move PR related register to a separated header file.
> > > DFL enumeration related registers, will still be kept in fpga-dfl.h.
> > > 
> > > Thanks
> > > Hao
> > > 
> > > > 
> > > > Thanks,
> > > > Alan
> > > > 
> > > > > --
> > > > > 1.8.3.1
> > > > >

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Wu Hao]

On Mon, Feb 05, 2018 at 10:36:45AM -0800, Luebbers, Enno wrote:
> Hi Hao,
> 
> On Sun, Feb 04, 2018 at 05:37:06PM +0800, Wu Hao wrote:
> > On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> > > Hi Hao, Alan,
> > > 
> > > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> > > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> > > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > > > > 
> > > > > Hi Hao,
> > > > > 
> > > > > A few comments below.   Besides that, looks good.
> > > > > 
> > > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> > > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> > > > > >
> > > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > > > > ----
> > > > > > v3: rename driver to dfl-fpga-fme-mgr
> > > > > >     implemented status callback for fpga manager
> > > > > >     rebased due to fpga api changes
> > > > > > ---
> > > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> > > > > >  drivers/fpga/Kconfig                               |   6 +
> > > > > >  drivers/fpga/Makefile                              |   1 +
> > > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> > > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> > > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> > > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> > > > > >
> > > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > > > new file mode 100644
> > > > > > index 0000000..2d4f917
> > > > > > --- /dev/null
> > > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> > > > > > @@ -0,0 +1,8 @@
> > > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> > > > > > +Date:          November 2017
> > > > > > +KernelVersion:  4.15
> > > > > > +Contact:       Wu Hao <hao.wu@intel.com>
> > > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> > > > > > +               hardware. Userspace could use this information to check if
> > > > > > +               current hardware is compatible with given image before FPGA
> > > > > > +               programming.
> > > > > 
> > > > > I'm a little confused by this.  I can understand that the PR bitstream
> > > > > has a dependency on the FPGA's static image, but I don't understand
> > > > > the dependency of the bistream on the hardware that is used to program
> > > > > the bitstream to the FPGA.
> > > > 
> > > > Sorry for the confusion, the interface_id is used to indicate the version of
> > > > the hardware for partial reconfiguration (it's part of the static image of
> > > > the FPGA device). Will improve the description on this.
> > > > 
> > > 
> > > The interface_id expresses the compatibility of the static region with PR
> > > bitstreams generated for it. It changes every time a new static region is
> > > generated.
> > > 
> > > Would it make more sense to have the interface_id exposed as part of the FME
> > > device (which represents the static region)? I'm not sure - it kind of also
> > > makes sense here, where you would have all the information in one place (if the
> > > interface_id matches, I can use this component to program a bitstream).
> > 
> > Hi Enno
> > 
> > Yes, this interface is under fpga-dfl-fme-mgr.0, and fpga-dfl-fme-mgr.0 is
> > under fpga-dfl-fme.0. It's part of the FME device for sure. From another
> > point of view, it means if anyone wants to do PR on this Intel FPGA device,
> > he needs to find the FME device firstly, and then check if any fpga manager
> > created under this FME device, if yes, check the interface_id before PR via
> > the FME device node ioctl.
> 
> That sounds good, thank you!
> 
> > 
> > > 
> > > Sorry for my limited understanding of the infrastructure - would this same
> > > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
> > > it would need to expose multiple interface_ids (or we'd have to track both
> > > interface IDs and an identifier for the target PR region).
> > 
> > Yes, the fpga manager could be shared with different PR regions.
> > 
> > Sorry, I'm not sure where we need to expose multiple interface_ids and why.
> 
> It's basically a question of how to determine bitstream compatibility - either,
> there's a separate interface_id per reconfigurable region, or there is a single
> interface_id for the entire device. Both make sense from a certain perspective.
> 
> If there are multiple interface_ids per device (one per region), the driver
> would need to expose all of them. If there's only a single one, the driver only
> exposes that one ID - compatibility would be determined by looking at both that
> single interface_id _and_ the identifier/number of the targeted region.
> 
> I would prefer a separate interface_id per region - it seems more generic and
> flexible.

It's possible to have per region interface_id (or even both per dev interface_id
and per region interface_id at the same time), but per FME PR sub feature
implementation, it supports multiple PR regions, but only provide one interface
id, so at least in this case, it's not per-region information per my
understanding. We can consider it later when hardware really supports it. : )

Thanks
Hao

> 
> Thanks
> - Enno
> 
> > 
> > Thanks
> > Hao
> > 
> > > 
> > > > > 
> > > > > > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > > > > > index 57da904..0171ecb 100644
> > > > > > --- a/drivers/fpga/Kconfig
> > > > > > +++ b/drivers/fpga/Kconfig
> > > > > > @@ -150,6 +150,12 @@ config FPGA_DFL_FME
> > > > > >           FPGA platform level management features. There shall be 1 FME
> > > > > >           per DFL based FPGA device.
> > > > > >
> > > > > > +config FPGA_DFL_FME_MGR
> > > > > > +       tristate "FPGA DFL FME Manager Driver"
> > > > > > +       depends on FPGA_DFL_FME
> > > > > > +       help
> > > > > > +         Say Y to enable FPGA Manager driver for FPGA Management Engine.
> > > > > > +
> > > > > >  config INTEL_FPGA_DFL_PCI
> > > > > >         tristate "Intel FPGA DFL PCIe Device Driver"
> > > > > >         depends on PCI && FPGA_DFL
> > > > > > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > > > > > index cc75bb3..6378580 100644
> > > > > > --- a/drivers/fpga/Makefile
> > > > > > +++ b/drivers/fpga/Makefile
> > > > > > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> > > > > >  # FPGA Device Feature List Support
> > > > > >  obj-$(CONFIG_FPGA_DFL)                 += fpga-dfl.o
> > > > > >  obj-$(CONFIG_FPGA_DFL_FME)             += fpga-dfl-fme.o
> > > > > > +obj-$(CONFIG_FPGA_DFL_FME_MGR)         += fpga-dfl-fme-mgr.o
> > > > > >
> > > > > >  fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
> > > > > >
> > > > > > diff --git a/drivers/fpga/fpga-dfl-fme-mgr.c b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > > > new file mode 100644
> > > > > > index 0000000..70356ce
> > > > > > --- /dev/null
> > > > > > +++ b/drivers/fpga/fpga-dfl-fme-mgr.c
> > > > > > @@ -0,0 +1,318 @@
> > > > > > +/*
> > > > > > + * FPGA Manager Driver for FPGA Management Engine (FME)
> > > > > > + *
> > > > > > + * Copyright (C) 2017 Intel Corporation, Inc.
> > > > > > + *
> > > > > > + * Authors:
> > > > > > + *   Kang Luwei <luwei.kang@intel.com>
> > > > > > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > > > + *   Wu Hao <hao.wu@intel.com>
> > > > > > + *   Joseph Grecco <joe.grecco@intel.com>
> > > > > > + *   Enno Luebbers <enno.luebbers@intel.com>
> > > > > > + *   Tim Whisonant <tim.whisonant@intel.com>
> > > > > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > > > > > + *   Christopher Rauer <christopher.rauer@intel.com>
> > > > > > + *   Henry Mitchel <henry.mitchel@intel.com>
> > > > > > + *
> > > > > > + * This work is licensed under the terms of the GNU GPL version 2.
> > > > > > + * SPDX-License-Identifier: GPL-2.0
> > > > > > + */
> > > > > > +
> > > > > > +#include <linux/module.h>
> > > > > > +#include <linux/iopoll.h>
> > > > > > +#include <linux/fpga/fpga-mgr.h>
> > > > > > +
> > > > > > +#include "fpga-dfl.h"
> > > > > > +#include "dfl-fme.h"
> > > > > > +
> > > > > > +#define PR_WAIT_TIMEOUT   8000000
> > > > > > +#define PR_HOST_STATUS_IDLE    0
> > > > > > +
> > > > > > +struct fme_mgr_priv {
> > > > > > +       void __iomem *ioaddr;
> > > > > > +       u64 pr_error;
> > > > > > +};
> > > > > > +
> > > > > > +static ssize_t interface_id_show(struct device *dev,
> > > > > > +                                struct device_attribute *attr, char *buf)
> > > > > > +{
> > > > > > +       struct fpga_manager *mgr = dev_get_drvdata(dev);
> > > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > > +       u64 intfc_id_l, intfc_id_h;
> > > > > > +
> > > > > > +       intfc_id_l = readq(priv->ioaddr + FME_PR_INTFC_ID_L);
> > > > > > +       intfc_id_h = readq(priv->ioaddr + FME_PR_INTFC_ID_H);
> > > > > > +
> > > > > > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > > > > > +                       (unsigned long long)intfc_id_h,
> > > > > > +                       (unsigned long long)intfc_id_l);
> > > > > > +}
> > > > > > +static DEVICE_ATTR_RO(interface_id);
> > > > > > +
> > > > > > +static const struct attribute *fme_mgr_attrs[] = {
> > > > > > +       &dev_attr_interface_id.attr,
> > > > > > +       NULL,
> > > > > > +};
> > > > > > +
> > > > > > +static u64 pr_error_to_mgr_status(u64 err)
> > > > > > +{
> > > > > > +       u64 status = 0;
> > > > > > +
> > > > > > +       if (err & FME_PR_ERR_OPERATION_ERR)
> > > > > > +               status |= FPGA_MGR_STATUS_OPERATION_ERR;
> > > > > > +       if (err & FME_PR_ERR_CRC_ERR)
> > > > > > +               status |= FPGA_MGR_STATUS_CRC_ERR;
> > > > > > +       if (err & FME_PR_ERR_INCOMPATIBLE_BS)
> > > > > > +               status |= FPGA_MGR_STATUS_INCOMPATIBLE_IMAGE_ERR;
> > > > > > +       if (err & FME_PR_ERR_PROTOCOL_ERR)
> > > > > > +               status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> > > > > > +       if (err & FME_PR_ERR_FIFO_OVERFLOW)
> > > > > > +               status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> > > > > > +
> > > > > > +       return status;
> > > > > > +}
> > > > > > +
> > > > > > +static u64 fme_mgr_pr_error_handle(void __iomem *fme_pr)
> > > > > > +{
> > > > > > +       u64 pr_status, pr_error;
> > > > > > +
> > > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > > > +       if (!(pr_status & FME_PR_STS_PR_STS))
> > > > > > +               return 0;
> > > > > > +
> > > > > > +       pr_error = readq(fme_pr + FME_PR_ERR);
> > > > > > +       writeq(pr_error, fme_pr + FME_PR_ERR);
> > > > > > +
> > > > > > +       return pr_error;
> > > > > > +}
> > > > > > +
> > > > > > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> > > > > > +                             struct fpga_image_info *info,
> > > > > > +                             const char *buf, size_t count)
> > > > > > +{
> > > > > > +       struct device *dev = &mgr->dev;
> > > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > > > +       u64 pr_ctrl, pr_status;
> > > > > > +
> > > > > > +       if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> > > > > > +               dev_err(dev, "only support partial reconfiguration.\n");
> > > > > 
> > > > > supports
> > > > > 
> > > > > > +               return -EINVAL;
> > > > > > +       }
> > > > > > +
> > > > > > +       dev_dbg(dev, "resetting PR before initiated PR\n");
> > > > > > +
> > > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > > +       pr_ctrl |= FME_PR_CTRL_PR_RST;
> > > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > > +
> > > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > > > +                              pr_ctrl & FME_PR_CTRL_PR_RSTACK, 1,
> > > > > > +                              PR_WAIT_TIMEOUT)) {
> > > > > > +               dev_err(dev, "maximum PR timeout\n");
> > > > > 
> > > > > We have two dev_err's with the same "maximum PR timeout" so the user
> > > > > would not be able to see at which point the timeout occurred.
> > > > > 
> > > > > I suggest to get rid of the word 'maximum' for both.  This one could
> > > > > be 'reset ack timeout" or something similar.
> > > > 
> > > > Sure, will switch to a more specific message per your suggestion. Thanks.
> > > > 
> > > > > 
> > > > > > +               return -ETIMEDOUT;
> > > > > > +       }
> > > > > > +
> > > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RST;
> > > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > > +
> > > > > > +       dev_dbg(dev,
> > > > > > +               "waiting for PR resource in HW to be initialized and ready\n");
> > > > > > +
> > > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_STS, pr_status,
> > > > > > +                              (pr_status & FME_PR_STS_PR_STS) ==
> > > > > > +                              FME_PR_STS_PR_STS_IDLE, 1, PR_WAIT_TIMEOUT)) {
> > > > > > +               dev_err(dev, "maximum PR timeout\n");
> > > > > 
> > > > > "PR_STS timeout"?  Or something better.
> > > > > 
> > > > > > +               priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > > > +               return -ETIMEDOUT;
> > > > > > +       }
> > > > > > +
> > > > > > +       dev_dbg(dev, "check and clear previous PR error\n");
> > > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > > > +       if (priv->pr_error)
> > > > > > +               dev_dbg(dev, "previous PR error detected %llx\n",
> > > > > > +                       (unsigned long long)priv->pr_error);
> > > > > > +
> > > > > > +       dev_dbg(dev, "set PR port ID\n");
> > > > > > +
> > > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > > +       pr_ctrl &= ~FME_PR_CTRL_PR_RGN_ID;
> > > > > > +       pr_ctrl |= FIELD_PREP(FME_PR_CTRL_PR_RGN_ID, info->region_id);
> > > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > > +
> > > > > > +       return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static int fme_mgr_write(struct fpga_manager *mgr,
> > > > > > +                        const char *buf, size_t count)
> > > > > > +{
> > > > > > +       struct device *dev = &mgr->dev;
> > > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > > > +       u64 pr_ctrl, pr_status, pr_data;
> > > > > > +       int delay = 0, pr_credit, i = 0;
> > > > > > +
> > > > > > +       dev_dbg(dev, "start request\n");
> > > > > > +
> > > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > > +       pr_ctrl |= FME_PR_CTRL_PR_START;
> > > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > > +
> > > > > > +       dev_dbg(dev, "pushing data from bitstream to HW\n");
> > > > > > +
> > > > > > +       pr_status = readq(fme_pr + FME_PR_STS);
> > > > > > +       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > > > +
> > > > > > +       while (count > 0) {
> > > > > > +               while (pr_credit <= 1) {
> > > > > > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > > > > > +                               dev_err(dev, "maximum try\n");
> > > > > 
> > > > > It looks like this is waiting for an entry in a queue and timing out
> > > > > here.  Could you add a some comments for pr_credit above and this
> > > > > loop?  Also a better error message, perhaps "PR credit timeout"?
> > > > 
> > > > Driver needs to read the PR credit to know if it could push PR data
> > > > to hardware or not. I will add more description here on this PR credit,
> > > > and use "PR credit timeout" as error message.
> > > > 
> > > > > 
> > > > > > +                               return -ETIMEDOUT;
> > > > > > +                       }
> > > > > > +                       udelay(1);
> > > > > > +
> > > > > > +                       pr_status = readq(fme_pr + FME_PR_STS);
> > > > > > +                       pr_credit = FIELD_GET(FME_PR_STS_PR_CREDIT, pr_status);
> > > > > > +               }
> > > > > > +
> > > > > > +               if (count >= 4) {
> > > > > > +                       pr_data = 0;
> > > > > > +                       pr_data |= FIELD_PREP(FME_PR_DATA_PR_DATA_RAW,
> > > > > > +                                             *(((u32 *)buf) + i));
> > > > > > +                       writeq(pr_data, fme_pr + FME_PR_DATA);
> > > > > > +                       count -= 4;
> > > > > > +                       pr_credit--;
> > > > > > +                       i++;
> > > > > > +               } else {
> > > > > > +                       WARN_ON(1);
> > > > > > +                       return -EINVAL;
> > > > > > +               }
> > > > > > +       }
> > > > > > +
> > > > > > +       return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> > > > > > +                                 struct fpga_image_info *info)
> > > > > > +{
> > > > > > +       struct device *dev = &mgr->dev;
> > > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > > +       void __iomem *fme_pr = priv->ioaddr;
> > > > > > +       u64 pr_ctrl;
> > > > > > +
> > > > > > +       pr_ctrl = readq(fme_pr + FME_PR_CTRL);
> > > > > > +       pr_ctrl |= FME_PR_CTRL_PR_COMPLETE;
> > > > > > +       writeq(pr_ctrl, fme_pr + FME_PR_CTRL);
> > > > > > +
> > > > > > +       dev_dbg(dev, "green bitstream push complete\n");
> > > > > > +       dev_dbg(dev, "waiting for HW to release PR resource\n");
> > > > > > +
> > > > > > +       if (readq_poll_timeout(fme_pr + FME_PR_CTRL, pr_ctrl,
> > > > > > +                              !(pr_ctrl & FME_PR_CTRL_PR_START), 1,
> > > > > > +                              PR_WAIT_TIMEOUT)) {
> > > > > > +               dev_err(dev, "maximum try.\n");
> > > > > 
> > > > > Some message specific to here also, please.
> > > > > 
> > > > > > +               return -ETIMEDOUT;
> > > > > > +       }
> > > > > > +
> > > > > > +       dev_dbg(dev, "PR operation complete, checking status\n");
> > > > > > +       priv->pr_error = fme_mgr_pr_error_handle(fme_pr);
> > > > > > +       if (priv->pr_error) {
> > > > > > +               dev_dbg(dev, "PR error detected %llx\n",
> > > > > > +                       (unsigned long long)priv->pr_error);
> > > > > > +               return -EIO;
> > > > > > +       }
> > > > > > +
> > > > > > +       dev_dbg(dev, "PR done successfully\n");
> > > > > > +
> > > > > > +       return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> > > > > > +{
> > > > > > +       return FPGA_MGR_STATE_UNKNOWN;
> > > > > > +}
> > > > > > +
> > > > > > +static u64 fme_mgr_status(struct fpga_manager *mgr)
> > > > > > +{
> > > > > > +       struct fme_mgr_priv *priv = mgr->priv;
> > > > > > +
> > > > > > +       return pr_error_to_mgr_status(priv->pr_error);
> > > > > > +}
> > > > > > +
> > > > > > +static const struct fpga_manager_ops fme_mgr_ops = {
> > > > > > +       .write_init = fme_mgr_write_init,
> > > > > > +       .write = fme_mgr_write,
> > > > > > +       .write_complete = fme_mgr_write_complete,
> > > > > > +       .state = fme_mgr_state,
> > > > > > +       .status = fme_mgr_status,
> > > > > > +};
> > > > > > +
> > > > > > +static int fme_mgr_probe(struct platform_device *pdev)
> > > > > > +{
> > > > > > +       struct device *dev = &pdev->dev;
> > > > > > +       struct fme_mgr_priv *priv;
> > > > > > +       struct fpga_manager *mgr;
> > > > > > +       struct resource *res;
> > > > > > +       int ret;
> > > > > > +
> > > > > > +       priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> > > > > > +       if (!priv)
> > > > > > +               return -ENOMEM;
> > > > > > +
> > > > > > +       res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> > > > > > +       priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> > > > > > +       if (IS_ERR(priv->ioaddr))
> > > > > > +               return PTR_ERR(priv->ioaddr);
> > > > > > +
> > > > > > +       ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > > > +       if (ret)
> > > > > > +               return ret;
> > > > > > +
> > > > > > +       mgr = devm_kzalloc(dev, sizeof(*mgr), GFP_KERNEL);
> > > > > > +       if (!mgr)
> > > > > > +               goto sysfs_remove_exit;
> > > > > > +
> > > > > > +       mgr->name = "DFL FPGA Manager";
> > > > > > +       mgr->mops = &fme_mgr_ops;
> > > > > > +       mgr->priv = priv;
> > > > > > +       mgr->parent = dev;
> > > > > > +       platform_set_drvdata(pdev, mgr);
> > > > > > +
> > > > > > +       ret = fpga_mgr_register(mgr);
> > > > > > +       if (ret) {
> > > > > > +               dev_err(dev, "unable to register FPGA manager\n");
> > > > > > +               goto sysfs_remove_exit;
> > > > > > +       }
> > > > > > +
> > > > > > +       return 0;
> > > > > > +
> > > > > > +sysfs_remove_exit:
> > > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > > > +       return ret;
> > > > > > +}
> > > > > > +
> > > > > > +static int fme_mgr_remove(struct platform_device *pdev)
> > > > > > +{
> > > > > > +       struct fpga_manager *mgr = platform_get_drvdata(pdev);
> > > > > > +
> > > > > > +       fpga_mgr_unregister(mgr);
> > > > > > +       sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > > > > > +
> > > > > > +       return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static struct platform_driver fme_mgr_driver = {
> > > > > > +       .driver = {
> > > > > > +               .name    = FPGA_DFL_FME_MGR,
> > > > > > +       },
> > > > > > +       .probe   = fme_mgr_probe,
> > > > > > +       .remove  = fme_mgr_remove,
> > > > > > +};
> > > > > > +
> > > > > > +module_platform_driver(fme_mgr_driver);
> > > > > > +
> > > > > > +MODULE_DESCRIPTION("FPGA Manager for FPGA Management Engine");
> > > > > > +MODULE_AUTHOR("Intel Corporation");
> > > > > > +MODULE_LICENSE("GPL v2");
> > > > > > +MODULE_ALIAS("platform:fpga-dfl-fme-mgr");
> > > > > > diff --git a/drivers/fpga/fpga-dfl.h b/drivers/fpga/fpga-dfl.h
> > > > > > index e5a1094..d45eb82 100644
> > > > > > --- a/drivers/fpga/fpga-dfl.h
> > > > > > +++ b/drivers/fpga/fpga-dfl.h
> > > > > > @@ -130,7 +130,44 @@
> > > > > >
> > > > > >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > > > > >  #define FME_PR_DFH             DFH
> > > > > > -#define FME_PR_SIZE            DFH_SIZE
> > > > > > +#define FME_PR_CTRL            0x8
> > > > > > +#define FME_PR_STS             0x10
> > > > > > +#define FME_PR_DATA            0x18
> > > > > > +#define FME_PR_ERR             0x20
> > > > > > +#define FME_PR_INTFC_ID_H      0xA8
> > > > > > +#define FME_PR_INTFC_ID_L      0xB0
> > > > > > +#define FME_PR_SIZE            0xB8
> > > > > > +
> > > > > > +/* FME PR Control Register Bitfield */
> > > > > > +#define FME_PR_CTRL_PR_RST     BIT(0)  /* Reset PR engine */
> > > > > > +#define FME_PR_CTRL_PR_RSTACK  BIT(4)  /* Ack for PR engine reset */
> > > > > > +#define FME_PR_CTRL_PR_RGN_ID  GENMASK_ULL(9, 7)       /* PR Region ID */
> > > > > > +#define FME_PR_CTRL_PR_START   BIT(12) /* Start to request for PR service */
> > > > > > +#define FME_PR_CTRL_PR_COMPLETE        BIT(13) /* PR data push complete notification */
> > > > > > +
> > > > > > +/* FME PR Status Register Bitfield */
> > > > > > +/* Number of available entries in HW queue inside the PR engine. */
> > > > > > +#define FME_PR_STS_PR_CREDIT   GENMASK_ULL(8, 0)
> > > > > > +#define FME_PR_STS_PR_STS      BIT(16) /* PR operation status */
> > > > > > +#define FME_PR_STS_PR_STS_IDLE 0
> > > > > > +#define FME_PR_STS_PR_CTRLR_STS        GENMASK_ULL(22, 20)     /* Controller status */
> > > > > > +#define FME_PR_STS_PR_HOST_STS GENMASK_ULL(27, 24)     /* PR host status */
> > > > > > +
> > > > > > +/* FME PR Data Register Bitfield */
> > > > > > +/* PR data from the raw-binary file. */
> > > > > > +#define FME_PR_DATA_PR_DATA_RAW        GENMASK_ULL(32, 0)
> > > > > > +
> > > > > > +/* FME PR Error Register */
> > > > > > +/* Previous PR Operation errors detected. */
> > > > > > +#define FME_PR_ERR_OPERATION_ERR       BIT(0)
> > > > > > +/* CRC error detected. */
> > > > > > +#define FME_PR_ERR_CRC_ERR             BIT(1)
> > > > > > +/* Incompatible PR bitstream detected. */
> > > > > > +#define FME_PR_ERR_INCOMPATIBLE_BS     BIT(2)
> > > > > > +/* PR data push protocol violated. */
> > > > > > +#define FME_PR_ERR_PROTOCOL_ERR                BIT(3)
> > > > > > +/* PR data fifo overflow error detected */
> > > > > > +#define FME_PR_ERR_FIFO_OVERFLOW       BIT(4)
> > > > > >
> > > > > >  /* FME HSSI Sub Feature Register Set */
> > > > > >  #define FME_HSSI_DFH           DFH
> > > > > 
> > > > > I see fpga-dfl.h as enumeration code which is separate from any driver
> > > > > implementation specifics other than what's required for the DFL
> > > > > enumeration scheme.    These PR engine #defines should move to a .h
> > > > > that is dedicated to this specific PR hardware device. If someone else
> > > > > adds a different PR device to the framework, their PR driver would
> > > > > also have its own .h.  Same for any other modules that aren't central
> > > > > to DFL enumeration.
> > > > 
> > > > Sure, will move PR related register to a separated header file.
> > > > DFL enumeration related registers, will still be kept in fpga-dfl.h.
> > > > 
> > > > Thanks
> > > > Hao
> > > > 
> > > > > 
> > > > > Thanks,
> > > > > Alan
> > > > > 
> > > > > > --
> > > > > > 1.8.3.1
> > > > > >

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Alan Tull]

On Mon, Feb 5, 2018 at 7:47 PM, Wu Hao <hao.wu@intel.com> wrote:
> On Mon, Feb 05, 2018 at 10:36:45AM -0800, Luebbers, Enno wrote:
>> Hi Hao,
>>
>> On Sun, Feb 04, 2018 at 05:37:06PM +0800, Wu Hao wrote:
>> > On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
>> > > Hi Hao, Alan,
>> > >
>> > > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
>> > > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
>> > > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
>> > > > >
>> > > > > Hi Hao,
>> > > > >
>> > > > > A few comments below.   Besides that, looks good.
>> > > > >
>> > > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
>> > > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
>> > > > > >
>> > > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> > > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> > > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> > > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> > > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> > > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > > > > > ----
>> > > > > > v3: rename driver to dfl-fpga-fme-mgr
>> > > > > >     implemented status callback for fpga manager
>> > > > > >     rebased due to fpga api changes
>> > > > > > ---
>> > > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
>> > > > > >  drivers/fpga/Kconfig                               |   6 +
>> > > > > >  drivers/fpga/Makefile                              |   1 +
>> > > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
>> > > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
>> > > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
>> > > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> > > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
>> > > > > >
>> > > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> > > > > > new file mode 100644
>> > > > > > index 0000000..2d4f917
>> > > > > > --- /dev/null
>> > > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> > > > > > @@ -0,0 +1,8 @@
>> > > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
>> > > > > > +Date:          November 2017
>> > > > > > +KernelVersion:  4.15
>> > > > > > +Contact:       Wu Hao <hao.wu@intel.com>
>> > > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
>> > > > > > +               hardware. Userspace could use this information to check if
>> > > > > > +               current hardware is compatible with given image before FPGA
>> > > > > > +               programming.
>> > > > >
>> > > > > I'm a little confused by this.  I can understand that the PR bitstream
>> > > > > has a dependency on the FPGA's static image, but I don't understand
>> > > > > the dependency of the bistream on the hardware that is used to program
>> > > > > the bitstream to the FPGA.
>> > > >
>> > > > Sorry for the confusion, the interface_id is used to indicate the version of
>> > > > the hardware for partial reconfiguration (it's part of the static image of
>> > > > the FPGA device). Will improve the description on this.
>> > > >
>> > >
>> > > The interface_id expresses the compatibility of the static region with PR
>> > > bitstreams generated for it. It changes every time a new static region is
>> > > generated.
>> > >
>> > > Would it make more sense to have the interface_id exposed as part of the FME
>> > > device (which represents the static region)? I'm not sure - it kind of also
>> > > makes sense here, where you would have all the information in one place (if the
>> > > interface_id matches, I can use this component to program a bitstream).
>> >
>> > Hi Enno
>> >
>> > Yes, this interface is under fpga-dfl-fme-mgr.0, and fpga-dfl-fme-mgr.0 is
>> > under fpga-dfl-fme.0. It's part of the FME device for sure. From another
>> > point of view, it means if anyone wants to do PR on this Intel FPGA device,
>> > he needs to find the FME device firstly, and then check if any fpga manager
>> > created under this FME device, if yes, check the interface_id before PR via
>> > the FME device node ioctl.
>>
>> That sounds good, thank you!
>>
>> >
>> > >
>> > > Sorry for my limited understanding of the infrastructure - would this same
>> > > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
>> > > it would need to expose multiple interface_ids (or we'd have to track both
>> > > interface IDs and an identifier for the target PR region).
>> >
>> > Yes, the fpga manager could be shared with different PR regions.
>> >
>> > Sorry, I'm not sure where we need to expose multiple interface_ids and why.
>>
>> It's basically a question of how to determine bitstream compatibility - either,
>> there's a separate interface_id per reconfigurable region, or there is a single
>> interface_id for the entire device. Both make sense from a certain perspective.
>>
>> If there are multiple interface_ids per device (one per region), the driver
>> would need to expose all of them. If there's only a single one, the driver only
>> exposes that one ID - compatibility would be determined by looking at both that
>> single interface_id _and_ the identifier/number of the targeted region.
>>
>> I would prefer a separate interface_id per region - it seems more generic and
>> flexible.

Hi Enno,

I agree with this.

>
> It's possible to have per region interface_id (or even both per dev interface_id
> and per region interface_id at the same time), but per FME PR sub feature
> implementation, it supports multiple PR regions, but only provide one interface
> id, so at least in this case, it's not per-region information per my
> understanding. We can consider it later when hardware really supports it. : )

Hi Hao,

I understand that in the case of this PR hardware, the region to
program is selected when the region_id to program is written to a PR
hardware control register.  For another example, Arria10 has a hard PR
hardware and the PR bitstream lands in the area of the FPGA for which
it was compiled.  In that case, for the PR bitstream to be compatible
with a PR region, the layout of the edge connections also needs to be
compatible, so compatibility is per-region in that case instead of
per-PR hardware.   And besides, as I said yesterday, the hard PR
hardware would not know what the static region ID is when this
framework is used with such a device.

That's why I think making the id per-region may be more future proof,
even if it may see unnecessary in the case of the original blue bits
this was written for.

Alan

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Wu Hao]

On Mon, Feb 05, 2018 at 10:25:54PM -0600, Alan Tull wrote:
> On Mon, Feb 5, 2018 at 7:47 PM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Feb 05, 2018 at 10:36:45AM -0800, Luebbers, Enno wrote:
> >> Hi Hao,
> >>
> >> On Sun, Feb 04, 2018 at 05:37:06PM +0800, Wu Hao wrote:
> >> > On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> >> > > Hi Hao, Alan,
> >> > >
> >> > > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> >> > > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> >> > > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> > > > >
> >> > > > > Hi Hao,
> >> > > > >
> >> > > > > A few comments below.   Besides that, looks good.
> >> > > > >
> >> > > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> >> > > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> >> > > > > >
> >> > > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> >> > > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >> > > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> >> > > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> >> > > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> >> > > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> > > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> >> > > > > > ----
> >> > > > > > v3: rename driver to dfl-fpga-fme-mgr
> >> > > > > >     implemented status callback for fpga manager
> >> > > > > >     rebased due to fpga api changes
> >> > > > > > ---
> >> > > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> >> > > > > >  drivers/fpga/Kconfig                               |   6 +
> >> > > > > >  drivers/fpga/Makefile                              |   1 +
> >> > > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> >> > > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> >> > > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> >> > > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> > > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> >> > > > > >
> >> > > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> > > > > > new file mode 100644
> >> > > > > > index 0000000..2d4f917
> >> > > > > > --- /dev/null
> >> > > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> > > > > > @@ -0,0 +1,8 @@
> >> > > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> >> > > > > > +Date:          November 2017
> >> > > > > > +KernelVersion:  4.15
> >> > > > > > +Contact:       Wu Hao <hao.wu@intel.com>
> >> > > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> >> > > > > > +               hardware. Userspace could use this information to check if
> >> > > > > > +               current hardware is compatible with given image before FPGA
> >> > > > > > +               programming.
> >> > > > >
> >> > > > > I'm a little confused by this.  I can understand that the PR bitstream
> >> > > > > has a dependency on the FPGA's static image, but I don't understand
> >> > > > > the dependency of the bistream on the hardware that is used to program
> >> > > > > the bitstream to the FPGA.
> >> > > >
> >> > > > Sorry for the confusion, the interface_id is used to indicate the version of
> >> > > > the hardware for partial reconfiguration (it's part of the static image of
> >> > > > the FPGA device). Will improve the description on this.
> >> > > >
> >> > >
> >> > > The interface_id expresses the compatibility of the static region with PR
> >> > > bitstreams generated for it. It changes every time a new static region is
> >> > > generated.
> >> > >
> >> > > Would it make more sense to have the interface_id exposed as part of the FME
> >> > > device (which represents the static region)? I'm not sure - it kind of also
> >> > > makes sense here, where you would have all the information in one place (if the
> >> > > interface_id matches, I can use this component to program a bitstream).
> >> >
> >> > Hi Enno
> >> >
> >> > Yes, this interface is under fpga-dfl-fme-mgr.0, and fpga-dfl-fme-mgr.0 is
> >> > under fpga-dfl-fme.0. It's part of the FME device for sure. From another
> >> > point of view, it means if anyone wants to do PR on this Intel FPGA device,
> >> > he needs to find the FME device firstly, and then check if any fpga manager
> >> > created under this FME device, if yes, check the interface_id before PR via
> >> > the FME device node ioctl.
> >>
> >> That sounds good, thank you!
> >>
> >> >
> >> > >
> >> > > Sorry for my limited understanding of the infrastructure - would this same
> >> > > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
> >> > > it would need to expose multiple interface_ids (or we'd have to track both
> >> > > interface IDs and an identifier for the target PR region).
> >> >
> >> > Yes, the fpga manager could be shared with different PR regions.
> >> >
> >> > Sorry, I'm not sure where we need to expose multiple interface_ids and why.
> >>
> >> It's basically a question of how to determine bitstream compatibility - either,
> >> there's a separate interface_id per reconfigurable region, or there is a single
> >> interface_id for the entire device. Both make sense from a certain perspective.
> >>
> >> If there are multiple interface_ids per device (one per region), the driver
> >> would need to expose all of them. If there's only a single one, the driver only
> >> exposes that one ID - compatibility would be determined by looking at both that
> >> single interface_id _and_ the identifier/number of the targeted region.
> >>
> >> I would prefer a separate interface_id per region - it seems more generic and
> >> flexible.
> 
> Hi Enno,
> 
> I agree with this.
> 
> >
> > It's possible to have per region interface_id (or even both per dev interface_id
> > and per region interface_id at the same time), but per FME PR sub feature
> > implementation, it supports multiple PR regions, but only provide one interface
> > id, so at least in this case, it's not per-region information per my
> > understanding. We can consider it later when hardware really supports it. : )
> 
> Hi Hao,
> 
> I understand that in the case of this PR hardware, the region to
> program is selected when the region_id to program is written to a PR
> hardware control register.  For another example, Arria10 has a hard PR
> hardware and the PR bitstream lands in the area of the FPGA for which
> it was compiled.  In that case, for the PR bitstream to be compatible
> with a PR region, the layout of the edge connections also needs to be
> compatible, so compatibility is per-region in that case instead of
> per-PR hardware.

Hi Alan,

Thanks a lot for the explanation. :)

I fully understand the consideration of adding per-region interface_id.

> And besides, as I said yesterday, the hard PR
> hardware would not know what the static region ID is when this
> framework is used with such a device.

Yes, is it possible that hard PR hardware with different versions, requires
different images or different methods for compatibility checking?

> 
> That's why I think making the id per-region may be more future proof,
> even if it may see unnecessary in the case of the original blue bits
> this was written for.

I feel that per-PR hardware interface id is useful in some cases, and maybe
in some cases, both per-PR hardware and per-region interface ids are needed
for its compatibility checking, so shall we leave developers to decide to
implement per-PR hardware or per-region or both interface ids based on their
own hardware implementations? How do you think? :)

Thanks
Hao

> 
> Alan

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Alan Tull]

On Tue, Feb 6, 2018 at 12:47 AM, Wu Hao <hao.wu@intel.com> wrote:
> On Mon, Feb 05, 2018 at 10:25:54PM -0600, Alan Tull wrote:
>> On Mon, Feb 5, 2018 at 7:47 PM, Wu Hao <hao.wu@intel.com> wrote:
>> > On Mon, Feb 05, 2018 at 10:36:45AM -0800, Luebbers, Enno wrote:
>> >> Hi Hao,
>> >>
>> >> On Sun, Feb 04, 2018 at 05:37:06PM +0800, Wu Hao wrote:
>> >> > On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
>> >> > > Hi Hao, Alan,
>> >> > >
>> >> > > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
>> >> > > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
>> >> > > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
>> >> > > > >
>> >> > > > > Hi Hao,
>> >> > > > >
>> >> > > > > A few comments below.   Besides that, looks good.
>> >> > > > >
>> >> > > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
>> >> > > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
>> >> > > > > >
>> >> > > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> >> > > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> >> > > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> >> > > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> >> > > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> >> > > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> >> > > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> >> > > > > > ----
>> >> > > > > > v3: rename driver to dfl-fpga-fme-mgr
>> >> > > > > >     implemented status callback for fpga manager
>> >> > > > > >     rebased due to fpga api changes
>> >> > > > > > ---
>> >> > > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
>> >> > > > > >  drivers/fpga/Kconfig                               |   6 +
>> >> > > > > >  drivers/fpga/Makefile                              |   1 +
>> >> > > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
>> >> > > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
>> >> > > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
>> >> > > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> >> > > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
>> >> > > > > >
>> >> > > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> >> > > > > > new file mode 100644
>> >> > > > > > index 0000000..2d4f917
>> >> > > > > > --- /dev/null
>> >> > > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> >> > > > > > @@ -0,0 +1,8 @@
>> >> > > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
>> >> > > > > > +Date:          November 2017
>> >> > > > > > +KernelVersion:  4.15
>> >> > > > > > +Contact:       Wu Hao <hao.wu@intel.com>
>> >> > > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
>> >> > > > > > +               hardware. Userspace could use this information to check if
>> >> > > > > > +               current hardware is compatible with given image before FPGA
>> >> > > > > > +               programming.
>> >> > > > >
>> >> > > > > I'm a little confused by this.  I can understand that the PR bitstream
>> >> > > > > has a dependency on the FPGA's static image, but I don't understand
>> >> > > > > the dependency of the bistream on the hardware that is used to program
>> >> > > > > the bitstream to the FPGA.
>> >> > > >
>> >> > > > Sorry for the confusion, the interface_id is used to indicate the version of
>> >> > > > the hardware for partial reconfiguration (it's part of the static image of
>> >> > > > the FPGA device). Will improve the description on this.
>> >> > > >
>> >> > >
>> >> > > The interface_id expresses the compatibility of the static region with PR
>> >> > > bitstreams generated for it. It changes every time a new static region is
>> >> > > generated.
>> >> > >
>> >> > > Would it make more sense to have the interface_id exposed as part of the FME
>> >> > > device (which represents the static region)? I'm not sure - it kind of also
>> >> > > makes sense here, where you would have all the information in one place (if the
>> >> > > interface_id matches, I can use this component to program a bitstream).
>> >> >
>> >> > Hi Enno
>> >> >
>> >> > Yes, this interface is under fpga-dfl-fme-mgr.0, and fpga-dfl-fme-mgr.0 is
>> >> > under fpga-dfl-fme.0. It's part of the FME device for sure. From another
>> >> > point of view, it means if anyone wants to do PR on this Intel FPGA device,
>> >> > he needs to find the FME device firstly, and then check if any fpga manager
>> >> > created under this FME device, if yes, check the interface_id before PR via
>> >> > the FME device node ioctl.
>> >>
>> >> That sounds good, thank you!
>> >>
>> >> >
>> >> > >
>> >> > > Sorry for my limited understanding of the infrastructure - would this same
>> >> > > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
>> >> > > it would need to expose multiple interface_ids (or we'd have to track both
>> >> > > interface IDs and an identifier for the target PR region).
>> >> >
>> >> > Yes, the fpga manager could be shared with different PR regions.
>> >> >
>> >> > Sorry, I'm not sure where we need to expose multiple interface_ids and why.
>> >>
>> >> It's basically a question of how to determine bitstream compatibility - either,
>> >> there's a separate interface_id per reconfigurable region, or there is a single
>> >> interface_id for the entire device. Both make sense from a certain perspective.
>> >>
>> >> If there are multiple interface_ids per device (one per region), the driver
>> >> would need to expose all of them. If there's only a single one, the driver only
>> >> exposes that one ID - compatibility would be determined by looking at both that
>> >> single interface_id _and_ the identifier/number of the targeted region.
>> >>
>> >> I would prefer a separate interface_id per region - it seems more generic and
>> >> flexible.
>>
>> Hi Enno,
>>
>> I agree with this.
>>
>> >
>> > It's possible to have per region interface_id (or even both per dev interface_id
>> > and per region interface_id at the same time), but per FME PR sub feature
>> > implementation, it supports multiple PR regions, but only provide one interface
>> > id, so at least in this case, it's not per-region information per my
>> > understanding. We can consider it later when hardware really supports it. : )
>>
>> Hi Hao,
>>
>> I understand that in the case of this PR hardware, the region to
>> program is selected when the region_id to program is written to a PR
>> hardware control register.  For another example, Arria10 has a hard PR
>> hardware and the PR bitstream lands in the area of the FPGA for which
>> it was compiled.  In that case, for the PR bitstream to be compatible
>> with a PR region, the layout of the edge connections also needs to be
>> compatible, so compatibility is per-region in that case instead of
>> per-PR hardware.
>
> Hi Alan,
>
> Thanks a lot for the explanation. :)
>
> I fully understand the consideration of adding per-region interface_id.
>
>> And besides, as I said yesterday, the hard PR
>> hardware would not know what the static region ID is when this
>> framework is used with such a device.
>
> Yes, is it possible that hard PR hardware with different versions, requires
> different images or different methods for compatibility checking?

Because it is really hardware and not something in the FPGA fabric,
the hard PR hardware isn't going to change versions very often.  It
has to be designed to be flexible and not add any constraints on the
PR regions.  If some feature is added or a bug is fixed, that's just a
driver issue at most and should not affect PR region compatibility.
PR region compatibility would only be dependent on the static FPGA
image and the regions that are created in it.  It could be exported in
terms of a single static region ID or per-region ID.

>
>>
>> That's why I think making the id per-region may be more future proof,
>> even if it may see unnecessary in the case of the original blue bits
>> this was written for.
>
> I feel that per-PR hardware interface id is useful in some cases, and maybe
> in some cases, both per-PR hardware and per-region interface ids are needed
> for its compatibility checking, so shall we leave developers to decide to
> implement per-PR hardware or per-region or both interface ids based on their
> own hardware implementations? How do you think? :)

That gives us 3 sets of id's.  Seems overly complicated and the
userspace would have to figure out which set of id's to use.  I want
to see an interface that isn't more complicated than it needs to be
but still can be expected to be ok for the future (as far as we can
anticipate).

Would per-region id's cause any problems that you can see?  I
understand that the region id's would all be the same value for a
given PR hardware in your use case, but that doesn't seem like it
would be hard to implement or that it opens up some possible failure.

Alan

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Wu Hao]

On Tue, Feb 06, 2018 at 12:53:44PM -0600, Alan Tull wrote:
> On Tue, Feb 6, 2018 at 12:47 AM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Feb 05, 2018 at 10:25:54PM -0600, Alan Tull wrote:
> >> On Mon, Feb 5, 2018 at 7:47 PM, Wu Hao <hao.wu@intel.com> wrote:
> >> > On Mon, Feb 05, 2018 at 10:36:45AM -0800, Luebbers, Enno wrote:
> >> >> Hi Hao,
> >> >>
> >> >> On Sun, Feb 04, 2018 at 05:37:06PM +0800, Wu Hao wrote:
> >> >> > On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
> >> >> > > Hi Hao, Alan,
> >> >> > >
> >> >> > > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
> >> >> > > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
> >> >> > > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> >> > > > >
> >> >> > > > > Hi Hao,
> >> >> > > > >
> >> >> > > > > A few comments below.   Besides that, looks good.
> >> >> > > > >
> >> >> > > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
> >> >> > > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
> >> >> > > > > >
> >> >> > > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> >> >> > > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >> >> > > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> >> >> > > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> >> >> > > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> >> >> > > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> >> > > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> >> >> > > > > > ----
> >> >> > > > > > v3: rename driver to dfl-fpga-fme-mgr
> >> >> > > > > >     implemented status callback for fpga manager
> >> >> > > > > >     rebased due to fpga api changes
> >> >> > > > > > ---
> >> >> > > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
> >> >> > > > > >  drivers/fpga/Kconfig                               |   6 +
> >> >> > > > > >  drivers/fpga/Makefile                              |   1 +
> >> >> > > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
> >> >> > > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
> >> >> > > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
> >> >> > > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> >> > > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
> >> >> > > > > >
> >> >> > > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> >> > > > > > new file mode 100644
> >> >> > > > > > index 0000000..2d4f917
> >> >> > > > > > --- /dev/null
> >> >> > > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
> >> >> > > > > > @@ -0,0 +1,8 @@
> >> >> > > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
> >> >> > > > > > +Date:          November 2017
> >> >> > > > > > +KernelVersion:  4.15
> >> >> > > > > > +Contact:       Wu Hao <hao.wu@intel.com>
> >> >> > > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
> >> >> > > > > > +               hardware. Userspace could use this information to check if
> >> >> > > > > > +               current hardware is compatible with given image before FPGA
> >> >> > > > > > +               programming.
> >> >> > > > >
> >> >> > > > > I'm a little confused by this.  I can understand that the PR bitstream
> >> >> > > > > has a dependency on the FPGA's static image, but I don't understand
> >> >> > > > > the dependency of the bistream on the hardware that is used to program
> >> >> > > > > the bitstream to the FPGA.
> >> >> > > >
> >> >> > > > Sorry for the confusion, the interface_id is used to indicate the version of
> >> >> > > > the hardware for partial reconfiguration (it's part of the static image of
> >> >> > > > the FPGA device). Will improve the description on this.
> >> >> > > >
> >> >> > >
> >> >> > > The interface_id expresses the compatibility of the static region with PR
> >> >> > > bitstreams generated for it. It changes every time a new static region is
> >> >> > > generated.
> >> >> > >
> >> >> > > Would it make more sense to have the interface_id exposed as part of the FME
> >> >> > > device (which represents the static region)? I'm not sure - it kind of also
> >> >> > > makes sense here, where you would have all the information in one place (if the
> >> >> > > interface_id matches, I can use this component to program a bitstream).
> >> >> >
> >> >> > Hi Enno
> >> >> >
> >> >> > Yes, this interface is under fpga-dfl-fme-mgr.0, and fpga-dfl-fme-mgr.0 is
> >> >> > under fpga-dfl-fme.0. It's part of the FME device for sure. From another
> >> >> > point of view, it means if anyone wants to do PR on this Intel FPGA device,
> >> >> > he needs to find the FME device firstly, and then check if any fpga manager
> >> >> > created under this FME device, if yes, check the interface_id before PR via
> >> >> > the FME device node ioctl.
> >> >>
> >> >> That sounds good, thank you!
> >> >>
> >> >> >
> >> >> > >
> >> >> > > Sorry for my limited understanding of the infrastructure - would this same
> >> >> > > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
> >> >> > > it would need to expose multiple interface_ids (or we'd have to track both
> >> >> > > interface IDs and an identifier for the target PR region).
> >> >> >
> >> >> > Yes, the fpga manager could be shared with different PR regions.
> >> >> >
> >> >> > Sorry, I'm not sure where we need to expose multiple interface_ids and why.
> >> >>
> >> >> It's basically a question of how to determine bitstream compatibility - either,
> >> >> there's a separate interface_id per reconfigurable region, or there is a single
> >> >> interface_id for the entire device. Both make sense from a certain perspective.
> >> >>
> >> >> If there are multiple interface_ids per device (one per region), the driver
> >> >> would need to expose all of them. If there's only a single one, the driver only
> >> >> exposes that one ID - compatibility would be determined by looking at both that
> >> >> single interface_id _and_ the identifier/number of the targeted region.
> >> >>
> >> >> I would prefer a separate interface_id per region - it seems more generic and
> >> >> flexible.
> >>
> >> Hi Enno,
> >>
> >> I agree with this.
> >>
> >> >
> >> > It's possible to have per region interface_id (or even both per dev interface_id
> >> > and per region interface_id at the same time), but per FME PR sub feature
> >> > implementation, it supports multiple PR regions, but only provide one interface
> >> > id, so at least in this case, it's not per-region information per my
> >> > understanding. We can consider it later when hardware really supports it. : )
> >>
> >> Hi Hao,
> >>
> >> I understand that in the case of this PR hardware, the region to
> >> program is selected when the region_id to program is written to a PR
> >> hardware control register.  For another example, Arria10 has a hard PR
> >> hardware and the PR bitstream lands in the area of the FPGA for which
> >> it was compiled.  In that case, for the PR bitstream to be compatible
> >> with a PR region, the layout of the edge connections also needs to be
> >> compatible, so compatibility is per-region in that case instead of
> >> per-PR hardware.
> >
> > Hi Alan,
> >
> > Thanks a lot for the explanation. :)
> >
> > I fully understand the consideration of adding per-region interface_id.
> >
> >> And besides, as I said yesterday, the hard PR
> >> hardware would not know what the static region ID is when this
> >> framework is used with such a device.
> >
> > Yes, is it possible that hard PR hardware with different versions, requires
> > different images or different methods for compatibility checking?
> 
> Because it is really hardware and not something in the FPGA fabric,
> the hard PR hardware isn't going to change versions very often.  It
> has to be designed to be flexible and not add any constraints on the
> PR regions.  If some feature is added or a bug is fixed, that's just a
> driver issue at most and should not affect PR region compatibility.
> PR region compatibility would only be dependent on the static FPGA
> image and the regions that are created in it.  It could be exported in
> terms of a single static region ID or per-region ID.
> 
> >
> >>
> >> That's why I think making the id per-region may be more future proof,
> >> even if it may see unnecessary in the case of the original blue bits
> >> this was written for.
> >
> > I feel that per-PR hardware interface id is useful in some cases, and maybe
> > in some cases, both per-PR hardware and per-region interface ids are needed
> > for its compatibility checking, so shall we leave developers to decide to
> > implement per-PR hardware or per-region or both interface ids based on their
> > own hardware implementations? How do you think? :)
> 
> That gives us 3 sets of id's.  Seems overly complicated and the
> userspace would have to figure out which set of id's to use.  I want
> to see an interface that isn't more complicated than it needs to be
> but still can be expected to be ok for the future (as far as we can
> anticipate).
> 
> Would per-region id's cause any problems that you can see?  I
> understand that the region id's would all be the same value for a
> given PR hardware in your use case, but that doesn't seem like it
> would be hard to implement or that it opens up some possible failure.

Hi Alan

No, I don't have any concern on per-region id at all. I will remove this
sysfs interface_id interface from this driver, and another patch to add
a common sysfs interface under the fpga-region. I feel maybe we can use
"compat_id" instead of "id" here to avoid confusion for that sysfs
interface under fpga-region, as we added a region_id to fpga_image_info,
and they have totally different purposes. How do you think?

Thanks
Hao

Re: [PATCH v3 14/21] fpga: dfl: add fpga manager platform driver for FME

[Alan Tull]

On Tue, Feb 6, 2018 at 10:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> On Tue, Feb 06, 2018 at 12:53:44PM -0600, Alan Tull wrote:
>> On Tue, Feb 6, 2018 at 12:47 AM, Wu Hao <hao.wu@intel.com> wrote:
>> > On Mon, Feb 05, 2018 at 10:25:54PM -0600, Alan Tull wrote:
>> >> On Mon, Feb 5, 2018 at 7:47 PM, Wu Hao <hao.wu@intel.com> wrote:
>> >> > On Mon, Feb 05, 2018 at 10:36:45AM -0800, Luebbers, Enno wrote:
>> >> >> Hi Hao,
>> >> >>
>> >> >> On Sun, Feb 04, 2018 at 05:37:06PM +0800, Wu Hao wrote:
>> >> >> > On Fri, Feb 02, 2018 at 04:26:26PM -0800, Luebbers, Enno wrote:
>> >> >> > > Hi Hao, Alan,
>> >> >> > >
>> >> >> > > On Fri, Feb 02, 2018 at 05:42:13PM +0800, Wu Hao wrote:
>> >> >> > > > On Thu, Feb 01, 2018 at 04:00:36PM -0600, Alan Tull wrote:
>> >> >> > > > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
>> >> >> > > > >
>> >> >> > > > > Hi Hao,
>> >> >> > > > >
>> >> >> > > > > A few comments below.   Besides that, looks good.
>> >> >> > > > >
>> >> >> > > > > > This patch adds fpga manager driver for FPGA Management Engine (FME). It
>> >> >> > > > > > implements fpga_manager_ops for FPGA Partial Reconfiguration function.
>> >> >> > > > > >
>> >> >> > > > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> >> >> > > > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> >> >> > > > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> >> >> > > > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> >> >> > > > > > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> >> >> > > > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> >> >> > > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> >> >> > > > > > ----
>> >> >> > > > > > v3: rename driver to dfl-fpga-fme-mgr
>> >> >> > > > > >     implemented status callback for fpga manager
>> >> >> > > > > >     rebased due to fpga api changes
>> >> >> > > > > > ---
>> >> >> > > > > >  .../ABI/testing/sysfs-platform-fpga-dfl-fme-mgr    |   8 +
>> >> >> > > > > >  drivers/fpga/Kconfig                               |   6 +
>> >> >> > > > > >  drivers/fpga/Makefile                              |   1 +
>> >> >> > > > > >  drivers/fpga/fpga-dfl-fme-mgr.c                    | 318 +++++++++++++++++++++
>> >> >> > > > > >  drivers/fpga/fpga-dfl.h                            |  39 ++-
>> >> >> > > > > >  5 files changed, 371 insertions(+), 1 deletion(-)
>> >> >> > > > > >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> >> >> > > > > >  create mode 100644 drivers/fpga/fpga-dfl-fme-mgr.c
>> >> >> > > > > >
>> >> >> > > > > > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> >> >> > > > > > new file mode 100644
>> >> >> > > > > > index 0000000..2d4f917
>> >> >> > > > > > --- /dev/null
>> >> >> > > > > > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-fme-mgr
>> >> >> > > > > > @@ -0,0 +1,8 @@
>> >> >> > > > > > +What:          /sys/bus/platform/devices/fpga-dfl-fme-mgr.0/interface_id
>> >> >> > > > > > +Date:          November 2017
>> >> >> > > > > > +KernelVersion:  4.15
>> >> >> > > > > > +Contact:       Wu Hao <hao.wu@intel.com>
>> >> >> > > > > > +Description:   Read-only. It returns interface id of partial reconfiguration
>> >> >> > > > > > +               hardware. Userspace could use this information to check if
>> >> >> > > > > > +               current hardware is compatible with given image before FPGA
>> >> >> > > > > > +               programming.
>> >> >> > > > >
>> >> >> > > > > I'm a little confused by this.  I can understand that the PR bitstream
>> >> >> > > > > has a dependency on the FPGA's static image, but I don't understand
>> >> >> > > > > the dependency of the bistream on the hardware that is used to program
>> >> >> > > > > the bitstream to the FPGA.
>> >> >> > > >
>> >> >> > > > Sorry for the confusion, the interface_id is used to indicate the version of
>> >> >> > > > the hardware for partial reconfiguration (it's part of the static image of
>> >> >> > > > the FPGA device). Will improve the description on this.
>> >> >> > > >
>> >> >> > >
>> >> >> > > The interface_id expresses the compatibility of the static region with PR
>> >> >> > > bitstreams generated for it. It changes every time a new static region is
>> >> >> > > generated.
>> >> >> > >
>> >> >> > > Would it make more sense to have the interface_id exposed as part of the FME
>> >> >> > > device (which represents the static region)? I'm not sure - it kind of also
>> >> >> > > makes sense here, where you would have all the information in one place (if the
>> >> >> > > interface_id matches, I can use this component to program a bitstream).
>> >> >> >
>> >> >> > Hi Enno
>> >> >> >
>> >> >> > Yes, this interface is under fpga-dfl-fme-mgr.0, and fpga-dfl-fme-mgr.0 is
>> >> >> > under fpga-dfl-fme.0. It's part of the FME device for sure. From another
>> >> >> > point of view, it means if anyone wants to do PR on this Intel FPGA device,
>> >> >> > he needs to find the FME device firstly, and then check if any fpga manager
>> >> >> > created under this FME device, if yes, check the interface_id before PR via
>> >> >> > the FME device node ioctl.
>> >> >>
>> >> >> That sounds good, thank you!
>> >> >>
>> >> >> >
>> >> >> > >
>> >> >> > > Sorry for my limited understanding of the infrastructure - would this same
>> >> >> > > "fpga-dfl-fme-mgr.0" be used for PR if we had multiple PR regions? In that case
>> >> >> > > it would need to expose multiple interface_ids (or we'd have to track both
>> >> >> > > interface IDs and an identifier for the target PR region).
>> >> >> >
>> >> >> > Yes, the fpga manager could be shared with different PR regions.
>> >> >> >
>> >> >> > Sorry, I'm not sure where we need to expose multiple interface_ids and why.
>> >> >>
>> >> >> It's basically a question of how to determine bitstream compatibility - either,
>> >> >> there's a separate interface_id per reconfigurable region, or there is a single
>> >> >> interface_id for the entire device. Both make sense from a certain perspective.
>> >> >>
>> >> >> If there are multiple interface_ids per device (one per region), the driver
>> >> >> would need to expose all of them. If there's only a single one, the driver only
>> >> >> exposes that one ID - compatibility would be determined by looking at both that
>> >> >> single interface_id _and_ the identifier/number of the targeted region.
>> >> >>
>> >> >> I would prefer a separate interface_id per region - it seems more generic and
>> >> >> flexible.
>> >>
>> >> Hi Enno,
>> >>
>> >> I agree with this.
>> >>
>> >> >
>> >> > It's possible to have per region interface_id (or even both per dev interface_id
>> >> > and per region interface_id at the same time), but per FME PR sub feature
>> >> > implementation, it supports multiple PR regions, but only provide one interface
>> >> > id, so at least in this case, it's not per-region information per my
>> >> > understanding. We can consider it later when hardware really supports it. : )
>> >>
>> >> Hi Hao,
>> >>
>> >> I understand that in the case of this PR hardware, the region to
>> >> program is selected when the region_id to program is written to a PR
>> >> hardware control register.  For another example, Arria10 has a hard PR
>> >> hardware and the PR bitstream lands in the area of the FPGA for which
>> >> it was compiled.  In that case, for the PR bitstream to be compatible
>> >> with a PR region, the layout of the edge connections also needs to be
>> >> compatible, so compatibility is per-region in that case instead of
>> >> per-PR hardware.
>> >
>> > Hi Alan,
>> >
>> > Thanks a lot for the explanation. :)
>> >
>> > I fully understand the consideration of adding per-region interface_id.
>> >
>> >> And besides, as I said yesterday, the hard PR
>> >> hardware would not know what the static region ID is when this
>> >> framework is used with such a device.
>> >
>> > Yes, is it possible that hard PR hardware with different versions, requires
>> > different images or different methods for compatibility checking?
>>
>> Because it is really hardware and not something in the FPGA fabric,
>> the hard PR hardware isn't going to change versions very often.  It
>> has to be designed to be flexible and not add any constraints on the
>> PR regions.  If some feature is added or a bug is fixed, that's just a
>> driver issue at most and should not affect PR region compatibility.
>> PR region compatibility would only be dependent on the static FPGA
>> image and the regions that are created in it.  It could be exported in
>> terms of a single static region ID or per-region ID.
>>
>> >
>> >>
>> >> That's why I think making the id per-region may be more future proof,
>> >> even if it may see unnecessary in the case of the original blue bits
>> >> this was written for.
>> >
>> > I feel that per-PR hardware interface id is useful in some cases, and maybe
>> > in some cases, both per-PR hardware and per-region interface ids are needed
>> > for its compatibility checking, so shall we leave developers to decide to
>> > implement per-PR hardware or per-region or both interface ids based on their
>> > own hardware implementations? How do you think? :)
>>
>> That gives us 3 sets of id's.  Seems overly complicated and the
>> userspace would have to figure out which set of id's to use.  I want
>> to see an interface that isn't more complicated than it needs to be
>> but still can be expected to be ok for the future (as far as we can
>> anticipate).
>>
>> Would per-region id's cause any problems that you can see?  I
>> understand that the region id's would all be the same value for a
>> given PR hardware in your use case, but that doesn't seem like it
>> would be hard to implement or that it opens up some possible failure.
>
> Hi Alan
>
> No, I don't have any concern on per-region id at all. I will remove this
> sysfs interface_id interface from this driver, and another patch to add
> a common sysfs interface under the fpga-region. I feel maybe we can use
> "compat_id" instead of "id" here to avoid confusion for that sysfs
> interface under fpga-region, as we added a region_id to fpga_image_info,
> and they have totally different purposes. How do you think?

Hi Hao,

Sounds good.  Yes, that that would keep things clear.

Alan

>
> Thanks
> Hao

[PATCH v3 15/21] fpga: dfl: add fpga bridge platform driver for FME

[Wu Hao]

This patch adds fpga bridge platform driver for FPGA Management Engine.
It implements the enable_set call back for fpga bridge.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: rename driver to fpga-dfl-fme-br
    remove useless dev_dbg in probe function.
    rebased due to fpga api change.
---
 drivers/fpga/Kconfig           |  6 +++
 drivers/fpga/Makefile          |  1 +
 drivers/fpga/fpga-dfl-fme-br.c | 87 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+)
 create mode 100644 drivers/fpga/fpga-dfl-fme-br.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 0171ecb..3804064 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -156,6 +156,12 @@ config FPGA_DFL_FME_MGR
 	help
 	  Say Y to enable FPGA Manager driver for FPGA Management Engine.
 
+config FPGA_DFL_FME_BRIDGE
+	tristate "FPGA DFL FME Bridge Driver"
+	depends on FPGA_DFL_FME
+	help
+	  Say Y to enable FPGA Bridge driver for FPGA Management Engine.
+
 config INTEL_FPGA_DFL_PCI
 	tristate "Intel FPGA DFL PCIe Device Driver"
 	depends on PCI && FPGA_DFL
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 6378580..3c8c4c7 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -32,6 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 obj-$(CONFIG_FPGA_DFL)			+= fpga-dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)		+= fpga-dfl-fme.o
 obj-$(CONFIG_FPGA_DFL_FME_MGR)		+= fpga-dfl-fme-mgr.o
+obj-$(CONFIG_FPGA_DFL_FME_BRIDGE)	+= fpga-dfl-fme-br.o
 
 fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
 
diff --git a/drivers/fpga/fpga-dfl-fme-br.c b/drivers/fpga/fpga-dfl-fme-br.c
new file mode 100644
index 0000000..db2603b
--- /dev/null
+++ b/drivers/fpga/fpga-dfl-fme-br.c
@@ -0,0 +1,87 @@
+/*
+ * FPGA Bridge Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/module.h>
+#include <linux/fpga/fpga-bridge.h>
+
+#include "fpga-dfl.h"
+#include "dfl-fme.h"
+
+static int fme_bridge_enable_set(struct fpga_bridge *bridge, bool enable)
+{
+	struct fme_br_pdata *pdata = bridge->priv;
+	int ret = 0;
+
+	if (enable)
+		fpga_port_enable(pdata->port);
+	else
+		ret = fpga_port_disable(pdata->port);
+
+	return ret;
+}
+
+static const struct fpga_bridge_ops fme_bridge_ops = {
+	.enable_set = fme_bridge_enable_set,
+};
+
+static int fme_br_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct fme_br_pdata *pdata = dev_get_platdata(dev);
+	struct fpga_bridge *br;
+	int ret;
+
+	br = devm_kzalloc(dev, sizeof(*br), GFP_KERNEL);
+	if (!br)
+		return -ENOMEM;
+
+	br->name = "Intel FPGA FME Bridge";
+	br->br_ops = &fme_bridge_ops;
+	br->priv = pdata;
+	br->parent = dev;
+	platform_set_drvdata(pdev, br);
+
+	ret = fpga_bridge_register(br);
+	if (ret)
+		dev_err(dev, "unable to register FPGA Bridge\n");
+
+	return ret;
+}
+
+static int fme_br_remove(struct platform_device *pdev)
+{
+	struct fpga_bridge *br = platform_get_drvdata(pdev);
+
+	fpga_bridge_unregister(br);
+
+	return 0;
+}
+
+static struct platform_driver fme_br_driver = {
+	.driver	= {
+		.name    = FPGA_DFL_FME_BRIDGE,
+	},
+	.probe   = fme_br_probe,
+	.remove  = fme_br_remove,
+};
+
+module_platform_driver(fme_br_driver);
+
+MODULE_DESCRIPTION("FPGA Bridge for FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:fpga-dfl-fme-bridge");
-- 
1.8.3.1

Re: [PATCH v3 15/21] fpga: dfl: add fpga bridge platform driver for FME

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

One fix below.  Besides that, please add my ack.

> This patch adds fpga bridge platform driver for FPGA Management Engine.
> It implements the enable_set call back for fpga bridge.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>

Acked-by: Alan Tull <atull@kernel.org>

> diff --git a/drivers/fpga/fpga-dfl-fme-br.c b/drivers/fpga/fpga-dfl-fme-br.c
> new file mode 100644
> index 0000000..db2603b
> --- /dev/null
> +++ b/drivers/fpga/fpga-dfl-fme-br.c
> @@ -0,0 +1,87 @@
> +/*
> + * FPGA Bridge Driver for FPGA Management Engine (FME)
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Wu Hao <hao.wu@intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2.

If we're using the SPDX, then please get rid of the above line.

> + * SPDX-License-Identifier: GPL-2.0
> + */

The SPDX line should be the first line of the file with a // (for .c
files), please see:

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/Documentation/process/license-rules.rst

Same thing for other files.

Alan

Re: [PATCH v3 15/21] fpga: dfl: add fpga bridge platform driver for FME

[Wu Hao]

On Wed, Jan 31, 2018 at 09:16:58AM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> One fix below.  Besides that, please add my ack.
> 
> > This patch adds fpga bridge platform driver for FPGA Management Engine.
> > It implements the enable_set call back for fpga bridge.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> 
> Acked-by: Alan Tull <atull@kernel.org>
> 
> > diff --git a/drivers/fpga/fpga-dfl-fme-br.c b/drivers/fpga/fpga-dfl-fme-br.c
> > new file mode 100644
> > index 0000000..db2603b
> > --- /dev/null
> > +++ b/drivers/fpga/fpga-dfl-fme-br.c
> > @@ -0,0 +1,87 @@
> > +/*
> > + * FPGA Bridge Driver for FPGA Management Engine (FME)
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Joseph Grecco <joe.grecco@intel.com>
> > + *   Enno Luebbers <enno.luebbers@intel.com>
> > + *   Tim Whisonant <tim.whisonant@intel.com>
> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > + *   Henry Mitchel <henry.mitchel@intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2.
> 
> If we're using the SPDX, then please get rid of the above line.
> 
> > + * SPDX-License-Identifier: GPL-2.0
> > + */
> 
> The SPDX line should be the first line of the file with a // (for .c
> files), please see:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/Documentation/process/license-rules.rst
> 
> Same thing for other files.

Thanks for the review.

Sure, will fix this for all driver files in the next version.

Thanks
Hao

> 
> Alan

Re: [PATCH v3 15/21] fpga: dfl: add fpga bridge platform driver for FME

[Moritz Fischer]

[-- Attachment #1: Type: text/plain, Size: 2246 bytes --]

Hi Hao,

On Thu, Feb 01, 2018 at 01:15:43PM +0800, Wu Hao wrote:
> On Wed, Jan 31, 2018 at 09:16:58AM -0600, Alan Tull wrote:
> > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > 
> > Hi Hao,
> > 
> > One fix below.  Besides that, please add my ack.
> > 
> > > This patch adds fpga bridge platform driver for FPGA Management Engine.
> > > It implements the enable_set call back for fpga bridge.
> > >
> > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > 
> > Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
> > 
> > > diff --git a/drivers/fpga/fpga-dfl-fme-br.c b/drivers/fpga/fpga-dfl-fme-br.c
> > > new file mode 100644
> > > index 0000000..db2603b
> > > --- /dev/null
> > > +++ b/drivers/fpga/fpga-dfl-fme-br.c
> > > @@ -0,0 +1,87 @@
> > > +/*
> > > + * FPGA Bridge Driver for FPGA Management Engine (FME)
> > > + *
> > > + * Copyright (C) 2017 Intel Corporation, Inc.
> > > + *
> > > + * Authors:
> > > + *   Wu Hao <hao.wu@intel.com>
> > > + *   Joseph Grecco <joe.grecco@intel.com>
> > > + *   Enno Luebbers <enno.luebbers@intel.com>
> > > + *   Tim Whisonant <tim.whisonant@intel.com>
> > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > > + *   Henry Mitchel <henry.mitchel@intel.com>
> > > + *
> > > + * This work is licensed under the terms of the GNU GPL version 2.
> > 
> > If we're using the SPDX, then please get rid of the above line.
> > 
> > > + * SPDX-License-Identifier: GPL-2.0
> > > + */
> > 
> > The SPDX line should be the first line of the file with a // (for .c
> > files), please see:
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/Documentation/process/license-rules.rst
> > 
> > Same thing for other files.
> 
> Thanks for the review.
> 
> Sure, will fix this for all driver files in the next version.
> 
> Thanks
> Hao
> 
> > 
> > Alan

Looks good, feel free to add my Acked-by: with fixes Alan requested,

Moritz

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v3 16/21] fpga: dfl: add fpga region platform driver for FME

[Wu Hao]

This patch adds fpga region platform driver for FPGA Management Engine.
It register an fpga region with given fpga manager / bridge device.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: rename driver to fpga-dfl-fme-region
    fix fpga_mgr_put order problem in remove function.
    rebased due to fpga api changes.
---
 drivers/fpga/Kconfig               |  6 +++
 drivers/fpga/Makefile              |  1 +
 drivers/fpga/fpga-dfl-fme-region.c | 92 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+)
 create mode 100644 drivers/fpga/fpga-dfl-fme-region.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 3804064..2df9dfd 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -162,6 +162,12 @@ config FPGA_DFL_FME_BRIDGE
 	help
 	  Say Y to enable FPGA Bridge driver for FPGA Management Engine.
 
+config FPGA_DFL_FME_REGION
+	tristate "FPGA DFL FME Region Driver"
+	depends on FPGA_DFL_FME
+	help
+	  Say Y to enable FPGA Region driver for FPGA Management Engine.
+
 config INTEL_FPGA_DFL_PCI
 	tristate "Intel FPGA DFL PCIe Device Driver"
 	depends on PCI && FPGA_DFL
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 3c8c4c7..11777d3 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -33,6 +33,7 @@ obj-$(CONFIG_FPGA_DFL)			+= fpga-dfl.o
 obj-$(CONFIG_FPGA_DFL_FME)		+= fpga-dfl-fme.o
 obj-$(CONFIG_FPGA_DFL_FME_MGR)		+= fpga-dfl-fme-mgr.o
 obj-$(CONFIG_FPGA_DFL_FME_BRIDGE)	+= fpga-dfl-fme-br.o
+obj-$(CONFIG_FPGA_DFL_FME_REGION)	+= fpga-dfl-fme-region.o
 
 fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
 
diff --git a/drivers/fpga/fpga-dfl-fme-region.c b/drivers/fpga/fpga-dfl-fme-region.c
new file mode 100644
index 0000000..0b988ab
--- /dev/null
+++ b/drivers/fpga/fpga-dfl-fme-region.c
@@ -0,0 +1,92 @@
+/*
+ * FPGA Region Driver for FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/module.h>
+#include <linux/fpga/fpga-region.h>
+
+#include "fpga-dfl.h"
+#include "dfl-fme.h"
+
+static int fme_region_get_bridges(struct fpga_region *region)
+{
+	struct fme_region_pdata *pdata = region->priv;
+	struct device *dev = &pdata->br->dev;
+
+	return fpga_bridge_get_to_list(dev, region->info, &region->bridge_list);
+}
+
+static int fme_region_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct fme_region_pdata *pdata = dev_get_platdata(dev);
+	struct fpga_region *region;
+	struct fpga_manager *mgr;
+	int ret;
+
+	mgr = fpga_mgr_get(&pdata->mgr->dev);
+	if (IS_ERR(mgr))
+		return -EPROBE_DEFER;
+
+	region = devm_kzalloc(dev, sizeof(*region), GFP_KERNEL);
+	if (!region) {
+		ret = -ENOMEM;
+		goto eprobe_mgr_put;
+	}
+
+	region->mgr = mgr;
+	region->get_bridges = fme_region_get_bridges;
+	region->priv = pdata;
+	region->parent = dev;
+	platform_set_drvdata(pdev, region);
+
+	ret = fpga_region_register(region);
+	if (ret)
+		goto eprobe_mgr_put;
+
+	dev_dbg(dev, "DFL FME FPGA Region probed\n");
+
+	return 0;
+
+eprobe_mgr_put:
+	fpga_mgr_put(mgr);
+	return ret;
+}
+
+static int fme_region_remove(struct platform_device *pdev)
+{
+	struct fpga_region *region = dev_get_drvdata(&pdev->dev);
+
+	fpga_region_unregister(region);
+	fpga_mgr_put(region->mgr);
+
+	return 0;
+}
+
+static struct platform_driver fme_region_driver = {
+	.driver	= {
+		.name    = FPGA_DFL_FME_REGION,
+	},
+	.probe   = fme_region_probe,
+	.remove  = fme_region_remove,
+};
+
+module_platform_driver(fme_region_driver);
+
+MODULE_DESCRIPTION("FPGA Region for FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:fpga-dfl-fme-region");
-- 
1.8.3.1

Re: [PATCH v3 16/21] fpga: dfl: add fpga region platform driver for FME

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

Just fix the SPDX, otherwise it looks good.

> This patch adds fpga region platform driver for FPGA Management Engine.
> It register an fpga region with given fpga manager / bridge device.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
Acked-by: Alan Tull <atull@kernel.org>


> diff --git a/drivers/fpga/fpga-dfl-fme-region.c b/drivers/fpga/fpga-dfl-fme-region.c
> new file mode 100644
> index 0000000..0b988ab
> --- /dev/null
> +++ b/drivers/fpga/fpga-dfl-fme-region.c
> @@ -0,0 +1,92 @@
> +/*
> + * FPGA Region Driver for FPGA Management Engine (FME)
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Wu Hao <hao.wu@intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2.
> + * SPDX-License-Identifier: GPL-2.0
> + */
> +
> +#include <linux/module.h>

Thanks,
Alan

Re: [PATCH v3 16/21] fpga: dfl: add fpga region platform driver for FME

[Wu Hao]

On Wed, Jan 31, 2018 at 02:46:28PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> Just fix the SPDX, otherwise it looks good.

Sure, will fix it. Thanks for the review. :)

Thanks
Hao

> 
> > This patch adds fpga region platform driver for FPGA Management Engine.
> > It register an fpga region with given fpga manager / bridge device.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> Acked-by: Alan Tull <atull@kernel.org>
> 
> 
> > diff --git a/drivers/fpga/fpga-dfl-fme-region.c b/drivers/fpga/fpga-dfl-fme-region.c
> > new file mode 100644
> > index 0000000..0b988ab
> > --- /dev/null
> > +++ b/drivers/fpga/fpga-dfl-fme-region.c
> > @@ -0,0 +1,92 @@
> > +/*
> > + * FPGA Region Driver for FPGA Management Engine (FME)
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Joseph Grecco <joe.grecco@intel.com>
> > + *   Enno Luebbers <enno.luebbers@intel.com>
> > + *   Tim Whisonant <tim.whisonant@intel.com>
> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > + *   Henry Mitchel <henry.mitchel@intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2.
> > + * SPDX-License-Identifier: GPL-2.0
> > + */
> > +
> > +#include <linux/module.h>
> 
> Thanks,
> Alan

Re: [PATCH v3 16/21] fpga: dfl: add fpga region platform driver for FME

[Moritz Fischer]

[-- Attachment #1: Type: text/plain, Size: 1851 bytes --]

On Thu, Feb 01, 2018 at 01:23:45PM +0800, Wu Hao wrote:
> On Wed, Jan 31, 2018 at 02:46:28PM -0600, Alan Tull wrote:
> > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > 
> > Hi Hao,
> > 
> > Just fix the SPDX, otherwise it looks good.
> 
> Sure, will fix it. Thanks for the review. :)
> 
> Thanks
> Hao
> 
> > 
> > > This patch adds fpga region platform driver for FPGA Management Engine.
> > > It register an fpga region with given fpga manager / bridge device.
> > >
> > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
> > 
> > 
> > > diff --git a/drivers/fpga/fpga-dfl-fme-region.c b/drivers/fpga/fpga-dfl-fme-region.c
> > > new file mode 100644
> > > index 0000000..0b988ab
> > > --- /dev/null
> > > +++ b/drivers/fpga/fpga-dfl-fme-region.c
> > > @@ -0,0 +1,92 @@
> > > +/*
> > > + * FPGA Region Driver for FPGA Management Engine (FME)
> > > + *
> > > + * Copyright (C) 2017 Intel Corporation, Inc.
> > > + *
> > > + * Authors:
> > > + *   Wu Hao <hao.wu@intel.com>
> > > + *   Joseph Grecco <joe.grecco@intel.com>
> > > + *   Enno Luebbers <enno.luebbers@intel.com>
> > > + *   Tim Whisonant <tim.whisonant@intel.com>
> > > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > > + *   Henry Mitchel <henry.mitchel@intel.com>
> > > + *
> > > + * This work is licensed under the terms of the GNU GPL version 2.
> > > + * SPDX-License-Identifier: GPL-2.0
> > > + */
> > > +
> > > +#include <linux/module.h>
> > 
> > Thanks,
> > Alan

Thanks,
Moritz

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v3 17/21] fpga: dfl: add FPGA Accelerated Function Unit driver basic framework

[Wu Hao]

On DFL FPGA devices, the Accelerated Function Unit (AFU), can be
reprogrammed for different functions. It connects to the FPGA
infrastructure("blue bistream") via a Port. Port CSRs are implemented
separately from the AFU CSRs to provide control and status of the Port.
Once valid green bitstream is programmed into the AFU, it allows access
to the AFU CSRs in the AFU MMIO space.

This patch only implements basic driver framework for AFU, including
device file operation framework.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: rename driver to dfl-afu-main
---
 drivers/fpga/Kconfig        |   9 +++
 drivers/fpga/Makefile       |   2 +
 drivers/fpga/dfl-afu-main.c | 161 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 172 insertions(+)
 create mode 100644 drivers/fpga/dfl-afu-main.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 2df9dfd..fa1905b 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -168,6 +168,15 @@ config FPGA_DFL_FME_REGION
 	help
 	  Say Y to enable FPGA Region driver for FPGA Management Engine.
 
+config FPGA_DFL_AFU
+	tristate "FPGA DFL AFU Driver"
+	depends on FPGA_DFL
+	help
+	  This is the driver for FPGA Accelerated Function Unit (AFU) which
+	  implements AFU and Port management features. A User AFU connects
+	  to the FPGA infrastructure via a Port. There may be more than 1
+	  Port/AFU per DFL based FPGA device.
+
 config INTEL_FPGA_DFL_PCI
 	tristate "Intel FPGA DFL PCIe Device Driver"
 	depends on PCI && FPGA_DFL
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 11777d3..e234fa8 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -34,8 +34,10 @@ obj-$(CONFIG_FPGA_DFL_FME)		+= fpga-dfl-fme.o
 obj-$(CONFIG_FPGA_DFL_FME_MGR)		+= fpga-dfl-fme-mgr.o
 obj-$(CONFIG_FPGA_DFL_FME_BRIDGE)	+= fpga-dfl-fme-br.o
 obj-$(CONFIG_FPGA_DFL_FME_REGION)	+= fpga-dfl-fme-region.o
+obj-$(CONFIG_FPGA_DFL_AFU)		+= fpga-dfl-afu.o
 
 fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
+fpga-dfl-afu-objs := dfl-afu-main.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_INTEL_FPGA_DFL_PCI)	+= intel-dfl-pci.o
diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
new file mode 100644
index 0000000..d9f4b81
--- /dev/null
+++ b/drivers/fpga/dfl-afu-main.c
@@ -0,0 +1,161 @@
+/*
+ * Driver for FPGA Accelerated Function Unit (AFU)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "fpga-dfl.h"
+
+static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT HDR Init.\n");
+
+	return 0;
+}
+
+static void port_hdr_uinit(struct platform_device *pdev,
+			   struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
+}
+
+static const struct feature_ops port_hdr_ops = {
+	.init = port_hdr_init,
+	.uinit = port_hdr_uinit,
+};
+
+static struct feature_driver port_feature_drvs[] = {
+	{
+		.name = PORT_FEATURE_HEADER,
+		.ops = &port_hdr_ops,
+	},
+	{
+		.ops = NULL,
+	}
+};
+
+static int afu_open(struct inode *inode, struct file *filp)
+{
+	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
+	struct feature_platform_data *pdata;
+	int ret;
+
+	pdata = dev_get_platdata(&fdev->dev);
+	if (WARN_ON(!pdata))
+		return -ENODEV;
+
+	ret = feature_dev_use_begin(pdata);
+	if (ret)
+		return ret;
+
+	dev_dbg(&fdev->dev, "Device File Open\n");
+	filp->private_data = fdev;
+
+	return 0;
+}
+
+static int afu_release(struct inode *inode, struct file *filp)
+{
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	dev_dbg(&pdev->dev, "Device File Release\n");
+
+	feature_dev_use_end(pdata);
+
+	return 0;
+}
+
+static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct feature *f;
+	long ret;
+
+	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
+
+	switch (cmd) {
+	default:
+		/*
+		 * Let sub-feature's ioctl function to handle the cmd
+		 * Sub-feature's ioctl returns -ENODEV when cmd is not
+		 * handled in this sub feature, and returns 0 and other
+		 * error code if cmd is handled.
+		 */
+		fpga_dev_for_each_feature(pdata, f)
+			if (f->ops && f->ops->ioctl) {
+				ret = f->ops->ioctl(pdev, f, cmd, arg);
+				if (ret == -ENODEV)
+					continue;
+				else
+					return ret;
+			}
+	}
+
+	return -EINVAL;
+}
+
+static const struct file_operations afu_fops = {
+	.owner = THIS_MODULE,
+	.open = afu_open,
+	.release = afu_release,
+	.unlocked_ioctl = afu_ioctl,
+};
+
+static int afu_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	dev_dbg(&pdev->dev, "%s\n", __func__);
+
+	ret = fpga_dev_feature_init(pdev, port_feature_drvs);
+	if (ret)
+		return ret;
+
+	ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
+	if (ret)
+		fpga_dev_feature_uinit(pdev);
+
+	return ret;
+}
+
+static int afu_remove(struct platform_device *pdev)
+{
+	dev_dbg(&pdev->dev, "%s\n", __func__);
+
+	fpga_dev_feature_uinit(pdev);
+	fpga_unregister_dev_ops(pdev);
+
+	return 0;
+}
+
+static struct platform_driver afu_driver = {
+	.driver	= {
+		.name    = FPGA_FEATURE_DEV_PORT,
+	},
+	.probe   = afu_probe,
+	.remove  = afu_remove,
+};
+
+module_platform_driver(afu_driver);
+
+MODULE_DESCRIPTION("FPGA Accelerated Function Unit driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:fpga-dfl-port");
-- 
1.8.3.1

[PATCH v3 18/21] fpga: dfl: afu: add header sub feature support

[Wu Hao]

The port header register set is always present for port, it is mainly
for capability, control and status of the ports that AFU connected to.

This patch implements header sub feature support. Below user interfaces
are created by this patch.

Sysfs interface:
* /sys/class/fpga_region/<regionX>/<fpga-dfl-port.x>/id
  Read-only. Port ID.

Ioctl interface:
* FPGA_PORT_RESET
  Reset the FPGA Port and its AFU.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v3: rename driver name to fpga-dfl-afu
    add more description for reset ioctl.
    fix some checkpatch issues.
---
 .../ABI/testing/sysfs-platform-fpga-dfl-afu        |  7 ++++
 drivers/fpga/dfl-afu-main.c                        | 44 +++++++++++++++++++++-
 include/uapi/linux/fpga-dfl.h                      | 17 +++++++++
 3 files changed, 67 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu

diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
new file mode 100644
index 0000000..f4bcd94
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
@@ -0,0 +1,7 @@
+What:		/sys/bus/platform/devices/fpga-dfl-port.0/id
+Date:		November 2017
+KernelVersion:  4.15
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. It returns id of this port. One DFL FPGA device
+		may have more than one port. Userspace could use this id to
+		distinguish different ports under same FPGA device.
diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index d9f4b81..b01376c 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c
@@ -18,25 +18,66 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/fpga-dfl.h>
 
 #include "fpga-dfl.h"
 
+static ssize_t
+id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	int id = fpga_port_id(to_platform_device(dev));
+
+	return scnprintf(buf, PAGE_SIZE, "%d\n", id);
+}
+static DEVICE_ATTR_RO(id);
+
+static const struct attribute *port_hdr_attrs[] = {
+	&dev_attr_id.attr,
+	NULL,
+};
+
 static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "PORT HDR Init.\n");
 
-	return 0;
+	fpga_port_reset(pdev);
+
+	return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);
 }
 
 static void port_hdr_uinit(struct platform_device *pdev,
 			   struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
+
+	sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
+}
+
+static long
+port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
+	       unsigned int cmd, unsigned long arg)
+{
+	long ret;
+
+	switch (cmd) {
+	case FPGA_PORT_RESET:
+		if (!arg)
+			ret = fpga_port_reset(pdev);
+		else
+			ret = -EINVAL;
+		break;
+	default:
+		dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
+		ret = -ENODEV;
+	}
+
+	return ret;
 }
 
 static const struct feature_ops port_hdr_ops = {
 	.init = port_hdr_init,
 	.uinit = port_hdr_uinit,
+	.ioctl = port_hdr_ioctl,
 };
 
 static struct feature_driver port_feature_drvs[] = {
@@ -76,6 +117,7 @@ static int afu_release(struct inode *inode, struct file *filp)
 
 	dev_dbg(&pdev->dev, "Device File Release\n");
 
+	fpga_port_reset(pdev);
 	feature_dev_use_end(pdata);
 
 	return 0;
diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 75bdf88..9bf273d 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h
@@ -30,8 +30,11 @@
 #define FPGA_MAGIC 0xB6
 
 #define FPGA_BASE 0
+#define PORT_BASE 0x40
 #define FME_BASE 0x80
 
+/* Common IOCTLs for both FME and AFU file descriptor */
+
 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
  *
@@ -50,6 +53,20 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for AFU file descriptor */
+
+/**
+ * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
+ *
+ * Reset the FPGA Port and its AFU. No parameters are supported.
+ * Userspace can do Port reset at any time, e.g during DMA or PR. But
+ * it should never cause any system level issue, only functional failure
+ * (e.g DMA or PR operation failure) and be recoverable from the failure.
+ * Return: 0 on success, -errno of failure
+ */
+
+#define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
+
 /* IOCTLs for FME file descriptor */
 
 /**
-- 
1.8.3.1

Re: [PATCH v3 18/21] fpga: dfl: afu: add header sub feature support

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

Thanks for documenting the reset.  Please add this documentation to
the function header that is in fpga-dfl.h.

One question for clarification below.

This patch looks good.

> The port header register set is always present for port, it is mainly
> for capability, control and status of the ports that AFU connected to.
>
> This patch implements header sub feature support. Below user interfaces
> are created by this patch.
>
> Sysfs interface:
> * /sys/class/fpga_region/<regionX>/<fpga-dfl-port.x>/id
>   Read-only. Port ID.
>
> Ioctl interface:
> * FPGA_PORT_RESET
>   Reset the FPGA Port and its AFU.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>

Acked-by: Alan Tull <atull@kernel.org>

> ----
> v3: rename driver name to fpga-dfl-afu
>     add more description for reset ioctl.
>     fix some checkpatch issues.
> ---
>  .../ABI/testing/sysfs-platform-fpga-dfl-afu        |  7 ++++
>  drivers/fpga/dfl-afu-main.c                        | 44 +++++++++++++++++++++-
>  include/uapi/linux/fpga-dfl.h                      | 17 +++++++++
>  3 files changed, 67 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
>
> diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
> new file mode 100644
> index 0000000..f4bcd94
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
> @@ -0,0 +1,7 @@
> +What:          /sys/bus/platform/devices/fpga-dfl-port.0/id
> +Date:          November 2017
> +KernelVersion:  4.15
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read-only. It returns id of this port. One DFL FPGA device
> +               may have more than one port. Userspace could use this id to
> +               distinguish different ports under same FPGA device.

Potentially >1 port per FPGA, but only one port per fpga-region, right?

> diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
> index d9f4b81..b01376c 100644
> --- a/drivers/fpga/dfl-afu-main.c
> +++ b/drivers/fpga/dfl-afu-main.c
> @@ -18,25 +18,66 @@
>
>  #include <linux/kernel.h>
>  #include <linux/module.h>
> +#include <linux/fpga-dfl.h>
>
>  #include "fpga-dfl.h"
>
> +static ssize_t
> +id_show(struct device *dev, struct device_attribute *attr, char *buf)
> +{
> +       int id = fpga_port_id(to_platform_device(dev));
> +
> +       return scnprintf(buf, PAGE_SIZE, "%d\n", id);
> +}
> +static DEVICE_ATTR_RO(id);
> +
> +static const struct attribute *port_hdr_attrs[] = {
> +       &dev_attr_id.attr,
> +       NULL,
> +};
> +
>  static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
>  {
>         dev_dbg(&pdev->dev, "PORT HDR Init.\n");
>
> -       return 0;
> +       fpga_port_reset(pdev);
> +
> +       return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);
>  }
>
>  static void port_hdr_uinit(struct platform_device *pdev,
>                            struct feature *feature)
>  {
>         dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
> +
> +       sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
> +}
> +
> +static long
> +port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
> +              unsigned int cmd, unsigned long arg)
> +{
> +       long ret;
> +
> +       switch (cmd) {
> +       case FPGA_PORT_RESET:
> +               if (!arg)
> +                       ret = fpga_port_reset(pdev);
> +               else
> +                       ret = -EINVAL;
> +               break;
> +       default:
> +               dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
> +               ret = -ENODEV;
> +       }
> +
> +       return ret;
>  }
>
>  static const struct feature_ops port_hdr_ops = {
>         .init = port_hdr_init,
>         .uinit = port_hdr_uinit,
> +       .ioctl = port_hdr_ioctl,
>  };
>
>  static struct feature_driver port_feature_drvs[] = {
> @@ -76,6 +117,7 @@ static int afu_release(struct inode *inode, struct file *filp)
>
>         dev_dbg(&pdev->dev, "Device File Release\n");
>
> +       fpga_port_reset(pdev);
>         feature_dev_use_end(pdata);
>
>         return 0;
> diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
> index 75bdf88..9bf273d 100644
> --- a/include/uapi/linux/fpga-dfl.h
> +++ b/include/uapi/linux/fpga-dfl.h
> @@ -30,8 +30,11 @@
>  #define FPGA_MAGIC 0xB6
>
>  #define FPGA_BASE 0
> +#define PORT_BASE 0x40
>  #define FME_BASE 0x80
>
> +/* Common IOCTLs for both FME and AFU file descriptor */
> +
>  /**
>   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
>   *
> @@ -50,6 +53,20 @@
>
>  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>
> +/* IOCTLs for AFU file descriptor */
> +
> +/**
> + * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
> + *
> + * Reset the FPGA Port and its AFU. No parameters are supported.
> + * Userspace can do Port reset at any time, e.g during DMA or PR. But
> + * it should never cause any system level issue, only functional failure
> + * (e.g DMA or PR operation failure) and be recoverable from the failure.
> + * Return: 0 on success, -errno of failure
> + */
> +
> +#define FPGA_PORT_RESET                _IO(FPGA_MAGIC, PORT_BASE + 0)
> +
>  /* IOCTLs for FME file descriptor */
>
>  /**
> --
> 1.8.3.1
>

Re: [PATCH v3 18/21] fpga: dfl: afu: add header sub feature support

[Wu Hao]

On Mon, Feb 12, 2018 at 11:43:28AM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> Thanks for documenting the reset.  Please add this documentation to
> the function header that is in fpga-dfl.h.

Sure, thanks for the review. 

I will add the description to the function header.

> 
> One question for clarification below.
> 
> This patch looks good.
> 
> > The port header register set is always present for port, it is mainly
> > for capability, control and status of the ports that AFU connected to.
> >
> > This patch implements header sub feature support. Below user interfaces
> > are created by this patch.
> >
> > Sysfs interface:
> > * /sys/class/fpga_region/<regionX>/<fpga-dfl-port.x>/id
> >   Read-only. Port ID.
> >
> > Ioctl interface:
> > * FPGA_PORT_RESET
> >   Reset the FPGA Port and its AFU.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> 
> Acked-by: Alan Tull <atull@kernel.org>
> 
> > ----
> > v3: rename driver name to fpga-dfl-afu
> >     add more description for reset ioctl.
> >     fix some checkpatch issues.
> > ---
> >  .../ABI/testing/sysfs-platform-fpga-dfl-afu        |  7 ++++
> >  drivers/fpga/dfl-afu-main.c                        | 44 +++++++++++++++++++++-
> >  include/uapi/linux/fpga-dfl.h                      | 17 +++++++++
> >  3 files changed, 67 insertions(+), 1 deletion(-)
> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
> >
> > diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
> > new file mode 100644
> > index 0000000..f4bcd94
> > --- /dev/null
> > +++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
> > @@ -0,0 +1,7 @@
> > +What:          /sys/bus/platform/devices/fpga-dfl-port.0/id
> > +Date:          November 2017
> > +KernelVersion:  4.15
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read-only. It returns id of this port. One DFL FPGA device
> > +               may have more than one port. Userspace could use this id to
> > +               distinguish different ports under same FPGA device.
> 
> Potentially >1 port per FPGA, but only one port per fpga-region, right?

Yes.

Thanks
Hao

> 
> > diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
> > index d9f4b81..b01376c 100644
> > --- a/drivers/fpga/dfl-afu-main.c
> > +++ b/drivers/fpga/dfl-afu-main.c
> > @@ -18,25 +18,66 @@
> >
> >  #include <linux/kernel.h>
> >  #include <linux/module.h>
> > +#include <linux/fpga-dfl.h>
> >
> >  #include "fpga-dfl.h"
> >
> > +static ssize_t
> > +id_show(struct device *dev, struct device_attribute *attr, char *buf)
> > +{
> > +       int id = fpga_port_id(to_platform_device(dev));
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "%d\n", id);
> > +}
> > +static DEVICE_ATTR_RO(id);
> > +
> > +static const struct attribute *port_hdr_attrs[] = {
> > +       &dev_attr_id.attr,
> > +       NULL,
> > +};
> > +
> >  static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
> >  {
> >         dev_dbg(&pdev->dev, "PORT HDR Init.\n");
> >
> > -       return 0;
> > +       fpga_port_reset(pdev);
> > +
> > +       return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);
> >  }
> >
> >  static void port_hdr_uinit(struct platform_device *pdev,
> >                            struct feature *feature)
> >  {
> >         dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
> > +
> > +       sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
> > +}
> > +
> > +static long
> > +port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
> > +              unsigned int cmd, unsigned long arg)
> > +{
> > +       long ret;
> > +
> > +       switch (cmd) {
> > +       case FPGA_PORT_RESET:
> > +               if (!arg)
> > +                       ret = fpga_port_reset(pdev);
> > +               else
> > +                       ret = -EINVAL;
> > +               break;
> > +       default:
> > +               dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
> > +               ret = -ENODEV;
> > +       }
> > +
> > +       return ret;
> >  }
> >
> >  static const struct feature_ops port_hdr_ops = {
> >         .init = port_hdr_init,
> >         .uinit = port_hdr_uinit,
> > +       .ioctl = port_hdr_ioctl,
> >  };
> >
> >  static struct feature_driver port_feature_drvs[] = {
> > @@ -76,6 +117,7 @@ static int afu_release(struct inode *inode, struct file *filp)
> >
> >         dev_dbg(&pdev->dev, "Device File Release\n");
> >
> > +       fpga_port_reset(pdev);
> >         feature_dev_use_end(pdata);
> >
> >         return 0;
> > diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
> > index 75bdf88..9bf273d 100644
> > --- a/include/uapi/linux/fpga-dfl.h
> > +++ b/include/uapi/linux/fpga-dfl.h
> > @@ -30,8 +30,11 @@
> >  #define FPGA_MAGIC 0xB6
> >
> >  #define FPGA_BASE 0
> > +#define PORT_BASE 0x40
> >  #define FME_BASE 0x80
> >
> > +/* Common IOCTLs for both FME and AFU file descriptor */
> > +
> >  /**
> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> >   *
> > @@ -50,6 +53,20 @@
> >
> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
> >
> > +/* IOCTLs for AFU file descriptor */
> > +
> > +/**
> > + * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
> > + *
> > + * Reset the FPGA Port and its AFU. No parameters are supported.
> > + * Userspace can do Port reset at any time, e.g during DMA or PR. But
> > + * it should never cause any system level issue, only functional failure
> > + * (e.g DMA or PR operation failure) and be recoverable from the failure.
> > + * Return: 0 on success, -errno of failure
> > + */
> > +
> > +#define FPGA_PORT_RESET                _IO(FPGA_MAGIC, PORT_BASE + 0)
> > +
> >  /* IOCTLs for FME file descriptor */
> >
> >  /**
> > --
> > 1.8.3.1
> >

[PATCH v3 19/21] fpga: dfl: afu: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Wu Hao]

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
This patch implements above 2 ioctls in FPGA Accelerated Function Unit
(AFU) driver.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
----
v2: rebased
v3: rebased as driver renamed to fpga-dfl-afu
    fix one checkpatch issue
---
 drivers/fpga/dfl-afu-main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index b01376c..d3d534c 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c
@@ -123,6 +123,13 @@ static int afu_release(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
+				      unsigned long arg)
+{
+	/* No extension support for now */
+	return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;
@@ -133,6 +140,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
 
 	switch (cmd) {
+	case FPGA_GET_API_VERSION:
+		return FPGA_API_VERSION;
+	case FPGA_CHECK_EXTENSION:
+		return afu_ioctl_check_extension(pdata, arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
-- 
1.8.3.1

Re: [PATCH v3 19/21] fpga: dfl: afu: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

I'm adding my "Acked-by' below.  When you post v4, please add it so
that we can keep track of what got acked.

Thanks,
Alan

> FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> need to be supported by all feature devices drivers including FME and AFU.
> This patch implements above 2 ioctls in FPGA Accelerated Function Unit
> (AFU) driver.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
Acked-by: Alan Tull <atull@kernel.org>

Re: [PATCH v3 19/21] fpga: dfl: afu: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Wu Hao]

On Wed, Jan 31, 2018 at 08:52:36AM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> I'm adding my "Acked-by' below.  When you post v4, please add it so
> that we can keep track of what got acked.

Sure, thanks a lot for the code review. :)

Hao

> 
> Thanks,
> Alan
> 
> > FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> > need to be supported by all feature devices drivers including FME and AFU.
> > This patch implements above 2 ioctls in FPGA Accelerated Function Unit
> > (AFU) driver.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> Acked-by: Alan Tull <atull@kernel.org>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 19/21] fpga: dfl: afu: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Moritz Fischer]

[-- Attachment #1: Type: text/plain, Size: 1398 bytes --]

On Thu, Feb 01, 2018 at 01:16:25PM +0800, Wu Hao wrote:
> On Wed, Jan 31, 2018 at 08:52:36AM -0600, Alan Tull wrote:
> > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > 
> > Hi Hao,
> > 
> > I'm adding my "Acked-by' below.  When you post v4, please add it so
> > that we can keep track of what got acked.
> 
> Sure, thanks a lot for the code review. :)
> 
> Hao
> 
> > 
> > Thanks,
> > Alan
> > 
> > > FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> > > need to be supported by all feature devices drivers including FME and AFU.
> > > This patch implements above 2 ioctls in FPGA Accelerated Function Unit
> > > (AFU) driver.
> > >
> > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Looks good,

Moritz

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v3 19/21] fpga: dfl: afu: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Wu Hao]

On Thu, Feb 01, 2018 at 04:13:09PM +0100, Moritz Fischer wrote:
> On Thu, Feb 01, 2018 at 01:16:25PM +0800, Wu Hao wrote:
> > On Wed, Jan 31, 2018 at 08:52:36AM -0600, Alan Tull wrote:
> > > On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:
> > > 
> > > Hi Hao,
> > > 
> > > I'm adding my "Acked-by' below.  When you post v4, please add it so
> > > that we can keep track of what got acked.
> > 
> > Sure, thanks a lot for the code review. :)
> > 
> > Hao
> > 
> > > 
> > > Thanks,
> > > Alan
> > > 
> > > > FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> > > > need to be supported by all feature devices drivers including FME and AFU.
> > > > This patch implements above 2 ioctls in FPGA Accelerated Function Unit
> > > > (AFU) driver.
> > > >
> > > > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > > > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > Acked-by: Alan Tull <atull@kernel.org>
> Acked-by: Moritz Fischer <mdf@kernel.org>
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Looks good,

Thanks a lot for the code review. :)

Hao

> 
> Moritz

[PATCH v3 20/21] fpga: dfl: afu: add user afu sub feature support

[Wu Hao]

From: Xiao Guangrong <guangrong.xiao@linux.intel.com>

User Accelerated Function Unit sub feature exposes the MMIO region of
the AFU. After valid green bitstream (GBS) is programmed and port is
enabled, then this MMIO region could be accessed.

This patch adds support to enumerate the AFU MMIO region and expose it
to userspace via mmap file operation. Below interfaces are exposed to user:

Sysfs interface:
* /sys/class/fpga_region/<regionX>/<fpga-dfl-port.x>/afu_id
  Read-only. Indicate which green bitstream is programmed to this AFU.

Ioctl interfaces:
* FPGA_PORT_GET_INFO
  Provide info to userspace on the number of supported region.
  Only UAFU region is supported now.

* FPGA_PORT_GET_REGION_INFO
  Provide region information, including access permission, region size,
  offset from the start of device fd.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
------
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    add sysfs documentation.
    switched to GPLv2 license.
v3: rename driver to fpga-dfl-afu
    fix coding style and checkpatch issue.
    only allow afu_id to be read when port isn't in reset.
---
 .../ABI/testing/sysfs-platform-fpga-dfl-afu        |   9 +
 drivers/fpga/Makefile                              |   2 +-
 drivers/fpga/dfl-afu-main.c                        | 211 ++++++++++++++++++++-
 drivers/fpga/dfl-afu-region.c                      | 127 +++++++++++++
 drivers/fpga/dfl-afu.h                             |  54 ++++++
 include/uapi/linux/fpga-dfl.h                      |  47 +++++
 6 files changed, 446 insertions(+), 4 deletions(-)
 create mode 100644 drivers/fpga/dfl-afu-region.c
 create mode 100644 drivers/fpga/dfl-afu.h

diff --git a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
index f4bcd94..54bcfad 100644
--- a/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
+++ b/Documentation/ABI/testing/sysfs-platform-fpga-dfl-afu
@@ -5,3 +5,12 @@ Contact:	Wu Hao <hao.wu@intel.com>
 Description:	Read-only. It returns id of this port. One DFL FPGA device
 		may have more than one port. Userspace could use this id to
 		distinguish different ports under same FPGA device.
+
+What:		/sys/bus/platform/devices/fpga-dfl-port.0/afu_id
+Date:		November 2017
+KernelVersion:  4.15
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. User can program different green bitstreams (GBS) to
+		FPGA Accelerator Function Unit (AFU) for different functions.
+		It returns uuid which could be used to identify which GBS is
+		programmed in this AFU.
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index e234fa8..f0dd09f 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -37,7 +37,7 @@ obj-$(CONFIG_FPGA_DFL_FME_REGION)	+= fpga-dfl-fme-region.o
 obj-$(CONFIG_FPGA_DFL_AFU)		+= fpga-dfl-afu.o
 
 fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
-fpga-dfl-afu-objs := dfl-afu-main.o
+fpga-dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_INTEL_FPGA_DFL_PCI)	+= intel-dfl-pci.o
diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index d3d534c..ca37a51 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c
@@ -18,9 +18,10 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/uaccess.h>
 #include <linux/fpga-dfl.h>
 
-#include "fpga-dfl.h"
+#include "dfl-afu.h"
 
 static ssize_t
 id_show(struct device *dev, struct device_attribute *attr, char *buf)
@@ -80,12 +81,74 @@ static void port_hdr_uinit(struct platform_device *pdev,
 	.ioctl = port_hdr_ioctl,
 };
 
+static ssize_t
+afu_id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(dev);
+	void __iomem *base;
+	u64 guidl, guidh;
+
+	base = get_feature_ioaddr_by_index(dev, PORT_FEATURE_ID_UAFU);
+
+	mutex_lock(&pdata->lock);
+	if (pdata->disable_count) {
+		mutex_unlock(&pdata->lock);
+		return -EBUSY;
+	}
+
+	guidl = readq(base + GUID_L);
+	guidh = readq(base + GUID_H);
+	mutex_unlock(&pdata->lock);
+
+	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n", guidh, guidl);
+}
+static DEVICE_ATTR_RO(afu_id);
+
+static const struct attribute *port_uafu_attrs[] = {
+	&dev_attr_afu_id.attr,
+	NULL
+};
+
+static int port_uafu_init(struct platform_device *pdev, struct feature *feature)
+{
+	struct resource *res = &pdev->resource[feature->resource_index];
+	u32 flags = FPGA_REGION_READ | FPGA_REGION_WRITE | FPGA_REGION_MMAP;
+	int ret;
+
+	dev_dbg(&pdev->dev, "PORT AFU Init.\n");
+
+	ret = afu_region_add(dev_get_platdata(&pdev->dev),
+			     FPGA_PORT_INDEX_UAFU, resource_size(res),
+			     res->start, flags);
+	if (ret)
+		return ret;
+
+	return sysfs_create_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static void port_uafu_uinit(struct platform_device *pdev,
+			    struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT AFU UInit.\n");
+
+	sysfs_remove_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static const struct feature_ops port_uafu_ops = {
+	.init = port_uafu_init,
+	.uinit = port_uafu_uinit,
+};
+
 static struct feature_driver port_feature_drvs[] = {
 	{
 		.name = PORT_FEATURE_HEADER,
 		.ops = &port_hdr_ops,
 	},
 	{
+		.name = PORT_FEATURE_UAFU,
+		.ops = &port_uafu_ops,
+	},
+	{
 		.ops = NULL,
 	}
 };
@@ -130,6 +193,64 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
 	return 0;
 }
 
+static long
+afu_ioctl_get_info(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_info info;
+	struct fpga_afu *afu;
+	unsigned long minsz;
+
+	minsz = offsetofend(struct fpga_port_info, num_umsgs);
+
+	if (copy_from_user(&info, arg, minsz))
+		return -EFAULT;
+
+	if (info.argsz < minsz)
+		return -EINVAL;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	info.flags = 0;
+	info.num_regions = afu->num_regions;
+	info.num_umsgs = afu->num_umsgs;
+	mutex_unlock(&pdata->lock);
+
+	if (copy_to_user(arg, &info, sizeof(info)))
+		return -EFAULT;
+
+	return 0;
+}
+
+static long
+afu_ioctl_get_region_info(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_region_info rinfo;
+	struct fpga_afu_region region;
+	unsigned long minsz;
+	long ret;
+
+	minsz = offsetofend(struct fpga_port_region_info, offset);
+
+	if (copy_from_user(&rinfo, arg, minsz))
+		return -EFAULT;
+
+	if (rinfo.argsz < minsz || rinfo.padding)
+		return -EINVAL;
+
+	ret = afu_get_region_by_index(pdata, rinfo.index, &region);
+	if (ret)
+		return ret;
+
+	rinfo.flags = region.flags;
+	rinfo.size = region.size;
+	rinfo.offset = region.offset;
+
+	if (copy_to_user(arg, &rinfo, sizeof(rinfo)))
+		return -EFAULT;
+
+	return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;
@@ -144,6 +265,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 		return FPGA_API_VERSION;
 	case FPGA_CHECK_EXTENSION:
 		return afu_ioctl_check_extension(pdata, arg);
+	case FPGA_PORT_GET_INFO:
+		return afu_ioctl_get_info(pdata, (void __user *)arg);
+	case FPGA_PORT_GET_REGION_INFO:
+		return afu_ioctl_get_region_info(pdata, (void __user *)arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
@@ -164,27 +289,106 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	return -EINVAL;
 }
 
+static int afu_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+	struct fpga_afu_region region;
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	u64 size = vma->vm_end - vma->vm_start;
+	u64 offset;
+	int ret;
+
+	if (!(vma->vm_flags & VM_SHARED))
+		return -EINVAL;
+
+	offset = vma->vm_pgoff << PAGE_SHIFT;
+	ret = afu_get_region_by_offset(pdata, offset, size, &region);
+	if (ret)
+		return ret;
+
+	if (!(region.flags & FPGA_REGION_MMAP))
+		return -EINVAL;
+
+	if ((vma->vm_flags & VM_READ) && !(region.flags & FPGA_REGION_READ))
+		return -EPERM;
+
+	if ((vma->vm_flags & VM_WRITE) && !(region.flags & FPGA_REGION_WRITE))
+		return -EPERM;
+
+	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+	return remap_pfn_range(vma, vma->vm_start,
+			(region.phys + (offset - region.offset)) >> PAGE_SHIFT,
+			size, vma->vm_page_prot);
+}
+
 static const struct file_operations afu_fops = {
 	.owner = THIS_MODULE,
 	.open = afu_open,
 	.release = afu_release,
 	.unlocked_ioctl = afu_ioctl,
+	.mmap = afu_mmap,
 };
 
+static int afu_dev_init(struct platform_device *pdev)
+{
+	struct fpga_afu *afu;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	afu = devm_kzalloc(&pdev->dev, sizeof(*afu), GFP_KERNEL);
+	if (!afu)
+		return -ENOMEM;
+
+	afu->pdata = pdata;
+
+	mutex_lock(&pdata->lock);
+	fpga_pdata_set_private(pdata, afu);
+	afu_region_init(pdata);
+	mutex_unlock(&pdata->lock);
+	return 0;
+}
+
+static int afu_dev_destroy(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_afu *afu;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	afu_region_destroy(pdata);
+	fpga_pdata_set_private(pdata, NULL);
+	mutex_unlock(&pdata->lock);
+
+	devm_kfree(&pdev->dev, afu);
+
+	return 0;
+}
+
 static int afu_probe(struct platform_device *pdev)
 {
 	int ret;
 
 	dev_dbg(&pdev->dev, "%s\n", __func__);
 
+	ret = afu_dev_init(pdev);
+	if (ret)
+		goto exit;
+
 	ret = fpga_dev_feature_init(pdev, port_feature_drvs);
 	if (ret)
-		return ret;
+		goto dev_destroy;
 
 	ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
-	if (ret)
+	if (ret) {
 		fpga_dev_feature_uinit(pdev);
+		goto dev_destroy;
+	}
+
+	return 0;
 
+dev_destroy:
+	afu_dev_destroy(pdev);
+exit:
 	return ret;
 }
 
@@ -194,6 +398,7 @@ static int afu_remove(struct platform_device *pdev)
 
 	fpga_dev_feature_uinit(pdev);
 	fpga_unregister_dev_ops(pdev);
+	afu_dev_destroy(pdev);
 
 	return 0;
 }
diff --git a/drivers/fpga/dfl-afu-region.c b/drivers/fpga/dfl-afu-region.c
new file mode 100644
index 0000000..1f711ce
--- /dev/null
+++ b/drivers/fpga/dfl-afu-region.c
@@ -0,0 +1,127 @@
+/*
+ * Driver for FPGA Accelerated Function Unit (AFU) Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+#include "dfl-afu.h"
+
+void afu_region_init(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	INIT_LIST_HEAD(&afu->regions);
+}
+
+#define for_each_region(region, afu)	\
+	list_for_each_entry((region), &(afu)->regions, node)
+static struct fpga_afu_region *get_region_by_index(struct fpga_afu *afu,
+						   u32 region_index)
+{
+	struct fpga_afu_region *region;
+
+	for_each_region(region, afu)
+		if (region->index == region_index)
+			return region;
+
+	return NULL;
+}
+
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+		   u64 region_size, u64 phys, u32 flags)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	region = devm_kzalloc(&pdata->dev->dev, sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return -ENOMEM;
+
+	region->index = region_index;
+	region->size = region_size;
+	region->phys = phys;
+	region->flags = flags;
+
+	mutex_lock(&pdata->lock);
+
+	afu = fpga_pdata_get_private(pdata);
+
+	/* check if @index already exists */
+	if (get_region_by_index(afu, region_index)) {
+		mutex_unlock(&pdata->lock);
+		ret = -EEXIST;
+		goto exit;
+	}
+
+	region_size = PAGE_ALIGN(region_size);
+	region->offset = afu->region_cur_offset;
+	list_add(&region->node, &afu->regions);
+
+	afu->region_cur_offset += region_size;
+	afu->num_regions++;
+	mutex_unlock(&pdata->lock);
+
+	return 0;
+
+exit:
+	devm_kfree(&pdata->dev->dev, region);
+	return ret;
+}
+
+void afu_region_destroy(struct feature_platform_data *pdata)
+{
+	struct fpga_afu_region *tmp, *region;
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	list_for_each_entry_safe(region, tmp, &afu->regions, node)
+		devm_kfree(&pdata->dev->dev, region);
+}
+
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+			    u32 region_index, struct fpga_afu_region *pregion)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	region = get_region_by_index(afu, region_index);
+	if (!region) {
+		ret = -EINVAL;
+		goto exit;
+	}
+	*pregion = *region;
+exit:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
+
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+			     u64 offset, u64 size,
+			     struct fpga_afu_region *pregion)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	for_each_region(region, afu)
+		if (region->offset <= offset &&
+		    region->offset + region->size >= offset + size) {
+			*pregion = *region;
+			goto exit;
+		}
+	ret = -EINVAL;
+exit:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
diff --git a/drivers/fpga/dfl-afu.h b/drivers/fpga/dfl-afu.h
new file mode 100644
index 0000000..78f9696
--- /dev/null
+++ b/drivers/fpga/dfl-afu.h
@@ -0,0 +1,54 @@
+/*
+ * Header file for FPGA Accelerated Function Unit (AFU) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *     Wu Hao <hao.wu@intel.com>
+ *     Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *     Joseph Grecco <joe.grecco@intel.com>
+ *     Enno Luebbers <enno.luebbers@intel.com>
+ *     Tim Whisonant <tim.whisonant@intel.com>
+ *     Ananda Ravuri <ananda.ravuri@intel.com>
+ *     Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#ifndef __DFL_AFU_H
+#define __DFL_AFU_H
+
+#include <linux/mm.h>
+
+#include "fpga-dfl.h"
+
+struct fpga_afu_region {
+	u32 index;
+	u32 flags;
+	u64 size;
+	u64 offset;
+	u64 phys;
+	struct list_head node;
+};
+
+struct fpga_afu {
+	u64 region_cur_offset;
+	int num_regions;
+	u8 num_umsgs;
+	struct list_head regions;
+
+	struct feature_platform_data *pdata;
+};
+
+void afu_region_init(struct feature_platform_data *pdata);
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+		   u64 region_size, u64 phys, u32 flags);
+void afu_region_destroy(struct feature_platform_data *pdata);
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+			    u32 region_index, struct fpga_afu_region *pregion);
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+			     u64 offset, u64 size,
+			     struct fpga_afu_region *pregion);
+
+#endif
diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 9bf273d..78937b0 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h
@@ -67,6 +67,53 @@
 
 #define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
 
+/**
+ * FPGA_PORT_GET_INFO - _IOR(FPGA_MAGIC, PORT_BASE + 1, struct fpga_port_info)
+ *
+ * Retrieve information about the fpga port.
+ * Driver fills the info in provided struct fpga_port_info.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_info {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	/* Output */
+	__u32 flags;		/* Zero for now */
+	__u32 num_regions;	/* The number of supported regions */
+	__u32 num_umsgs;	/* The number of allocated umsgs */
+};
+
+#define FPGA_PORT_GET_INFO	_IO(FPGA_MAGIC, PORT_BASE + 1)
+
+/**
+ * FPGA_PORT_GET_REGION_INFO - _IOWR(FPGA_MAGIC, PORT_BASE + 2,
+ *						struct fpga_port_region_info)
+ *
+ * Retrieve information about a device region.
+ * Caller provides struct fpga_port_region_info with index value set.
+ * Driver returns the region info in other fields.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_region_info {
+	/* input */
+	__u32 argsz;		/* Structure length */
+	/* Output */
+	__u32 flags;		/* Access permission */
+#define FPGA_REGION_READ	(1 << 0)	/* Region is readable */
+#define FPGA_REGION_WRITE	(1 << 1)	/* Region is writable */
+#define FPGA_REGION_MMAP	(1 << 2)	/* Can be mmaped to userspace */
+	/* Input */
+	__u32 index;		/* Region index */
+#define FPGA_PORT_INDEX_UAFU	0		/* User AFU */
+#define FPGA_PORT_INDEX_STP	1		/* Signal Tap */
+	__u32 padding;
+	/* Output */
+	__u64 size;		/* Region size (bytes) */
+	__u64 offset;		/* Region offset from start of device fd */
+};
+
+#define FPGA_PORT_GET_REGION_INFO	_IO(FPGA_MAGIC, PORT_BASE + 2)
+
 /* IOCTLs for FME file descriptor */
 
 /**
-- 
1.8.3.1

[PATCH v3 21/21] fpga: dfl: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

[Wu Hao]

DMA memory regions are required for Accelerated Function Unit (AFU) usage.
These two ioctls allow user space applications to map user memory regions
for dma, and unmap them after use. Iova is returned from driver to user
space application via FPGA_PORT_DMA_MAP ioctl. Application needs to unmap
it after use, otherwise, driver will unmap them in device file release
operation.

Each AFU has its own rb tree to keep track of its mapped DMA regions.

Ioctl interfaces:
* FPGA_PORT_DMA_MAP
  Do the dma mapping per user_addr and length which provided by user.
  Return iova in provided struct afu_port_dma_map.

* FPGA_PORT_DMA_UNMAP
  Unmap the dma region per iova provided by user.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
-----
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    fixed kbuild warnings.
v3: improved commit description and fixed coding style issues.
    replaced fpga_pdata_to_pcidev with fpga_pdata_to_fpga_cdev
---
 drivers/fpga/Makefile             |   2 +-
 drivers/fpga/dfl-afu-dma-region.c | 465 ++++++++++++++++++++++++++++++++++++++
 drivers/fpga/dfl-afu-main.c       |  61 ++++-
 drivers/fpga/dfl-afu.h            |  18 ++
 include/uapi/linux/fpga-dfl.h     |  37 +++
 5 files changed, 581 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/dfl-afu-dma-region.c

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index f0dd09f..5715a78 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -37,7 +37,7 @@ obj-$(CONFIG_FPGA_DFL_FME_REGION)	+= fpga-dfl-fme-region.o
 obj-$(CONFIG_FPGA_DFL_AFU)		+= fpga-dfl-afu.o
 
 fpga-dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
-fpga-dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o
+fpga-dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o dfl-afu-dma-region.o
 
 # Drivers for FPGAs which implement DFL
 obj-$(CONFIG_INTEL_FPGA_DFL_PCI)	+= intel-dfl-pci.o
diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c
new file mode 100644
index 0000000..7b246bb
--- /dev/null
+++ b/drivers/fpga/dfl-afu-dma-region.c
@@ -0,0 +1,465 @@
+/*
+ * Driver for FPGA Accelerated Function Unit (AFU) DMA Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2.
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/sched/signal.h>
+#include <linux/uaccess.h>
+
+#include "dfl-afu.h"
+
+static void put_all_pages(struct page **pages, int npages)
+{
+	int i;
+
+	for (i = 0; i < npages; i++)
+		if (pages[i])
+			put_page(pages[i]);
+}
+
+void afu_dma_region_init(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	afu->dma_regions = RB_ROOT;
+}
+
+/**
+ * afu_dma_adjust_locked_vm - adjust locked memory
+ * @dev: port device
+ * @npages: number of pages
+ * @incr: increase or decrease locked memory
+ *
+ * Increase or decrease the locked memory size with npages input.
+ *
+ * Return 0 on success.
+ * Return -ENOMEM if locked memory size is over the limit and no CAP_IPC_LOCK.
+ */
+static int afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
+{
+	unsigned long locked, lock_limit;
+	int ret = 0;
+
+	/* the task is exiting. */
+	if (!current->mm)
+		return 0;
+
+	down_write(&current->mm->mmap_sem);
+
+	if (incr) {
+		locked = current->mm->locked_vm + npages;
+		lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+
+		if (locked > lock_limit && !capable(CAP_IPC_LOCK))
+			ret = -ENOMEM;
+		else
+			current->mm->locked_vm += npages;
+	} else {
+		if (WARN_ON_ONCE(npages > current->mm->locked_vm))
+			npages = current->mm->locked_vm;
+		current->mm->locked_vm -= npages;
+	}
+
+	dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
+		incr ? '+' : '-', npages << PAGE_SHIFT,
+		current->mm->locked_vm << PAGE_SHIFT, rlimit(RLIMIT_MEMLOCK),
+		ret ? "- execeeded" : "");
+
+	up_write(&current->mm->mmap_sem);
+
+	return ret;
+}
+
+/**
+ * afu_dma_pin_pages - pin pages of given dma memory region
+ * @pdata: feature device platform data
+ * @region: dma memory region to be pinned
+ *
+ * Pin all the pages of given fpga_afu_dma_region.
+ * Return 0 for success or negative error code.
+ */
+static int afu_dma_pin_pages(struct feature_platform_data *pdata,
+			     struct fpga_afu_dma_region *region)
+{
+	int npages = region->length >> PAGE_SHIFT;
+	struct device *dev = &pdata->dev->dev;
+	int ret, pinned;
+
+	ret = afu_dma_adjust_locked_vm(dev, npages, true);
+	if (ret)
+		return ret;
+
+	region->pages = kcalloc(npages, sizeof(struct page *), GFP_KERNEL);
+	if (!region->pages) {
+		ret = -ENOMEM;
+		goto unlock_vm;
+	}
+
+	pinned = get_user_pages_fast(region->user_addr, npages, 1,
+				     region->pages);
+	if (pinned < 0) {
+		ret = pinned;
+		goto put_pages;
+	} else if (pinned != npages) {
+		ret = -EFAULT;
+		goto free_pages;
+	}
+
+	dev_dbg(dev, "%d pages pinned\n", pinned);
+
+	return 0;
+
+put_pages:
+	put_all_pages(region->pages, pinned);
+free_pages:
+	kfree(region->pages);
+unlock_vm:
+	afu_dma_adjust_locked_vm(dev, npages, false);
+	return ret;
+}
+
+/**
+ * afu_dma_unpin_pages - unpin pages of given dma memory region
+ * @pdata: feature device platform data
+ * @region: dma memory region to be unpinned
+ *
+ * Unpin all the pages of given fpga_afu_dma_region.
+ * Return 0 for success or negative error code.
+ */
+static void afu_dma_unpin_pages(struct feature_platform_data *pdata,
+				struct fpga_afu_dma_region *region)
+{
+	long npages = region->length >> PAGE_SHIFT;
+	struct device *dev = &pdata->dev->dev;
+
+	put_all_pages(region->pages, npages);
+	kfree(region->pages);
+	afu_dma_adjust_locked_vm(dev, npages, false);
+
+	dev_dbg(dev, "%ld pages unpinned\n", npages);
+}
+
+/**
+ * afu_dma_check_continuous_pages - check if pages are continuous
+ * @region: dma memory region
+ *
+ * Return true if pages of given dma memory region have continuous physical
+ * address, otherwise return false.
+ */
+static bool afu_dma_check_continuous_pages(struct fpga_afu_dma_region *region)
+{
+	int npages = region->length >> PAGE_SHIFT;
+	int i;
+
+	for (i = 0; i < npages - 1; i++)
+		if (page_to_pfn(region->pages[i]) + 1 !=
+				page_to_pfn(region->pages[i + 1]))
+			return false;
+
+	return true;
+}
+
+/**
+ * dma_region_check_iova - check if memory area is fully contained in the region
+ * @region: dma memory region
+ * @iova: address of the dma memory area
+ * @size: size of the dma memory area
+ *
+ * Compare the dma memory area defined by @iova and @size with given dma region.
+ * Return true if memory area is fully contained in the region, otherwise false.
+ */
+static bool dma_region_check_iova(struct fpga_afu_dma_region *region,
+				  u64 iova, u64 size)
+{
+	if (!size && region->iova != iova)
+		return false;
+
+	return (region->iova <= iova) &&
+		(region->length + region->iova >= iova + size);
+}
+
+/**
+ * afu_dma_region_add - add given dma region to rbtree
+ * @pdata: feature device platform data
+ * @region: dma region to be added
+ *
+ * Return 0 for success, -EEXIST if dma region has already been added.
+ *
+ * Needs to be called with pdata->lock heold.
+ */
+static int afu_dma_region_add(struct feature_platform_data *pdata,
+			      struct fpga_afu_dma_region *region)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node **new, *parent = NULL;
+
+	dev_dbg(&pdata->dev->dev, "add region (iova = %llx)\n",
+		(unsigned long long)region->iova);
+
+	new = &afu->dma_regions.rb_node;
+
+	while (*new) {
+		struct fpga_afu_dma_region *this;
+
+		this = container_of(*new, struct fpga_afu_dma_region, node);
+
+		parent = *new;
+
+		if (dma_region_check_iova(this, region->iova, region->length))
+			return -EEXIST;
+
+		if (region->iova < this->iova)
+			new = &((*new)->rb_left);
+		else if (region->iova > this->iova)
+			new = &((*new)->rb_right);
+		else
+			return -EEXIST;
+	}
+
+	rb_link_node(&region->node, parent, new);
+	rb_insert_color(&region->node, &afu->dma_regions);
+
+	return 0;
+}
+
+/**
+ * afu_dma_region_remove - remove given dma region from rbtree
+ * @pdata: feature device platform data
+ * @region: dma region to be removed
+ *
+ * Needs to be called with pdata->lock heold.
+ */
+static void afu_dma_region_remove(struct feature_platform_data *pdata,
+				  struct fpga_afu_dma_region *region)
+{
+	struct fpga_afu *afu;
+
+	dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
+		(unsigned long long)region->iova);
+
+	afu = fpga_pdata_get_private(pdata);
+	rb_erase(&region->node, &afu->dma_regions);
+}
+
+/**
+ * afu_dma_region_destroy - destroy all regions in rbtree
+ * @pdata: feature device platform data
+ *
+ * Needs to be called with pdata->lock heold.
+ */
+void afu_dma_region_destroy(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node *node = rb_first(&afu->dma_regions);
+	struct fpga_afu_dma_region *region;
+
+	while (node) {
+		region = container_of(node, struct fpga_afu_dma_region, node);
+
+		dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
+			(unsigned long long)region->iova);
+
+		rb_erase(node, &afu->dma_regions);
+
+		if (region->iova)
+			dma_unmap_page(fpga_pdata_to_parent(pdata),
+				       region->iova, region->length,
+				       DMA_BIDIRECTIONAL);
+
+		if (region->pages)
+			afu_dma_unpin_pages(pdata, region);
+
+		node = rb_next(node);
+		kfree(region);
+	}
+}
+
+/**
+ * afu_dma_region_find - find the dma region from rbtree based on iova and size
+ * @pdata: feature device platform data
+ * @iova: address of the dma memory area
+ * @size: size of the dma memory area
+ *
+ * It finds the dma region from the rbtree based on @iova and @size:
+ * - if @size == 0, it finds the dma region which starts from @iova
+ * - otherwise, it finds the dma region which fully contains
+ *   [@iova, @iova+size)
+ * If nothing is matched returns NULL.
+ *
+ * Needs to be called with pdata->lock held.
+ */
+struct fpga_afu_dma_region *
+afu_dma_region_find(struct feature_platform_data *pdata, u64 iova, u64 size)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node *node = afu->dma_regions.rb_node;
+	struct device *dev = &pdata->dev->dev;
+
+	while (node) {
+		struct fpga_afu_dma_region *region;
+
+		region = container_of(node, struct fpga_afu_dma_region, node);
+
+		if (dma_region_check_iova(region, iova, size)) {
+			dev_dbg(dev, "find region (iova = %llx)\n",
+				(unsigned long long)region->iova);
+			return region;
+		}
+
+		if (iova < region->iova)
+			node = node->rb_left;
+		else if (iova > region->iova)
+			node = node->rb_right;
+		else
+			/* the iova region is not fully covered. */
+			break;
+	}
+
+	dev_dbg(dev, "region with iova %llx and size %llx is not found\n",
+		(unsigned long long)iova, (unsigned long long)size);
+
+	return NULL;
+}
+
+/**
+ * afu_dma_region_find_iova - find the dma region from rbtree by iova
+ * @pdata: feature device platform data
+ * @iova: address of the dma region
+ *
+ * Needs to be called with pdata->lock held.
+ */
+static struct fpga_afu_dma_region *
+afu_dma_region_find_iova(struct feature_platform_data *pdata, u64 iova)
+{
+	return afu_dma_region_find(pdata, iova, 0);
+}
+
+/**
+ * afu_dma_map_region - map memory region for dma
+ * @pdata: feature device platform data
+ * @user_addr: address of the memory region
+ * @length: size of the memory region
+ * @iova: pointer of iova address
+ *
+ * Map memory region defined by @user_addr and @length, and return dma address
+ * of the memory region via @iova.
+ * Return 0 for success, otherwise error code.
+ */
+int afu_dma_map_region(struct feature_platform_data *pdata,
+		       u64 user_addr, u64 length, u64 *iova)
+{
+	struct fpga_afu_dma_region *region;
+	int ret;
+
+	/*
+	 * Check Inputs, only accept page-aligned user memory region with
+	 * valid length.
+	 */
+	if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
+		return -EINVAL;
+
+	/* Check overflow */
+	if (user_addr + length < user_addr)
+		return -EINVAL;
+
+	if (!access_ok(VERIFY_WRITE, (void __user *)(unsigned long)user_addr,
+		       length))
+		return -EINVAL;
+
+	region = kzalloc(sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return -ENOMEM;
+
+	region->user_addr = user_addr;
+	region->length = length;
+
+	/* Pin the user memory region */
+	ret = afu_dma_pin_pages(pdata, region);
+	if (ret) {
+		dev_err(&pdata->dev->dev, "failed to pin memory region\n");
+		goto free_region;
+	}
+
+	/* Only accept continuous pages, return error else */
+	if (!afu_dma_check_continuous_pages(region)) {
+		dev_err(&pdata->dev->dev, "pages are not continuous\n");
+		ret = -EINVAL;
+		goto unpin_pages;
+	}
+
+	/* As pages are continuous then start to do DMA mapping */
+	region->iova = dma_map_page(fpga_pdata_to_parent(pdata),
+				    region->pages[0], 0,
+				    region->length,
+				    DMA_BIDIRECTIONAL);
+	if (dma_mapping_error(&pdata->dev->dev, region->iova)) {
+		dev_err(&pdata->dev->dev, "failed to map for dma\n");
+		ret = -EFAULT;
+		goto unpin_pages;
+	}
+
+	*iova = region->iova;
+
+	mutex_lock(&pdata->lock);
+	ret = afu_dma_region_add(pdata, region);
+	mutex_unlock(&pdata->lock);
+	if (ret) {
+		dev_err(&pdata->dev->dev, "failed to add dma region\n");
+		goto unmap_dma;
+	}
+
+	return 0;
+
+unmap_dma:
+	dma_unmap_page(fpga_pdata_to_parent(pdata),
+		       region->iova, region->length, DMA_BIDIRECTIONAL);
+unpin_pages:
+	afu_dma_unpin_pages(pdata, region);
+free_region:
+	kfree(region);
+	return ret;
+}
+
+/**
+ * afu_dma_unmap_region - unmap dma memory region
+ * @pdata: feature device platform data
+ * @iova: dma address of the region
+ *
+ * Unmap dma memory region based on @iova.
+ * Return 0 for success, otherwise error code.
+ */
+int afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova)
+{
+	struct fpga_afu_dma_region *region;
+
+	mutex_lock(&pdata->lock);
+	region = afu_dma_region_find_iova(pdata, iova);
+	if (!region) {
+		mutex_unlock(&pdata->lock);
+		return -EINVAL;
+	}
+
+	if (region->in_use) {
+		mutex_unlock(&pdata->lock);
+		return -EBUSY;
+	}
+
+	afu_dma_region_remove(pdata, region);
+	mutex_unlock(&pdata->lock);
+
+	dma_unmap_page(fpga_pdata_to_parent(pdata),
+		       region->iova, region->length, DMA_BIDIRECTIONAL);
+	afu_dma_unpin_pages(pdata, region);
+	kfree(region);
+
+	return 0;
+}
diff --git a/drivers/fpga/dfl-afu-main.c b/drivers/fpga/dfl-afu-main.c
index ca37a51..dac2c96 100644
--- a/drivers/fpga/dfl-afu-main.c
+++ b/drivers/fpga/dfl-afu-main.c
@@ -180,7 +180,11 @@ static int afu_release(struct inode *inode, struct file *filp)
 
 	dev_dbg(&pdev->dev, "Device File Release\n");
 
-	fpga_port_reset(pdev);
+	mutex_lock(&pdata->lock);
+	__fpga_port_reset(pdev);
+	afu_dma_region_destroy(pdata);
+	mutex_unlock(&pdata->lock);
+
 	feature_dev_use_end(pdata);
 
 	return 0;
@@ -251,6 +255,55 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
 	return 0;
 }
 
+static long
+afu_ioctl_dma_map(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_dma_map map;
+	unsigned long minsz;
+	long ret;
+
+	minsz = offsetofend(struct fpga_port_dma_map, iova);
+
+	if (copy_from_user(&map, arg, minsz))
+		return -EFAULT;
+
+	if (map.argsz < minsz || map.flags)
+		return -EINVAL;
+
+	ret = afu_dma_map_region(pdata, map.user_addr, map.length, &map.iova);
+	if (ret)
+		return ret;
+
+	if (copy_to_user(arg, &map, sizeof(map))) {
+		afu_dma_unmap_region(pdata, map.iova);
+		return -EFAULT;
+	}
+
+	dev_dbg(&pdata->dev->dev, "dma map: ua=%llx, len=%llx, iova=%llx\n",
+		(unsigned long long)map.user_addr,
+		(unsigned long long)map.length,
+		(unsigned long long)map.iova);
+
+	return 0;
+}
+
+static long
+afu_ioctl_dma_unmap(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_dma_unmap unmap;
+	unsigned long minsz;
+
+	minsz = offsetofend(struct fpga_port_dma_unmap, iova);
+
+	if (copy_from_user(&unmap, arg, minsz))
+		return -EFAULT;
+
+	if (unmap.argsz < minsz || unmap.flags)
+		return -EINVAL;
+
+	return afu_dma_unmap_region(pdata, unmap.iova);
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;
@@ -269,6 +322,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 		return afu_ioctl_get_info(pdata, (void __user *)arg);
 	case FPGA_PORT_GET_REGION_INFO:
 		return afu_ioctl_get_region_info(pdata, (void __user *)arg);
+	case FPGA_PORT_DMA_MAP:
+		return afu_ioctl_dma_map(pdata, (void __user *)arg);
+	case FPGA_PORT_DMA_UNMAP:
+		return afu_ioctl_dma_unmap(pdata, (void __user *)arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
@@ -344,6 +401,7 @@ static int afu_dev_init(struct platform_device *pdev)
 	mutex_lock(&pdata->lock);
 	fpga_pdata_set_private(pdata, afu);
 	afu_region_init(pdata);
+	afu_dma_region_init(pdata);
 	mutex_unlock(&pdata->lock);
 	return 0;
 }
@@ -356,6 +414,7 @@ static int afu_dev_destroy(struct platform_device *pdev)
 	mutex_lock(&pdata->lock);
 	afu = fpga_pdata_get_private(pdata);
 	afu_region_destroy(pdata);
+	afu_dma_region_destroy(pdata);
 	fpga_pdata_set_private(pdata, NULL);
 	mutex_unlock(&pdata->lock);
 
diff --git a/drivers/fpga/dfl-afu.h b/drivers/fpga/dfl-afu.h
index 78f9696..46c4e71f 100644
--- a/drivers/fpga/dfl-afu.h
+++ b/drivers/fpga/dfl-afu.h
@@ -32,11 +32,21 @@ struct fpga_afu_region {
 	struct list_head node;
 };
 
+struct fpga_afu_dma_region {
+	u64 user_addr;
+	u64 length;
+	u64 iova;
+	struct page **pages;
+	struct rb_node node;
+	bool in_use;
+};
+
 struct fpga_afu {
 	u64 region_cur_offset;
 	int num_regions;
 	u8 num_umsgs;
 	struct list_head regions;
+	struct rb_root dma_regions;
 
 	struct feature_platform_data *pdata;
 };
@@ -51,4 +61,12 @@ int afu_get_region_by_offset(struct feature_platform_data *pdata,
 			     u64 offset, u64 size,
 			     struct fpga_afu_region *pregion);
 
+void afu_dma_region_init(struct feature_platform_data *pdata);
+void afu_dma_region_destroy(struct feature_platform_data *pdata);
+int afu_dma_map_region(struct feature_platform_data *pdata,
+		       u64 user_addr, u64 length, u64 *iova);
+int afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova);
+struct fpga_afu_dma_region *afu_dma_region_find(
+		struct feature_platform_data *pdata, u64 iova, u64 size);
+
 #endif
diff --git a/include/uapi/linux/fpga-dfl.h b/include/uapi/linux/fpga-dfl.h
index 78937b0..a052e76 100644
--- a/include/uapi/linux/fpga-dfl.h
+++ b/include/uapi/linux/fpga-dfl.h
@@ -114,6 +114,43 @@ struct fpga_port_region_info {
 
 #define FPGA_PORT_GET_REGION_INFO	_IO(FPGA_MAGIC, PORT_BASE + 2)
 
+/**
+ * FPGA_PORT_DMA_MAP - _IOWR(FPGA_MAGIC, PORT_BASE + 3,
+ *						struct fpga_port_dma_map)
+ *
+ * Map the dma memory per user_addr and length which are provided by caller.
+ * Driver fills the iova in provided struct afu_port_dma_map.
+ * This interface only accepts page-size aligned user memory for dma mapping.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_dma_map {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u64 user_addr;        /* Process virtual address */
+	__u64 length;           /* Length of mapping (bytes)*/
+	/* Output */
+	__u64 iova;             /* IO virtual address */
+};
+
+#define FPGA_PORT_DMA_MAP	_IO(FPGA_MAGIC, PORT_BASE + 3)
+
+/**
+ * FPGA_PORT_DMA_UNMAP - _IOW(FPGA_MAGIC, PORT_BASE + 4,
+ *						struct fpga_port_dma_unmap)
+ *
+ * Unmap the dma memory per iova provided by caller.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_dma_unmap {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u64 iova;		/* IO virtual address */
+};
+
+#define FPGA_PORT_DMA_UNMAP	_IO(FPGA_MAGIC, PORT_BASE + 4)
+
 /* IOCTLs for FME file descriptor */
 
 /**
-- 
1.8.3.1

Re: [PATCH v3 00/21] Intel FPGA Device Drivers

[Alan Tull]

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@intel.com> wrote:

> This patch series depends on the below patchset from Alan Tull.
> [PATCH v2 0/5] fpga: don't use drvdata in common fpga code[1]
>
> [1] https://marc.info/?l=linux-fpga&m=151077942606263&w=2

Hi Hao,

Thanks for basing your v3 on my latest patches.  I'll be looking at them.

Alan